馃殻 captain.log - IPFS JavaScript implementation 馃専 #30

Open
diasdavid opened this Issue Sep 22, 2015 · 9 comments

Projects

None yet

1 participant

@diasdavid
Member

Aye mates!

This is your captain speaking, the IPFS Node.js ship as been to set sail and good winds are blowing! Thank you for all the messages and enthusiasm shared so far, we truly appreciate your support and since some of you are eager to get their hands on code and help with the development, I'll keep this issue updated every week with what has been going on and where you can help the most. To make sure you don't miss any updates, click the 'watch' button on the repo.

LOG - Week of 21st of September

Plan:

Where you can help during this week:

@RichardLitt RichardLitt referenced this issue in ipfs/pm Sep 23, 2015
Closed

Sprint Sep 21 #33

25 of 65 tasks complete
@diasdavid
Member

Aye! Check my new entry, arr :)

LOG - Week of 28th of September

Plan:

  • Have a interoperable version of libp2p in Go, to that the primitives and coverage become stronger so that then the remaining parts of IPFS can implemented with confidence (cli, bitswap, chunker, etc)

Where you can help during this week:

@diasdavid diasdavid changed the title from captain.log - IPFS Node.js implementation to captain.log - IPFS JavaScript implementation Dec 11, 2015
@diasdavid
Member

LOG - Week of December 21

Update:

Next:

Notes:

@diasdavid
Member

LOG - 26 of January

Update:

  • You can now do npm i -g jsipfs and use the already implemented commands, full list on project status https://github.com/ipfs/js-ipfs#project-status
  • We have an MerkleDAG implementation that interops with go-ipfs MerkleDAG implementation (this means you can write and read blocks from the same repo and will work for both implementations)
  • We have currently a WIP on browser testing, with only one obstacle from storing properly binary data in the browser
  • Lot's of good stuff

Next:

  • files add do the whole file/dir -> MerkleDAG thing, following the Data Importing spec ipfs/specs#57

Where you can help?

In lots of places, check https://github.com/ipfs/js-ipfs/issues for descriptions, there is a huge amount of things that were unblocked once IPFS Repo and MerkleDAG were implemented and that can be done in parallel :) Once we get this all done, we will focus again on libp2p to get the networking stack ready for prime time!

Thank you!

@diasdavid diasdavid referenced this issue in ipfs/pm Jan 26, 2016
Closed

Sprint January 25th #84

@diasdavid
Member

LOG - February 23 馃殻

We've been pretty busy working on the JavaScript implementation of IPFS. There are more things that you can use now and all of them interop with the Go implementation. Before starting with the formal update and next steps, I just wanted give a shout out to everyone that has been submitting patches or contributing with features to js-ipfs lately. Special thanks goes to:

Update

Extra - @RichardLitt has been putting an incredible amount of work in documenting the HTTP-API of IPFS, giving us a clear path on how to implement those routes correctly. You can find this endeavour in the http-api-spec repo

Goals

Our short-term goals for js-ipfs is to have complete feature parity with go-ipfs, with the exception of the DHT (now known as Peer Routing and Record Store). What this means is that if this goal is obtained, we will be able to:

  • swarm listen and connect between js-ipfs and go-ipfs nodes
  • use bitswap
  • run IPFS in the browser

This will mark a very significant milestone on the track of the js-ipfs development, enabling other developers to start building apps and services using the JavaScript libraries.

Peer Routing and Record Store are the stretch goals, since it would require go-ipfs to learn about IPLD and IPRS.

Good foundations

We've invested a significant portion of time in making tests from the beginning of the development of this project. Tests makes the codebase less brittle and help contributors feel confident that their patches and features don't break any other feature.

js-ipfs is divided at the macro level into 'ipfs-core', 'http-api' and 'cli'. It all starts with core: this is where all the functionality that IPFS is composed of is exposed through a standardized interface (WIP). Each feature that gets implemented must be accompanied by tests and these must run in Node.js and the Browser. In addition, each time one of these features manipulates blobs of data that exit the memory of the program (e.g. when they are written to disk), they must be validated and compared to the ones that are produced by go-ipfs, ensuring interoperability and parity (example)

Then, the http-api is built, which uses core and exposes its features through a RPC like HTTP API, specified by the http-api-spec. Tests are built by testing the http-api directly and by using the js-ipfs-api, a module that is up do date with go-ipfs implemented http-api.

Finally, the CLI should come. This should be tested by running commands while a daemon is both online and offline.

Let's look at an example, the jsipfs id feature

Where can you help

@diasdavid
Member
diasdavid commented Apr 5, 2016 edited

LOG - April 5 馃摉

We鈥檙e close!鈩. That is right, we are indeed pretty close to get a workable JavaScript implementation of IPFS that can be ran on the browser. I want to thank first, everyone that has been actively contributing to js-ipfs, with features, tests, documentation and experimentation! This participation is key to get the js-ipfs implementation out the door as soon as possible.

Update

We are now able to use Web Sockets with SPDY stream muxing, giving us the change to use js-ipfs on the browser and exchange blocks with any IPFS Node that is available through a public IP. Big ^5 to @dignifiedquire who fixed pako, the JavaScript implementation of zlib, the last mile to make this work completely.

@ngginever has been leading the charge on the files api front, jsipfs files get is almost in a ready state to be merged, powered by pifs-data-importing pipeline.

@noffle finished jsipfs init now works and made it fully compatible with go-ipfs. While at it, some documentation for it was also created as a bonus.

Overall js-ipfs keeps getting more and more complete. @xicombd lead most of the charges on getting the http-api endpoints and cli commands implemented, using the same IPFS core codebase.

We鈥檝e updated our ROADMAP to make it even more clear where we are and where he are headed.

Goal

We鈥檝e set up a goal of getting js-ipfs milestone 1 ready by the 25th of April. It is a ambitious and there might be some hiccups that delay this goal, but we feel confident as we are pretty close to be fulfil all the goals. Nevertheless, hitting milestone 1 is not just about getting something that works, it is about getting something that works and that people feel confident to use, that being said, one of the things we want to invest time in is on having good documentation of all the pieces that compose js-ipfs and test coverage.

Where you can help

There are a lot of things where everyone can help, I鈥檝e opened a issue for each of them with a task list, some focused on 鈥榝eatures鈥, others focused in 鈥榤odules鈥.

Here is the list of issues with what we are looking for to ship before milestone 1:

You will always be able to find in this repo, a collection of issues tagged with help wanted. This tag means that there is an action item that is currently not blocked by any other endeavour.

@diasdavid
Member
diasdavid commented Apr 22, 2016 edited

LOG - April 22 馃惓

Aye! This is a short captain.log entry to update everyone on the state of things. It is a short entry because we are really close of having js-ipfs ready and because of that, there is a set of very specific focus points we need to put our attention too. This entry enumerates those action items.

What's left

files API

  • get unixfs-engine fully operational in the browser. Currently there is a bug caused by changes on the readable-stream impl - ipfs/js-ipfs-unixfs-engine#31. @dignifiedquire is handling this
  • finish add, get and catcommands on `jsipfs cli. This includes through daemon (multipart messages) and directly calling core. @nginnever is tackling these. Follow on #60

libp2p

bitswap

  • 1st part: Use bitswap to fetch blocks (able to request and cancel with the message queue + state machine architecture discussed)
  • 2nd part: Serve blocks

module quality

We are mostly done with our module quality improvements (code coverage, readme, etc), down to 5 modules, namely:

  • js-libp2p-ipfs #125
  • js-libp2p-tcp #123
  • js-libp2p-swarm #122
  • js-libp2p-unixfs-engine #112
  • js-ipfs-api

@noffle is helping getting these done. If you want to help, jump in into one of those issues as contribute with one of the items that is left, feel also welcome to post questions :)

core api

@diasdavid
Member
diasdavid commented May 10, 2016 edited

LOG - May 10 馃殼

Arr! This is the last mile Captain.log for js-ipfs milestone 1. It contains the list of things that are missing in order to js-ipfs usable!

What's left

files API

libp2p

bitswap

  • Merge #195
  • browser tests
  • intensive tests (bitswap swarm between several browser and Node.js nodes)
  • interop tests

module quality

We are mostly done with our module quality improvements (code coverage, readme, etc), down to 5 modules, namely:

  • js-ipfs-api

core api

  • converge both js-ipfs-api and js-ipfs interfaces so that modules are interchangeable
  • core api spec ipfs/specs#85
    • object
@diasdavid
Member
diasdavid commented Jun 14, 2016 edited

LOG - June 14 鉀

Aye mates! This log entry is full of exciting updates, writing it made me really happy. The IPFS community has worked pretty hard and there is a lot to talk about and a lot of people talking and checking IPFS, and that is amazing!

What鈥檚 new in town

We now have two apps working on js-ipfs, in the browser, completely written in JavaScript! These are:

  • Orbit
  • Paperhub

Orbit

http://orbit.libp2p.io/

Distributed Chat Application using CRDT

Code at: https://github.com/haadcode/orbit

Paperhub

http://paperhub.libp2p.io/

Collaboration tool for distribution of Open Access Scientific material.

Code at: https://github.com/ipfs/paperhub

Both of these apps use orbit-db, a DB for P2P networks that uses a CRDT to synchronise the state between several peers. You can find [here] a fun demo and visualisation.

http://celebdil.benet.ai:8080/ipfs/Qmezm7g8mBpWyuPk6D84CNcfLKJwU6mpXuEN5GJZNkX3XK/

Huge thank you and lot鈥檚 of <3 <3 <3 to everyone that has been contributing so much to the project.

News

The IPFS project was present at the Decentralised Web Summit, hosted by the Internet Archive. Expect a lot of blog posts from us soon, meanwhile, you can check the lighting talks at https://archive.org/details/DecentralizedWeb20160609pt1

What鈥檚 next

There is still a lot of fun things to tackle, specially when it comes to interop between go-ipfs and js-ipfs, me and @whyrusleeping have been inspecting the wire protocols and found some differences in the implementations, most of them have been fixed, missing now only the Identify one. You can find the list below of things that are on the plan for this month and where you can provide most help too.

go and js interop (@diasdavid)

libp2p Transports + Connection (@diasdavid)

libp2p UX

  • libp2p
    • js-libp2p base, that libp2p-ipfs and libp2p-browser can Inherit, avoiding code duplication (@nginnever)
    • define interface-libp2p

libp2p Features

Testing

  • Upgrade the tests scripts to be self contained . This will improve iteration speed by saving a lot of time on running all the tests every single time. Optimally, they even should be runnable with node /test/test.js (not needing mocha directly)

CLI (@RichardLitt)

  • Improve CLI
    • detect incoherences with go-ipfs CLI

js-ipfs

Documentation

  • Better examples on how to use libp2p
  • Examples on how to use js-ipfs
  • libp2p spec upgrade - libp2p spec needs a really good update

General bug fixing

Peer endeavours

Stretch goals

@diasdavid diasdavid referenced this issue in ipfs/pm Jun 20, 2016
Closed

Sprint: June 20 鈫 July 3 #114

@diasdavid diasdavid self-assigned this Jul 27, 2016
@diasdavid
Member

LOG - August 13

image

Aye mates! Here is another exciting update on the js-ipfs project. This one contains: a new and extended Roadmap for the project; some really exciting feature updates; a description of where our attention is and a plug to our new project tracking process and as usual, a up to date list of items that you can tackle and contribute to js-ipfs!

馃尙 Release the sails! We've found a (road)map

The IPFS community has spent some quality time in July together (with in person + remote attendance) at the IPFS Workshop to come up with a plan for what are the new and exciting things coming up to the IPFS ecosystem. From those discussions, a new shiny roadmap emerged with very focused milestones, so that we can complete and make js-ipfs better incrementally, while keeping the focus in a subset of tasks at a time and ensure those are completed excellently.

You can find the Full Roadmap at this project repo, however, here is a tl;dr; for your reading pleasure:

馃専 Moar exciting news!

  • Moving to pull-streams - The 馃毑馃彔 was strong in this one, but we've finally made a call on the Control Flow library to use in order to solve the hiccups we have had in the past with Node.js Streams. This discussion took a lot of input from several people familiar with the js-ipfs project and the contenters for the Control Flow Library. You can read the whole thread at: #362
  • Automated Changelogs - We are getting better changelogs on release that are automatically generated. We've finalized our commit message convention, here, so that they easily parsable and human understandable. The issue to track the development of the tooling required to add this feature to our release process is here
  • On top of the changelog, we also created a channel to announce and propose changes to the js-ipfs interface. If you want to be notified about these, please subscribe to this issue
  • On the topic of js-ipfs interface, we've updated the files.add API to become more Developer friendly. Read more in this issue
  • Our js-ipfs got an upgrade. We moved from ronin to yargs, a more supported and used CLI toolkit that enables us to have menus for the js-ipfs CLI just like go-ipfs has.
  • We've started optimizing js-ipfs and talking about future optimizations, to make the library smaller and more performant - Full thread here

鈿★笍 Current focus

Our current focus is now identified by the Milestone where the team is currently hacking on (Milestone 2). In addition to the ROADMAP.md, we've created a waffle board for the js-ipfs project, so that we (including you) can easily identify easily which issues are related with a certain Milestone.

Important to note that this new Project Management Process is still being finalised, you can follow it on the PM repo.

馃挅 Where you can help

As always, issues that are tagged with 'help wanted' are open for contribution and that label means that they are unblocked, plus they are typically tagged with a difficulty level (let us know if we missed one).

Check our waffle board for a complete list across projects: https://waffle.io/ipfs/js-ipfs

If you are new to the project and want to get a better feeling of how are things, an always good way is to join our Monday hangouts. You can find all the details on the PM repo.

馃摪 Other things that might be of your interest

  • ipfsd-ctl is being revamped to use bin-wrapper and with it, give a facelift to the documentation. A lot of developers (ourselves included) use this module to spawn go-ipfs instances, it is super handy for testing and electron apps and now it is getting better. Thank you @JGAntunes for leading that!
  • Now libp2p is a official standalone project with its own organization, you can find all of its modules on the libp2p organization
  • Our self described formats have been promoted and now are standalone endeavour, in the process of standardization. See multiformats for more info.
  • PubSub Initiative - We've kicked off officially the endeavour to bring a PubSub system for IPFS. A lot of people in the IPFS community have joined to make this happen, to watch or join this effort, see: https://github.com/libp2p/pubsub
  • IPLD is coming, closer than ever - A new proposal for Content Identifiers (CID) has been made and with that a migration plan for IPFS to support both MerkleDAG and IPLD object (+ any other data structure), this is super exciting, find it all at: ipfs/specs#130
  • go-ipfs 0.4.3 Release Candidate is out, read all about it on its Captain.log - ipfs/go-ipfs#2247 (comment)

Thank you!

Thank you for reading this entry of the captain.log and a bigger thank you to everyone that has been contributing, either through code, bug fixing, bug reporting, ideas, reviews and so on, you are awesome!!!

@diasdavid diasdavid changed the title from captain.log - IPFS JavaScript implementation to 馃殻 captain.log - IPFS JavaScript implementation Aug 15, 2016
@diasdavid diasdavid changed the title from 馃殻 captain.log - IPFS JavaScript implementation to 馃殻 captain.log - IPFS JavaScript implementation 馃専 Aug 20, 2016
@diasdavid diasdavid referenced this issue in libp2p/js-libp2p-railing Sep 7, 2016
Closed

I'm not sure where I can start from #4

@diasdavid diasdavid added js-ipfs-ready and removed in progress labels Nov 22, 2016
@diasdavid diasdavid added backlog and removed js-ipfs-ready labels Dec 5, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment