Skip to content
This repository has been archived by the owner on Feb 8, 2023. It is now read-only.

Sprint: March 7-20 #97

Closed
RichardLitt opened this issue Mar 6, 2016 · 29 comments
Closed

Sprint: March 7-20 #97

RichardLitt opened this issue Mar 6, 2016 · 29 comments

Comments

@RichardLitt
Copy link
Member

Sprint March 7th -- 20th

Sprint Goals

Sprint Discussions

Schedule

Please take notes in a separate pad, if you can, and link it here.

Endeavour Lead Time (PDT - UTC/Z - CET) Pad
sync @RichardLitt 9:00pDT 17:00Z 18:00CET IRC: #ipfs on Freenode
apps on ipfs @dignifiedquire 10:30PDT 18:30Z 19:30CET https://public.etherpad-mozilla.org/p/ipfs-mar-7-apps-on-ipfs
infrastructure @lgierth 11:00PDT 19:00Z 20:00CET https://public.etherpad-mozilla.org/p/ipfs-mar-7-infrastructure
libp2p @diasdavid 11:30PDT 19:30Z 20:30CET https://public.etherpad-mozilla.org/p/ipfs-mar-7-libp2p
js-ipfs @diasdavid 12:00PDT 20:00Z 21:00CET https://public.etherpad-mozilla.org/p/ipfs-mar-7-js-ipfs
go-ipfs @whyrusleeping 12:30PDT 20:30Z 21:30CET https://public.etherpad-mozilla.org/p/ipfs-mar-7-go-ipfs

Please add the Agenda to the Pad before the endeavour sprint starts.

Sprint Deliverables

  • Add your tasks below in a comment, that way we only have people listed who are really in the sprint
  • Add links to issues down here. Only add things you can finish this sprint.
@dignifiedquire
Copy link
Member

Apps on IPFS

Lead: @dignifiequire

Participants

Agenda

  • Webui
  • dignified hacks (on the webui)
  • Station @ RaspberryPi
  • orbitDB

Notes

@daviddias
Copy link
Member

Library Peer to Peer!

Lead: David Dias - @diasdavid

Participants

Agenda

Update

  • Redo a lot of old code to new patterns
  • Lot's of new improvements, e.g. less stream wrapping
  • APIs are changing a bit, less magic

Next

@daviddias
Copy link
Member

IPFS JavaScript Implementation

Lead: David Dias - @diasdavid

Participants

Agenda

Update

  • Foundations
  • Contributions
  • data-importing (voxelot)

Next

  • jsipfs swarm (dependent on libp2p stuff)
  • complete jsipfs files
  • bitswap

@ghost
Copy link

ghost commented Mar 7, 2016

Infrastructure

Lead: @lgierth

Participants

Agenda

  • multireq
  • metrics
  • fetching & pinning

Notes

  • jbenet still seeing glitches with multireq
    • nginx metrics will hopefully shed some light
  • more metrics from go-ipfs!
    • number of peers by transport, bitswap wants/provs/bw, etc.
  • pinning performance
    • pinbot is not the issue
    • multireq neither
    • likely it's the fetching
  • fetch performance
    • some merkledags with depth>1 get stuck
    • it continues only after a restart, then gets stuck again later
    • reproducible with geoip
    • also reproducible with a certain gx import set
    • gets stuck on the same hash reproducibly
    • whyrusleeping is on it

@daviddias
Copy link
Member

DAVID DIAS TODO

js-ipfs

js-ipfs-api

libp2p

npm on IPFS

IPLD

  • IPLD hangout

webui

  • test and review Webui for 0.4.0 release

specs

extra

@haadcode
Copy link
Member

haadcode commented Mar 8, 2016

@haadcode sprint goals - 7-20 Mar

Summary

I'll be working on integrating orbit-db (1) to Orbit (2) in order to make sure orbit-db provides an easy embeddable p2p database for apps on IPFS. It also means I can refactor a huge bunch of the Orbit code and simplify the "backend" logic of the app as well as refactor the UI to a more manageable state. The overall goal is to make Orbit a little cleaner so that others can start contributing to is.

  1. https://github.com/haadcode/orbit-db
  2. https://github.com/haadcode/anonymous-networks

Tasks

Orbit
  • Integrate with orbit-db
  • Refactor everything
  • Make a stable build/release after the first two
  • Candy: App icon for Orbit
Pubsub

@dignifiedquire
Copy link
Member

@dignifiedquires TODO

  • Webui
    • Work on the files screen
    • Review and merge config PR
  • Station
    • Use latest ipfs-geoip + failure handling
  • WebRTC
    • Reading and research on the current landscape of implementations
    • Spec reading
  • Randor
    • Add nested files api operations
    • Fix parallel bug with pin-rm

@RichardLitt
Copy link
Member Author

Summary

I am going to focus on copy editing some of our blog posts that need to go out, because we should publish them and I think a little elbow grease will get them out the door. I will also be diving into js-ipfs and seeing if I can help out with building ipfs block there. Along with these I am going to be generally working through my backlog of issues and things to do.

Tasks

@hackergrrl
Copy link
Contributor

@noffle Sprint Goals

Summary

I want to make headway on the "init" process for js-ipfs: this means getting a clear understanding of what initializing a repo means with respect to IPFS, resolving any non-existant dependencies, and implementing the initialization logic. I don't have a clear view yet on the full dependency graph for this, so I'll try and update this list as the amount of work involved becomes clearer.

I'd also like to build hyperarray: an ordered list CRDT built on a hyperlog. This will form the underlying document model for an ipfs-pad document. I've been largely inspired by r-array's design and API.

Tasks

js-ipfs

  • spec document of 'ipfs init' process (started here)
  • outline work for 'repo init'
  • outline work for 'ipfs init' for js-ipfs

ipfs-pad

  • hyperarray
  • start an Architecture document

pm

  • update OKR document with my goals for the rest of Q1

@nginnever
Copy link
Member

Summary

I want to continue to make progress with my understanding of IPFS. I feel I am at around 70% knowledge and this is keeping me from awnsering some of the bigger questions about data importing like what structures should a browser be able to use in lack of an fs.. yet should we use unixfs and such. I will be moving towards more defined tasks from the go implementation like building out more commands. Also starting to take a look at libp2p and networking. And spending some time with react and the webui to keep my application development chops in line.

P.S. Also want to work with those geniuses up there on pub/sub and CRDT systems

TODO

- js-ipfs - [~] command cat (meow) - [~] command object get - js-ipfs-data-importing (js-ipfs issue #41) - [ ] streamImporter (more questions) - [X] BufferImporter (CR) - [~] export function - [X] browser / mocha tests - js-ipfs-merkle-dag - [X] web tests (needs https://github.com/ipfs/js-ipfs-repo/pull/29) - Webui - [ ] write some tests for new components - [~] document in contibuting.md to help others get involved

@ghost
Copy link

ghost commented Mar 11, 2016

@lgierth todo

@jbenet
Copy link
Member

jbenet commented Mar 12, 2016

@nicola and I will be having a hangout about IPLD this monday 3/14. interested researchers feel free to join. time TBD, probably after the general hangout. (@RichardLitt if you can assign a time that would be great) cc @mildred if interested

@nicola
Copy link
Member

nicola commented Mar 12, 2016

@rychipman will join the IPLD meeting ^

@RichardLitt
Copy link
Member Author

Set up for 3pm Eastern Time (Should be 8 CET, 7GMT) on Monday. In the calendar.

@RichardLitt
Copy link
Member Author

Note: tomorrow is a mid-fortnight sprint. We will have a Google Hangout at 1pm EST, 6PM CET. Link will be posted in IRC.

@nicola
Copy link
Member

nicola commented Mar 14, 2016

@RichardLitt @jbenet could we shift the meeting on IPLD just about 30 minutes? (@rychipman can only make then)

@mildred
Copy link

mildred commented Mar 14, 2016

I will join the IPLD meeting. Is it still 7pm UTC?

@RichardLitt
Copy link
Member Author

@nicola Sure! Which way - forwards or backwards?

@nicola
Copy link
Member

nicola commented Mar 14, 2016

@RichardLitt 3:30 EST :)

@RichardLitt
Copy link
Member Author

Done. Note: daylight savings affected the states. That mean sprint hangout will follow the calendar: 1pm EST, 10am PST, 5pm GMT, and 6pm CET.

@RichardLitt
Copy link
Member Author

IPFS Sprint Mid-week hangout Notes

@diasdavid

Focus on Libp2p, interop with go-p2p. Work on libp2p swarm. The api has completely changed. Before, I was exposing a simple API like dial, and then internally it would figure out which transport to use. Now, instead, we expose these internal steps as different function calls. You still have the nice high-level API to the other peer, but there's a lot more stuff exposed. There's more tests, too, making swarm less brittle. So on that, you now have a swarm version ready. What I'm working now is getting the websocket transport to the new transport scheme, which resulted from this upgrade on p2p swarm.

  • Francisco has been shipping a lot by tackling issues which weren't tagged. Implementing HTTP API endpoints and the CLI commands which work with those to call js-ipfs core functions. Huge props.
  • David Dias has had interesting conversations on swarm because when you have to deal with several transports at the same time, and several collections that get multiplexed over other connections, and then you have several parts of the code that will use this multiplex connection to open streams using different transports, and you want to abstract most of these things away and make it really simple, you get some problems. For example, if some user is transferring a huge file over IPFS, and decides to close his laptop, all of his connections will drop, and if those collections are multiplexed over a single connection, what is going to happen is that all of those points in the IPFS code which were transferring blocks will try to reestablish a collection. It'll call swarm again, asking it to dial to this other peer. Because these parts - for example, bitswap - don't need to understand that they are transferring across of the connection, they just need to know that they have a stream that they can use, it means that several parts of the code might try dialing the same peer at the same time. This is suboptimal. You get all of these streams which were multiplexed popping up errors because now they are on a single p2p connection. I am trying to come up with an elegant way of handling these errors. If too many peers dial, it becomes a racing condition (which peer dials first, etc), which is inefficient. So, I've opened an issue, got some feedback. And right now, I am getting more confident that a solution to this problem is a state machine, or a mutex strategy

https://github.com/diasdavid/js-libp2p-swarm/issues/24

This week

Go-p2p and js-p2p interop. Getting that done. If that goes well, he'll go into Bitswap. I might build bitswap on js-land, just to have some example code.

@dignifiedquire

@dignifiedquire has worked on the webui. He's got peers working, and all of the geolocation data is now fetched all through IPFS again. Mostly, he's been working on the files API. You can delete stuff, create files and folders.

Preview works: You can view text files, images, and videos. Anything that the browser could play for you, you can use in the webui.

This week

Getting the webui polished and finished. Also, fixing config. It was PRed by @sucha (TODO), who also worked on tests, and then Friedel broke it a bit.

@haad

Haads been working on Orbit and Orbit-db. Got it integrated into IPFS, so that's all working quite nicely on both sides. The turn around times are fairly good. It works, it loads.

There's a lot of bugs and things that need to be worked on. Thanks @noffle for bouncing ideas and giving feedback.

This week

Work on orbit; test @noffle's prototype and give feedback. No blockers.

@lgierth

Been sick recently, but everything that he's started is in progress. Not really finished yet, but getting there. Friday he's flyin go to New York.

Week in three words

Multireq, metrics, pages.

@nginnever

Lots of browser testing this week, for js-ipfs data importing and for js-merkledag. Everything is ok, except for one issue of not being able to find a function, although it works on @dignifiedquire's machine, but @diasdavid should be able to help. Also, writing tests for containers for the webui.

Next Week

Nat testing.

@noffle

Focused on ipfs-init working with js-ipfs. @noffle wrote a spec-like document showing how that works. He's also been thinking a lot of about the structure of distributed apps on top of IPFS. So, he's making a quick prototype of a distributed IPFS-powered wiki. If you're a reader, you just look through a gateway; if you're a publisher, you just publish a new root hash of the wiki. Might work well for a lot of applications. Inspired after playing with zeronet.

This Week

Finish this prototype, possibly work on the IPFS-Etherpad prototype.

@RichardLitt

PRed some things - went over all of the PRs in http-api-spec. Revieweed 0.4.0 blogpost and I need to do that again. Published the weekly. Started work on ipfs-block; also perused all of /faq and /support. Also hung out with @jbenet.

This week

Going to keep work on ipfs-block.

@whyrusleeping

Fixed a bug where IPFS would hang for large files; it was a multiplexer bug, in the end. There's a bug in Yamux which hasn't been fixed, where if you have too many in flight open streams, the connection will hang. If you raise that limit, the open ones will close down. The limit has been raised to around 16k, and merged that in, but there's still more to do there

This week

Ship 0.4.0. @RichardLitt needs to review stuff; Everyone to review the changelog. Other than that, should be pretty silent. Distributions to be updated, too.

@daviddias
Copy link
Member

DAVID DIAS CHECK IN

following the summary check in format. For a complete task list done during this sprint, check #97 (comment).

This last sprint I had the chance to give a lot of time to js-ipfs and libp2p, with the goal of getting js-ipfs working on the browser and exchanging blocks with other js-ipfs nodes and go-ipfs nodes. I've made a lot of contributions and updates to: js-libp2p-ipfs, js-ipfs-data-importing, js-multiaddr, js-mafmt, js-peer-id, js-ipfs-repo, js-libp2p-swarm, js-libp2p-tcp, js-libp2p-utp, js-libp2p-websockets, js-libp2p-multiplex. These changes bubbled up to js-ipfs.

I've found some (really nasty) rabbit wholes when using spdy over websockets which I will explain more at the libp2p hangout, which made a good hit at my time planning.

Also had a good time hacking with @xicombd, good discussions and remote hacking/debugging with @nginnever, good conversations with @dignifiedquire and @nginnever which resulted in the understanding of the new IPFS JavaScript Project guidelines, and last but not the least, the IPLD hangout with a bunch of IPLD enthusiasts.

I've also got the chance to write and submit the paper for ICWE and organize and host the first Research & Development meetup in Lisbon.

@RichardLitt
Copy link
Member Author

Summary

I feel my work has been a bit slow the past two weeks. I worked on the http-api-spec, some more, redoing ipfs object and opening a lot of PRs on go-ipfs. I also went through go-ipfs and added defaults to all of the options, which I think will help out in the future, and am getting a bit more comfortable with reading go. I've sent out two weeklies, and spent a lot of time helping organize the IPFS trip to New York for next week, and I spent a good amount of time going through ipfs/faq and ipfs/support checking up on all of the old issues which weren't updated. I did not get to help review the 0.4.0 ship, nor did I do significant work on js-ipfs block, which I am unhappy about. Hoping to get more work done this coming sprint, especially with the in-person coding time ahead.

Tasks

Extra

@dignifiedquire
Copy link
Member

@dignifiedquire Update

My main focus this time was the webui. I have made some good progress on the files explorer interface, refactoring some code and writing more tests. I'm now working on cleaning up the UI, preparing mockups and plans for the final phase of the implementation.

After a some discussions and issues with our JavaScript setups I've started to create dignified.js which is going to be the module to help us have a unified testing, linting and building solution in all our JavaScript projects. This is an ongoing effort that I'm working on together with @diasdavid and @nginnever.

Also in addition after the great IPLD hangout last week I've started implementing parts of it in rust with the goal of finding a typesafe and general interface on how to define mappings between raw IPLD and higher level abstractions like unix-fs.

@ghost
Copy link

ghost commented Mar 21, 2016

@lgierth sprint update

The past few weeks Berlin has been a huge drag on me, tons of org stuff to take care of, and was generally very easily distracted. My task list is testament to that :) I made a bit of progress on everything, but wasn't able to focus on any particular task very well. I feel quite energized though now that I'm in NYC, so I am to finish all of last sprint's stuff, and maybe a thing or two more.

My focus is still: v03x/v04x multireq stability, metrics, pages deployment, and cleaning up the mess which is the infrastructure repo.

We still seem to have subtle issues with v03x/v04x and multireq, which I haven't quite tracked down yet. One of them is timeouts, for which we have UptimeBuddy and Pingdom alerts in place now, as well as nginx-level metrics which are about to be ready. The other is performance/latency. I've taken the saturn host out of ipfs.io DNS for now, it's in DigitalOcean's Singapore datacenter which doesn't have the best connectivity it seems.

  • infrastructure
    • parse metrics from nginx logs (using mtail)
    • deploy mtail-nginx to all hosts
    • identity validation for *.ipfs.io cert taken over by @whyrusleeping
    • struggle with gc on saturn host
    • fix gc_capacity on gateway hosts
    • disable gc on storage hosts
    • migrate from docker to runc+runit
    • write down docker issues
  • fc00/cjdns
  • pages / dnslink
    • figure out the how -- similar to sshcommand
    • ssh blog@pages.i.ipfs.io publish <hash>
  • go-ipfs
    • bring back ipfs_* metrics
    • harden gateway path prefix
    • fix --enable-gc in docker container
  • errands
  • reminder to self
  • org stuff

@nginnever
Copy link
Member

@nginnever Update

This past week I continued to work on web testing the data-importing and merkle-dag modules. This kicked off conversation about new ways to handle our tests so that in the future things will be more consistent. Having this taken care of is allowing me to make much more progress now on jsipfs core.

I also began working on an export function for data-importing on a branch that will unMarshal our hashes and begin to pipe together streams of dag linked data for our files get and files cat commands.

Worked a bit with the webui but still need to study that type of react structure more to feel useful.

@hackergrrl
Copy link
Contributor

Summary

Lots of reading and a few PRs this week some libp2p modules -- mostly usability improvements. Big accomplishment of the week is that the majority of the ipfs initialization logic for js-ipfs is now written.

Several of us (haad, jbenet, whyrusleeping, nginnever) discussed approaches to pubsub in more detail, and I did some further background reading on previous discussions across ipfs/* repos.

Last weekend I wrote a prototype wiki powered by IPFS. A few pieces are written so far, but the overall goal is to gain a better understanding of the functionality still missing in IPFS to make apps like these relatively simple.

Tasks

@haadcode
Copy link
Member

@haadcode Sprint End Summary

I integrated orbit-db into Orbit and it's working nicely. Sending messages feels instantenous and responsive. With that, the "backend" code of Orbit is now a lot smaller and manageable. I was able to refactor the UI code of Orbit a little along the way but it still needs quite a bit of work. Orbit now has a nice new app icon for the desktop app and as a favicon in the browser. I refactored orbit-db a little by taking out the append-only log functionality into its own module (https://github.com/haadcode/ipfs-log) so that it can be used more easily as a building block in other apps, or eg. as a transport data structure for CRDTs.

Highlights

Tasks

Orbit
  • Integrate with orbit-db
  • Refactor everything
  • Make a stable build/release after the first two
  • Candy: App icon for Orbit
Pubsub
Extra

@whyrusleeping
Copy link
Member

Spent most of my effort on getting 0.4.0 ready to ship. This involved lots of testing work, and also a lot of organizational stuff.
I also did a large amount of performance analysis and have good ideas for improving perf overall moving forward.
My list of weekly accomplishments feels short, many of these are rather large items.

  • fixed double transfer encoding header
  • worked on 0.4.0 blog post
  • debugged gx on windows issues
  • wrote a large stress test for migrations
    • ~200k files, 2000 pins
  • worked on benchmarks for various ipfs operations (expect perf changes in the near future)
  • minor bugfix in go-datastore (prevents a panic)

@jbenet jbenet removed the sprint label Mar 21, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@mildred @jbenet @hackergrrl @dignifiedquire @RichardLitt @daviddias @whyrusleeping @nicola @nginnever @haadcode