Sprint Dec 14 #74

Closed
RichardLitt opened this Issue Dec 14, 2015 · 20 comments

Comments

Projects
None yet
9 participants
@RichardLitt
Member

RichardLitt commented Dec 14, 2015

Sprint Dec 14

  • Completed IPFS API Spec (core + transports) by Jan 1

Sprint Goals

Sprint Discussions

Schedule

Please take notes in a separate pad, if you can, and link it here.

Endeavor Lead Time (PDT - UTC/Z - CET) Pad
Apps on IPFS Juan Benet 10:30PDT 18:30Z 19:30CET https://public.etherpad-mozilla.org/p/ipfs-dec-14-apps-on-ipfs
infrastructure Lars Gierth 11:00PDT 19:00Z 20:00CET https://public.etherpad-mozilla.org/p/ipfs-dec-14-infrastructure
libp2p David Dias 11:30PDT 19:30Z 20:30CET https://public.etherpad-mozilla.org/p/ipfs-dec-14-libp2p
js-ipfs David Dias 12:00PDT 20:00Z 21:00CET https://public.etherpad-mozilla.org/p/ipfs-dec-14-js-ipfs
go-ipfs Jeromy Johnson 12:30PDT 20:30Z 21:30CET https://public.etherpad-mozilla.org/p/ipfs-dec-14-go-ipfs

Please add the Agenda to the Pad before the endeavor sprint starts.

Sprint Deliverables

  • Add your tasks below in a comment, that way we only have people listed who are really in the sprint
  • Add links to issues down here. Only add things you can finish this sprint.
@lgierth

This comment has been minimized.

Show comment
Hide comment
@lgierth

lgierth Dec 14, 2015

Member

infrastructure

Lead: @lgierth

Agenda

  • dev040 deployment
    • update strategy, also for the future
    • v04x.ipfs.io / v03x.ipfs.io
  • storage
    • biham and nihal
    • castor and sirius
  • deploying site updates automatically in a consistent way via IPFS or IPNS (project repos, website, maybe blog?)

Participants

Notes

  • dev040
    • open issue to come up with upgrade plan
    • include certain ipfs users who are embedding ipfs in their software and might have trouble updating
    • one option: have the multistream handshake be backwards-compatible
    • another option: break compat, but do so with a plan of how to do it better on both our end and their end
  • storage
    • nihal for internal use
    • biham "free for all"
    • copying castor's and sirius' data to biham, then shutting them down
    • daviddias can resume NPM on biham once the copying is finished
  • storage at jbenet's place
    • the host formerly known as earth will be resurrected in the coming days
    • will serve as OSX build machine
    • hands-on storage
  • deploying site updates automatically
    • spin up server which receives github hooks, builds sites, publishes to IPNS
    • start with project-repos (aka ci-status)
Member

lgierth commented Dec 14, 2015

infrastructure

Lead: @lgierth

Agenda

  • dev040 deployment
    • update strategy, also for the future
    • v04x.ipfs.io / v03x.ipfs.io
  • storage
    • biham and nihal
    • castor and sirius
  • deploying site updates automatically in a consistent way via IPFS or IPNS (project repos, website, maybe blog?)

Participants

Notes

  • dev040
    • open issue to come up with upgrade plan
    • include certain ipfs users who are embedding ipfs in their software and might have trouble updating
    • one option: have the multistream handshake be backwards-compatible
    • another option: break compat, but do so with a plan of how to do it better on both our end and their end
  • storage
    • nihal for internal use
    • biham "free for all"
    • copying castor's and sirius' data to biham, then shutting them down
    • daviddias can resume NPM on biham once the copying is finished
  • storage at jbenet's place
    • the host formerly known as earth will be resurrected in the coming days
    • will serve as OSX build machine
    • hands-on storage
  • deploying site updates automatically
    • spin up server which receives github hooks, builds sites, publishes to IPNS
    • start with project-repos (aka ci-status)
@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Dec 14, 2015

Member

Apps on IPFS

Lead: @dignifiedquire + @jbenet

Agenda

  • registry-mirror
  • have a node with full npm
  • make experience of getting it ready really nice
  • release 0.4.0 on go builder with latest bell and whistles
  • migration ipfs/infrastructure#134
  • ipfs/distributions
  • IPNS + CoreNetwork

You have 30 minutes for this agenda, 5 minutes before the meeting ends, consider moving the remaining items to a github discussion thread so that all the other sprint meetings can start in time.

Participants

Notes

registry-mirror

  • try the new 0.4.0 with Rabin again in Jeromy Machine (check with Jeromy)
  • use one of the 17TB nodes to be the main one for npm on IPFS
  • try with tar importer
  • improve installation process of 0.4.0 (check with Jeromy)

ipfs/distributions Update

  • design draft approved last week
  • in the middle of implmentation
  • Goal: Get it ready before 0.4.0

IPNS + Core Network

  • Core Network is using IPNS to publish/consume social network style streams
  • IPNS speed concerns
  • PubSub may fix this, either by sending updates directly, or pushing IPNS updates over PubSub
Member

diasdavid commented Dec 14, 2015

Apps on IPFS

Lead: @dignifiedquire + @jbenet

Agenda

  • registry-mirror
  • have a node with full npm
  • make experience of getting it ready really nice
  • release 0.4.0 on go builder with latest bell and whistles
  • migration ipfs/infrastructure#134
  • ipfs/distributions
  • IPNS + CoreNetwork

You have 30 minutes for this agenda, 5 minutes before the meeting ends, consider moving the remaining items to a github discussion thread so that all the other sprint meetings can start in time.

Participants

Notes

registry-mirror

  • try the new 0.4.0 with Rabin again in Jeromy Machine (check with Jeromy)
  • use one of the 17TB nodes to be the main one for npm on IPFS
  • try with tar importer
  • improve installation process of 0.4.0 (check with Jeromy)

ipfs/distributions Update

  • design draft approved last week
  • in the middle of implmentation
  • Goal: Get it ready before 0.4.0

IPNS + Core Network

  • Core Network is using IPNS to publish/consume social network style streams
  • IPNS speed concerns
  • PubSub may fix this, either by sending updates directly, or pushing IPNS updates over PubSub
@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Dec 14, 2015

Member

libp2p

Lead: @diasdavid

Agenda

  • factoring go-libp2p
  • swarm Interop
    • secio in JavaScript
  • Are we going TLS?
  • python libp2p
  • rust libp2p
  • update roadmap if needed #72

You have 30 minutes for this agenda, 5 minutes before the meeting ends, consider moving the remaining items to a github discussion thread so that all the other sprint meetings can start in time.

Participants

Notes

  • crypto channels
    • multistream enables us to have several protocols already
    • implement TLS 1.3 (check if there are implementations that work in JS) as a upgrade transport
    • look into curveCP
    • skip secio in JS land for now
    • list all of the interfaces in libp2p for Juan to bikeshedding
Member

diasdavid commented Dec 14, 2015

libp2p

Lead: @diasdavid

Agenda

  • factoring go-libp2p
  • swarm Interop
    • secio in JavaScript
  • Are we going TLS?
  • python libp2p
  • rust libp2p
  • update roadmap if needed #72

You have 30 minutes for this agenda, 5 minutes before the meeting ends, consider moving the remaining items to a github discussion thread so that all the other sprint meetings can start in time.

Participants

Notes

  • crypto channels
    • multistream enables us to have several protocols already
    • implement TLS 1.3 (check if there are implementations that work in JS) as a upgrade transport
    • look into curveCP
    • skip secio in JS land for now
    • list all of the interfaces in libp2p for Juan to bikeshedding
@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Dec 14, 2015

Member

js-ipfs

Lead: @diasdavid

Agenda

You have 30 minutes for this agenda, 5 minutes before the meeting ends, consider moving the remaining items to a github discussion thread so that all the other sprint meetings can start in time.

Participants

Notes

  • js-ipfs-repo
    • datastore
    • dir
      • levelDB -> /datastore
      • flatfs -> /blocks
      • might move to arenaStore in the long term
    • itemzise what is on leveldb-datastore (@jbenet)
      • DHT (provider records)
  • jsipfs
  • IPLD Data Importing
  • Bitswap
    • dag path notation - ipfs/notes#12
    • spec interface (upgrade with dag path notation) (ion is working this)
    • right now JS implementation can use the same 'dumb' approach
Member

diasdavid commented Dec 14, 2015

js-ipfs

Lead: @diasdavid

Agenda

You have 30 minutes for this agenda, 5 minutes before the meeting ends, consider moving the remaining items to a github discussion thread so that all the other sprint meetings can start in time.

Participants

Notes

  • js-ipfs-repo
    • datastore
    • dir
      • levelDB -> /datastore
      • flatfs -> /blocks
      • might move to arenaStore in the long term
    • itemzise what is on leveldb-datastore (@jbenet)
      • DHT (provider records)
  • jsipfs
  • IPLD Data Importing
  • Bitswap
    • dag path notation - ipfs/notes#12
    • spec interface (upgrade with dag path notation) (ion is working this)
    • right now JS implementation can use the same 'dumb' approach
@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Dec 14, 2015

Member

go-ipfs

Lead: @whyrusleeping (@jbenet subbing)

Agenda

  • ship distributions
  • ipfs-update testing
  • fs-repo-migration testing
  • 0.4.0 things
  • upgrade path and compatibility

Participants

Notes

Questions:

  • what's the 0.4.0 upgrade path? we'll ask users who care.
  • what's in leveldb? DHT Records mostly.
  • we could implement a kv-store (like pin set) on top of ipfs objects
    (basically ipfs-blob-store) and transition everything in one go.
  • DHT Records will be ipfs objects eventually anyway, under iprs.

TODOS:

Member

jbenet commented Dec 14, 2015

go-ipfs

Lead: @whyrusleeping (@jbenet subbing)

Agenda

  • ship distributions
  • ipfs-update testing
  • fs-repo-migration testing
  • 0.4.0 things
  • upgrade path and compatibility

Participants

Notes

Questions:

  • what's the 0.4.0 upgrade path? we'll ask users who care.
  • what's in leveldb? DHT Records mostly.
  • we could implement a kv-store (like pin set) on top of ipfs objects
    (basically ipfs-blob-store) and transition everything in one go.
  • DHT Records will be ipfs objects eventually anyway, under iprs.

TODOS:

@harlantwood

This comment has been minimized.

Show comment
Hide comment
@harlantwood

harlantwood Dec 14, 2015

Contributor

@harlantwood hopes to:

Contributor

harlantwood commented Dec 14, 2015

@harlantwood hopes to:

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Dec 14, 2015

Member

David Dias TODO


@jbenet there are some things you can do that will help me a ton if you manage to go through them quickly, these are:

  • Itemise the things available on the leveldb - /datastore on an IPFS Repo
  • Steps that happen on a IPFS init (this could be part of IPFS Spec)
  • Elaborate on your comment ipfs/js-ipfs#41 (comment), making it a spec
  • make a call on ipfs/specs#32, so that we can have full npm working on IPFS
Member

diasdavid commented Dec 14, 2015

David Dias TODO


@jbenet there are some things you can do that will help me a ton if you manage to go through them quickly, these are:

  • Itemise the things available on the leveldb - /datastore on an IPFS Repo
  • Steps that happen on a IPFS init (this could be part of IPFS Spec)
  • Elaborate on your comment ipfs/js-ipfs#41 (comment), making it a spec
  • make a call on ipfs/specs#32, so that we can have full npm working on IPFS

@eminence eminence referenced this issue Dec 15, 2015

Closed

Weekly Roundup #73

@RichardLitt

This comment has been minimized.

Show comment
Hide comment
@RichardLitt

RichardLitt Dec 15, 2015

Member

This Week

  • Sprint Management
  • API
    • Stub out the entire IPFS api at ipfs/api. PR api#13
    • Clean up and get the API working on Apiary.
Member

RichardLitt commented Dec 15, 2015

This Week

  • Sprint Management
  • API
    • Stub out the entire IPFS api at ipfs/api. PR api#13
    • Clean up and get the API working on Apiary.
@dignifiedquire

This comment has been minimized.

Show comment
Hide comment
@dignifiedquire

dignifiedquire Dec 15, 2015

Member

Todo

  • getting IPFS distributions ready for shipping
  • Work on rust-multiaddr
  • CR api doc PR from @RichardLitt
  • Move apiary source to ipfs/api
Member

dignifiedquire commented Dec 15, 2015

Todo

  • getting IPFS distributions ready for shipping
  • Work on rust-multiaddr
  • CR api doc PR from @RichardLitt
  • Move apiary source to ipfs/api
@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Dec 15, 2015

Member

TODO

  • fs-repo-migration shipped
  • dist.ipfs.io 'build things ourselves' shipped
  • new ipfs add make fewer objects
  • discuss directory sharding
  • look at modtimes in mfs
Member

whyrusleeping commented Dec 15, 2015

TODO

  • fs-repo-migration shipped
  • dist.ipfs.io 'build things ourselves' shipped
  • new ipfs add make fewer objects
  • discuss directory sharding
  • look at modtimes in mfs
@lgierth

This comment has been minimized.

Show comment
Hide comment
@lgierth

lgierth Dec 15, 2015

Member

@lgierth TODO

  • dev0.4.0
  • go-ipfs
    • harden gateway path prefix
    • fix gateway IPNS resolution recursion
  • infrastructure
    • order domains
    • castor data loss
Member

lgierth commented Dec 15, 2015

@lgierth TODO

  • dev0.4.0
  • go-ipfs
    • harden gateway path prefix
    • fix gateway IPNS resolution recursion
  • infrastructure
    • order domains
    • castor data loss
@chriscool

This comment has been minimized.

Show comment
Hide comment
@chriscool

chriscool Dec 15, 2015

Contributor

@chriscool TODO

  • ipfs-update
    • ⭐️ write more tests
    • started fixing ipfs-update
  • go-ipfs
Contributor

chriscool commented Dec 15, 2015

@chriscool TODO

  • ipfs-update
    • ⭐️ write more tests
    • started fixing ipfs-update
  • go-ipfs
@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Dec 16, 2015

Member

@jbenet TODO

  • new root for distributions: ipfs/ipfs-update#7 (comment)
  • human interfacing
    • talked with btrask about files/blocks
    • talked with parkan
    • update ipfs/notes#85
    • spec out ipfs-chain aggregation tool
    • spec out timestamping to btc tool
  • itemize what's in go-ipfs/$repo/leveldb
  • upgrade repo spec with "state only in ipfs objects"
  • write up transition out of leveldb plan for go-ipfs
  • finalize go-ipfs 0.4.0 upgrade path
  • resurrect osx earth node
  • spec doc for IPLD importing ipfs/js-ipfs#41 (comment)
  • begin kv-store on IPFS discussion
  • add papers to read to pub/sub ipfs/notes#64 (comment)
  • ipfs.io/docs/install use releases branch ipfs/website#82

if i told anyone i'd do something and it's not here, please let me know.

Member

jbenet commented Dec 16, 2015

@jbenet TODO

  • new root for distributions: ipfs/ipfs-update#7 (comment)
  • human interfacing
    • talked with btrask about files/blocks
    • talked with parkan
    • update ipfs/notes#85
    • spec out ipfs-chain aggregation tool
    • spec out timestamping to btc tool
  • itemize what's in go-ipfs/$repo/leveldb
  • upgrade repo spec with "state only in ipfs objects"
  • write up transition out of leveldb plan for go-ipfs
  • finalize go-ipfs 0.4.0 upgrade path
  • resurrect osx earth node
  • spec doc for IPLD importing ipfs/js-ipfs#41 (comment)
  • begin kv-store on IPFS discussion
  • add papers to read to pub/sub ipfs/notes#64 (comment)
  • ipfs.io/docs/install use releases branch ipfs/website#82

if i told anyone i'd do something and it's not here, please let me know.

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Dec 16, 2015

Member

@jbenet what I had for you was:

Member

diasdavid commented Dec 16, 2015

@jbenet what I had for you was:

@rht

This comment has been minimized.

Show comment
Hide comment
@rht

rht Dec 16, 2015

what's the 0.4.0 upgrade path? we'll ask users who care.

  • Is there a way to sample/snapshot the distribution of the versions the nodes are running?
  • What measure should be used for the network consensus? e.g. 1. the node itself, so that if 51% uses 0.3.8 then that's considered the 'majority', everything else are 'forks', 2. the number/size of hashes of each nodes
  • How are hotfixes broadcasted? Perhaps a semver header check + 101 status response can be used here. If a 0.4.0 node talks to a 0.3.5 node it could notify the latter to upgrade due to such and such list of vulnerabilities.

rht commented Dec 16, 2015

what's the 0.4.0 upgrade path? we'll ask users who care.

  • Is there a way to sample/snapshot the distribution of the versions the nodes are running?
  • What measure should be used for the network consensus? e.g. 1. the node itself, so that if 51% uses 0.3.8 then that's considered the 'majority', everything else are 'forks', 2. the number/size of hashes of each nodes
  • How are hotfixes broadcasted? Perhaps a semver header check + 101 status response can be used here. If a 0.4.0 node talks to a 0.3.5 node it could notify the latter to upgrade due to such and such list of vulnerabilities.
@RichardLitt

This comment has been minimized.

Show comment
Hide comment
@RichardLitt

RichardLitt Dec 21, 2015

Member

This Week

  • Sprint Management
  • API
    • ⭐️ Stub out the entire IPFS api at ipfs/api. PR api#13
    • Clean up and get the API working on Apiary.

Done

  • Helped plan IPFS Weekly. Comment pm#73
  • Added shields to ipfs-specs. PR specs#52
  • Updated project-dir with new data for project-repos. PR /ipfs/pull/137, PR ipfs#138
  • Wrote a doc on how to manually downgrade. PR fs-repo-migrations#10
  • Created and filled out ipfs/roundups
  • CR + M awesome-ipfs#20
  • Work on awesome-ipfs. Closed issues, opened PR awesome-ipfs#29, awesome-ipfs#30, awesome-ipfs#31
  • Added Travis to awesome-ipfs. awesome-ipfs#27
Member

RichardLitt commented Dec 21, 2015

This Week

  • Sprint Management
  • API
    • ⭐️ Stub out the entire IPFS api at ipfs/api. PR api#13
    • Clean up and get the API working on Apiary.

Done

  • Helped plan IPFS Weekly. Comment pm#73
  • Added shields to ipfs-specs. PR specs#52
  • Updated project-dir with new data for project-repos. PR /ipfs/pull/137, PR ipfs#138
  • Wrote a doc on how to manually downgrade. PR fs-repo-migrations#10
  • Created and filled out ipfs/roundups
  • CR + M awesome-ipfs#20
  • Work on awesome-ipfs. Closed issues, opened PR awesome-ipfs#29, awesome-ipfs#30, awesome-ipfs#31
  • Added Travis to awesome-ipfs. awesome-ipfs#27
@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Dec 21, 2015

Member

TODO

  • fs-repo-migration shipped ⭐️
  • dist.ipfs.io 'build things ourselves' shipped
    • closer now, builds dist.json file
  • new ipfs add make fewer objects
  • discuss directory sharding
  • look at modtimes in mfs
  • work on mfs locking issues and performance
Member

whyrusleeping commented Dec 21, 2015

TODO

  • fs-repo-migration shipped ⭐️
  • dist.ipfs.io 'build things ourselves' shipped
    • closer now, builds dist.json file
  • new ipfs add make fewer objects
  • discuss directory sharding
  • look at modtimes in mfs
  • work on mfs locking issues and performance
@dignifiedquire

This comment has been minimized.

Show comment
Hide comment
@dignifiedquire

dignifiedquire Dec 21, 2015

Member

@dignifiedquire sync update

Planned

  • [~] getting IPFS distributions ready for shipping
  • ⭐️ Work on rust-multiaddr
    • enque/dequeue
    • auto generated, published docs to github pages
    • support for IPFS and onion addresses
  • CR api doc PR from @RichardLitt
  • Move apiary source to ipfs/api

Extra

Special

  • Start benchmarking different request modules in node dignifiedquire/request-bench this is part of a more major undertaking that I might talk about if someone is interested as it is only partialy related to IPFS
Member

dignifiedquire commented Dec 21, 2015

@dignifiedquire sync update

Planned

  • [~] getting IPFS distributions ready for shipping
  • ⭐️ Work on rust-multiaddr
    • enque/dequeue
    • auto generated, published docs to github pages
    • support for IPFS and onion addresses
  • CR api doc PR from @RichardLitt
  • Move apiary source to ipfs/api

Extra

Special

  • Start benchmarking different request modules in node dignifiedquire/request-bench this is part of a more major undertaking that I might talk about if someone is interested as it is only partialy related to IPFS
@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Dec 21, 2015

Member

SPRINT CHECKIN

Member

diasdavid commented Dec 21, 2015

SPRINT CHECKIN

@lgierth

This comment has been minimized.

Show comment
Hide comment
@lgierth

lgierth Dec 21, 2015

Member

@lgierth sprint update

This sprint I managed to lose all of Castor's storage -- yay! :S And sorry again @davidar.

Apart from that dev040 is on the plate. We haven't gotten to discuss the migration strategy with users, I'll get to it right away.

Not really any * from me this week :)

  • dev0.4.0
  • go-ipfs
    • harden gateway path prefix
    • fix gateway IPNS resolution recursion
  • infrastructure
    • order domains
    • castor data loss
Member

lgierth commented Dec 21, 2015

@lgierth sprint update

This sprint I managed to lose all of Castor's storage -- yay! :S And sorry again @davidar.

Apart from that dev040 is on the plate. We haven't gotten to discuss the migration strategy with users, I'll get to it right away.

Not really any * from me this week :)

  • dev0.4.0
  • go-ipfs
    • harden gateway path prefix
    • fix gateway IPNS resolution recursion
  • infrastructure
    • order domains
    • castor data loss
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment