Implement unixfs sharding #3042

whyrusleeping · 2016-08-05T13:16:54Z

This PR implements a hash array mapped trie using merkledag nodes as a base layer for sharded unixfs directories.

This changeset is broken up into three distinct parts:

Implement the HAMT structure itself. This is a bunch of algorithmic code that i'd like some well thought out sanity checking on.
- Need confirmation on hash code numbering
- Thoughts on willf/bitset vs stdlib math/big.Int
- Implementation correctness
- Consensus on addition of fields to unixfs protobuf
Integration of HAMT code into the directory builder logic.
- provides us a nice abstraction to think about things in terms of a filesystem
- optimizes smaller directories by not sharding until we hit a threshold
Integration of directory builder code into mfs
- moves mfs from using a simple merkledag Node object to using a dirbuilder
- allows mfs to create sharded directories
- by way of the previous point, ipfs add and friends can now create sharded directories.

Thorough review will be very appreciated!

cc @Kubuxu @jbenet @kevina @diasdavid @dignifiedquire

also cc @haadcode as you might like to review the datastructure logic

whyrusleeping · 2016-08-05T13:19:51Z

Note: tests won't pass yet as I havent vendored some of the new libs. I havent quite decided between willf/bitset and just using the math/big.Int to handle the bitfield logic.

whyrusleeping · 2016-08-05T16:43:05Z

going with the stdlib, fewer imports and the stdlib is proven to be pretty stable. Its a hair bit less efficient on some operations, but we can optimize later if it becomes an issue.

kevina · 2016-08-05T20:13:32Z

@whyrusleeping, I am afraid I do not know anything about sharding, so I will have to first learn about that before I am of any use. Doing a quick search I discovered ipfs/notes#76, is there anything else that would help with context for this pull request?

jbenet · 2016-08-05T22:10:15Z

unixfs/hamt/hamt.go

+)
+
+const (
+	HashMurmur3 uint64 = iota


remember that this is configurable on the object

it will likely use multihash if this proposal goes through: multiformats/multihash#30

Yeah, i'd love to use an 'official' multihash table for this.

jbenet · 2016-08-05T22:16:21Z

@whyrusleeping this is a very big PR that will be critical to review + merge in stages. if this has any bugs it will be serious trouble and will take weeks of work to find and debug. please:

rip out unixfs into its own repo first, then we can iterate on the datastructure there first. once we're convinced it really works, we can pull this back into ipfs.
make it easy to test and verify the hamt data struct stuff is working well. meaning make some tests that shard large directories well. and be able to produce graphviz out of these to inspect and debug formation.
this is going to need very serious stress tests. so far i saw very light invariance testing but no serious stresses, with long sequences of operations expected to test how the HAMT grows and shrinks as stuff is added and removed, and testing that the exact same data structures are produced despite sequences of operations being different.

whyrusleeping · 2016-08-06T13:59:00Z

@kevina if you want to review the algorithm for the HAMT datastructure and use that to check my implementation, that would be very helpful.

https://en.wikipedia.org/wiki/Hash_array_mapped_trie

whyrusleeping · 2016-08-06T18:32:14Z

@jbenet I moved the HAMT code into its own repo and added a larger test to verify operation order doesnt change the resultant structure: https://github.com/whyrusleeping/unixfs-hamt

the hamt_stress_test.go will generate a random set of insertions and deletions and run through them in different permutations and verify the outputs are consistent.

jbenet · 2016-08-09T01:17:00Z

Thanks i will revivew this year

daviddias · 2016-11-10T09:13:17Z

@whyrusleeping @jbenet mentioned that it would be better rip out all the unixfs into its own repo and this actually might be extremely beneficial to point several people into something isolated they can review fully.

Also, I think the best review I can do is grab this implementation and write the implementation in JavaScript and possibly write the spec for it.

whyrusleeping · 2016-11-15T01:53:55Z

@diasdavid I had it pulled out at one point, but it was really just for the sake of 'putting the code somewhere else to look at it'. Actually extracting this code is difficult because it would require extracting a large number of other go-ipfs packages.

whyrusleeping · 2016-11-16T04:12:47Z

Rebased on latest master (1a365a8).

@diasdavid @dignifiedquire The actual HAMT code is isolated in this single file: https://github.com/ipfs/go-ipfs/blob/07683a7d9a6e83c859c84e75b2eb47d2aee79aa7/unixfs/hamt/hamt.go

All the review i'm looking for is just that file. the rest of the changes are just integration of that up through the stack and i'm not nearly as concerned about that part of the changeset.

dignifiedquire · 2016-11-17T13:08:52Z

unixfs/hamt/hamt.go

+	mask := new(big.Int).Sub(new(big.Int).Exp(big.NewInt(2), big.NewInt(int64(bp)), nil), big.NewInt(1))
+	mask.And(mask, ds.bitfield)
+
+	return popCount(mask)


I'm not entirely clear on how this function works/what exactly it's purpose is. And in turn I'm having a hard time verifying that the bit operations you do, do what they are supposed to be doing. Would be great to have a comment about what the function does and an explenation why the bit ops do what they do.

Not the popCount that one is clear, but rather the whole indexForBitPos.

License: MIT Signed-off-by: Jeromy <why@ipfs.io>

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

Kubuxu · 2017-03-22T23:17:43Z

Wow, rebased. Is this coming in soon (TM)?

whyrusleeping · 2017-03-23T05:14:09Z

@pgte Anything else we need to finish before this goes forward? I think we've agreed on not setting filesizes on the shards.

pgte · 2017-03-23T14:37:20Z

@whyrusleeping Correct, I don't think we need anything else.

Kubuxu · 2017-03-23T21:45:19Z

Apart from one removed test: #3042 (comment)

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

Kubuxu · 2017-03-24T00:53:35Z

mfs/dir.go

 	var out []NodeListing
-	err := d.ForEachEntry(context.TODO(), func(nl NodeListing) error {


I shouldn't have missed that.

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

whyrusleeping · 2017-03-24T03:51:46Z

Choo Choo! Here comes the merge train! 🚋🚋🚋🚋🚋🚋🚋🚋

Implement unixfs sharding

…ding Implement unixfs sharding

whyrusleeping added the need/review Needs a review label Aug 5, 2016

whyrusleeping force-pushed the feat/hamt-sharding branch 3 times, most recently from 74a6117 to 259aeb4 Compare August 5, 2016 16:23

jbenet reviewed Aug 5, 2016
View reviewed changes

whyrusleeping force-pushed the feat/hamt-sharding branch from 918e24b to 6c31225 Compare August 7, 2016 13:26

whyrusleeping added the status/in-progress In progress label Sep 14, 2016

flyingzumwalt added status/ready Ready to be worked status/blocked Unable to be worked further until needs are met and removed status/in-progress In progress labels Sep 26, 2016

whyrusleeping force-pushed the feat/hamt-sharding branch 2 times, most recently from d5d3b21 to 5ae869f Compare October 12, 2016 20:52

whyrusleeping added this to the Directory Sharding milestone Oct 18, 2016

whyrusleeping force-pushed the feat/hamt-sharding branch 2 times, most recently from c8f0fa0 to 10fb7a7 Compare November 1, 2016 07:49

whyrusleeping added status/in-progress In progress and removed status/ready Ready to be worked status/blocked Unable to be worked further until needs are met labels Nov 2, 2016

whyrusleeping force-pushed the feat/hamt-sharding branch from 4f84cfa to 07683a7 Compare November 16, 2016 03:36

dignifiedquire reviewed Nov 17, 2016

View reviewed changes

whyrusleeping and others added 5 commits March 21, 2017 20:19

fix mfs hamt integration

06fb495

License: MIT Signed-off-by: Jeromy <why@ipfs.io>

iterator technique for unixfs dir listing

c8af993

License: MIT Signed-off-by: Jeromy <why@ipfs.io>

add more docs on hamt

8d4791c

License: MIT Signed-off-by: Jeromy <why@ipfs.io>

chekc that size input to newHamtShard is a power of two

62bb7c9

License: MIT Signed-off-by: Jeromy <why@ipfs.io>

fix handling of filestore nodes in mfs during add

e876434

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

whyrusleeping force-pushed the feat/hamt-sharding branch from 14e7424 to e876434 Compare March 22, 2017 03:19

whyrusleeping force-pushed the feat/hamt-sharding branch 2 times, most recently from 774715f to f72c37e Compare March 23, 2017 05:04

Kubuxu approved these changes Mar 23, 2017

View reviewed changes

add global config switch for sharding

c4c6653

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

whyrusleeping force-pushed the feat/hamt-sharding branch from f72c37e to c4c6653 Compare March 23, 2017 23:42

Kubuxu reviewed Mar 24, 2017

View reviewed changes

fix go vet issues in hamt sharding PR

65b9716

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

whyrusleeping force-pushed the feat/hamt-sharding branch from 07da7ad to 65b9716 Compare March 24, 2017 01:02

whyrusleeping merged commit ac69697 into master Mar 24, 2017

whyrusleeping deleted the feat/hamt-sharding branch March 24, 2017 03:51

whyrusleeping removed the status/ready Ready to be worked label Mar 24, 2017

JustinDrake mentioned this pull request Aug 21, 2017

Large directories will break at the IPFS level OpenBazaar/openbazaar-go#639

Closed

jvmncs mentioned this pull request Apr 30, 2018

Extend Grid to take advantage of IPFS file sharding OpenMined/PyGrid-deprecated---see-PySyft-#181

Closed

daviddias mentioned this pull request Jun 25, 2018

Support for bigger dag nodes out of the box ipld/js-ipld-dag-cbor#52

Closed

schomatis mentioned this pull request Jun 26, 2018

What does it mean to be a directory in the UnixFS layer? #5157

Closed

momack2 added this to Needs Review in ipfs/go-ipfs May 9, 2019

laurentsenta pushed a commit to laurentsenta/kubo that referenced this pull request Feb 25, 2022

Merge pull request ipfs#3042 from ipfs/feat/hamt-sharding

a42ae17

Implement unixfs sharding

laurentsenta pushed a commit to laurentsenta/kubo that referenced this pull request Feb 25, 2022

go-ipfs-config: Merge pull request ipfs#3042 from ipfs/feat/hamt-shar…

01ddedd

…ding Implement unixfs sharding

laurentsenta pushed a commit to laurentsenta/kubo that referenced this pull request Mar 4, 2022

go-ipfs-config: Merge pull request ipfs#3042 from ipfs/feat/hamt-shar…

4299e72

…ding Implement unixfs sharding

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement unixfs sharding #3042

Implement unixfs sharding #3042

whyrusleeping commented Aug 5, 2016 •

edited by flyingzumwalt

whyrusleeping commented Aug 5, 2016

whyrusleeping commented Aug 5, 2016

kevina commented Aug 5, 2016 •

edited

jbenet Aug 5, 2016

jbenet Aug 5, 2016

whyrusleeping Aug 6, 2016

jbenet commented Aug 5, 2016 •

edited

whyrusleeping commented Aug 6, 2016

whyrusleeping commented Aug 6, 2016

jbenet commented Aug 9, 2016 •

edited by whyrusleeping

daviddias commented Nov 10, 2016

whyrusleeping commented Nov 15, 2016

whyrusleeping commented Nov 16, 2016

dignifiedquire Nov 17, 2016

dignifiedquire Nov 17, 2016

Kubuxu commented Mar 22, 2017

whyrusleeping commented Mar 23, 2017

pgte commented Mar 23, 2017

Kubuxu commented Mar 23, 2017

Kubuxu Mar 24, 2017

whyrusleeping commented Mar 24, 2017

		var out []NodeListing
		err := d.ForEachEntry(context.TODO(), func(nl NodeListing) error {

Implement unixfs sharding #3042

Implement unixfs sharding #3042

Conversation

whyrusleeping commented Aug 5, 2016 • edited by flyingzumwalt

whyrusleeping commented Aug 5, 2016

whyrusleeping commented Aug 5, 2016

kevina commented Aug 5, 2016 • edited

jbenet Aug 5, 2016

Choose a reason for hiding this comment

jbenet Aug 5, 2016

Choose a reason for hiding this comment

whyrusleeping Aug 6, 2016

Choose a reason for hiding this comment

jbenet commented Aug 5, 2016 • edited

whyrusleeping commented Aug 6, 2016

whyrusleeping commented Aug 6, 2016

jbenet commented Aug 9, 2016 • edited by whyrusleeping

daviddias commented Nov 10, 2016

whyrusleeping commented Nov 15, 2016

whyrusleeping commented Nov 16, 2016

dignifiedquire Nov 17, 2016

Choose a reason for hiding this comment

dignifiedquire Nov 17, 2016

Choose a reason for hiding this comment

Kubuxu commented Mar 22, 2017

whyrusleeping commented Mar 23, 2017

pgte commented Mar 23, 2017

Kubuxu commented Mar 23, 2017

Kubuxu Mar 24, 2017

Choose a reason for hiding this comment

whyrusleeping commented Mar 24, 2017

whyrusleeping commented Aug 5, 2016 •

edited by flyingzumwalt

kevina commented Aug 5, 2016 •

edited

jbenet commented Aug 5, 2016 •

edited

jbenet commented Aug 9, 2016 •

edited by whyrusleeping