Implement 'Raw Node' node type #3307

whyrusleeping · 2016-10-15T19:49:37Z

This PR implements a 'Raw Node' dag node that is essentially just a block. Among other things, this allows us to make unixfs not have leaf nodes with extra framing and fit nicely inside multiples of 1024 and 4096.

A new flag --raw-leaves has been added to ipfs add that enables this functionality.

Do note that if you create objects with this flag, you will not be able to transfer them to other peers who are running any version before this (meaning those objects will not be able to be processed by ipfs 0.4.3 nodes).

This depends on the previous PR that turns the merkledag.Node into an interface.

kevina · 2016-10-16T17:44:21Z

@whyrusleeping, what is the ETA for this. (Ideally I would like my filestore code (#2634) to go in before this.)

Other than the usual conflicts my main concern is that this is implemented in a way the filestore can use it. In order for that to happen the filestore needs to know what format the block is in. If it is the old format it needs to decode it, if it is raw it needs to do nothing.

whyrusleeping · 2016-10-17T20:27:20Z

@kevina you will be able to check the cid, or do a type switch on the object. It should actually make things easier for you

License: MIT Signed-off-by: Jeromy <why@ipfs.io>

whyrusleeping · 2016-10-18T00:20:11Z

@Kubuxu @lgierth could you two review this for me?

kevina · 2016-10-18T02:12:52Z

@whyrusleeping if possible please try not to rebase this, I merged this into kevina/filestore.

kevina · 2016-10-18T17:38:57Z

@whyrusleeping I notice that neither "ipfs cat" or "ipfs get" work with raw nodes. Can I take it that those parts are just not implemented yet?

Also (although it does not need to go into this pull request) I would suggest that the DagService GetLinks() method has a special case for Raw Nodes to avoid having to unnecessary retrieve a Raw Node when it is not required.

whyrusleeping · 2016-10-18T17:43:02Z

Ohhhh... yeah. I should make sure cat and get work with raw nodes. Thats an issue.

License: MIT Signed-off-by: Jeromy <why@ipfs.io>

kevina · 2016-10-18T20:39:59Z

@whyrusleeping I can cat large files, but not small ones, for example:

kevina@intel:/aux/home/gocode/src/github.com/ipfs/go-ipfs$ ipfs add --raw-leaves Makefile
added zdvgq8kK8fHCAsT89ehxvm78PJqJj6FjHkPEMZTkARQnSQRyy Makefile
kevina@intel:/aux/home/gocode/src/github.com/ipfs/go-ipfs$ ipfs cat zdvgq8kK8fHCAsT89ehxvm78PJqJj6FjHkPEMZTkARQnSQRyy
Error: expected protobuf dag node

Note: I don't mind if you rebase 3796e70 (but not ded60a7 as that was was merged in, while 3796e70 was cherry picked).

whyrusleeping · 2016-10-18T21:42:45Z

@kevina fixed the add/cat of small files

License: MIT Signed-off-by: Jeromy <why@ipfs.io>

kevina · 2016-10-18T22:47:13Z

test/sharness/t0040-add-and-cat.sh

 test_expect_success "ipfs add --only-hash succeeds" '
    echo "unknown content for only-hash" | ipfs add --only-hash -q > oh_hash
 '

-#TODO: this doesn't work when online hence separated out from test_add_cat_file
+#TODO: this doesnt work when online hence separated out from test_add_cat_file


Typo? "doesnt"

I removed it because it messed up my syntax highlighting.

kevina · 2016-10-18T22:50:39Z

@whyrusleeping have you tried this when HashOnRead() is true? It looks like the verification code in blockstore/blockstore.go needs updating.

License: MIT Signed-off-by: Jeromy <why@ipfs.io>

whyrusleeping · 2016-10-18T23:03:45Z

@kevina good catch, fixed! Anything else you can see?

kevina · 2016-10-19T01:05:36Z

good catch, fixed! Anything else you can see?

I am working on adding raw node support to the filestore, i let you know if I see anything else missing :)

Kubuxu · 2016-10-19T06:40:39Z

I would feel much better about it if we run full tests with it enabled by default.
I will do it when I get back home.

kevina · 2016-10-19T16:18:12Z

importer/helpers/helpers.go

@@ -124,9 +137,20 @@ func (n *UnixfsNode) SetData(data []byte) {
 	n.ufmt.Data = data
 }

+func (n *UnixfsNode) DataSize() uint64 {


Is there a reason you called this DataSize() and not FileSize()? FileSize() means the size of the file if this node was the root. In #3314 I added a similar helper method but I called it FileSize() and not DataSize().

Yeah, FileSize probably makes more sense, i'll change it.

whyrusleeping · 2016-10-19T17:56:56Z

@Kubuxu running the tests with it enabled by default will be hard. All the hashes will be different.

License: MIT Signed-off-by: Jeromy <why@ipfs.io>

Kubuxu

Left one comment.

I generally feel not great about the quasi state when some features will work some won't in case of raw nodes.

About tests: we should change our tests to keep the all added blocks/files in one place so later the hashes can be remapped easily.

Kubuxu · 2016-10-19T20:04:05Z

core/corehttp/gateway_handler.go

@@ -417,8 +418,14 @@ func (i *gatewayHandler) putHandler(w http.ResponseWriter, r *http.Request) {
 			return
 		}

+		pbnewnode, ok := newnode.(*dag.ProtoNode)
+		if !ok {
+			webError(w, "Cannot read non protobuf nodes through gateway", dag.ErrNotProtobuf, http.StatusBadRequest)


As far as I can see it is put handler?
Does reading rawnodes through gateway work?

Yeah, reading raw nodes through the gateway uses the dagreader, which works fine. This nasty hacky area of code is the writeable gateway, which we don't really use...

Can you change "Cannot read" to "Cannot write".

kevina · 2016-10-19T21:12:35Z

I have added this to the filestore and things seam to work. I would say to go ahead and let this is even if there are still places that might brake with the raw nodes.

RichardLitt · 2016-10-25T19:29:58Z

Hey @christynelson467 - thanks for the message. Please try and keep comments relevant to the conversation at hand. Open a new issue if you have any questions about go-ipfs. Thanks.

whyrusleeping added the status/in-progress In progress label Oct 15, 2016

whyrusleeping mentioned this pull request Oct 15, 2016

Should format exist in leaves? #230

Closed

kevina mentioned this pull request Oct 16, 2016

Add PosInfo() method to blocks and merkeldag nodes. #3309

Closed

whyrusleeping force-pushed the feat/raw-nodes branch from 4b1629d to 92c74f0 Compare October 17, 2016 20:39

unixfs: allow use of raw merkledag nodes for unixfs files

ded60a7

License: MIT Signed-off-by: Jeromy <why@ipfs.io>

whyrusleeping force-pushed the feat/raw-nodes branch from 92c74f0 to ded60a7 Compare October 17, 2016 23:51

kevina mentioned this pull request Oct 18, 2016

Create a FilestoreNode object to carry PosInfo #3314

Merged

raw dag: make raw nodes work in cat and get, add tests

3796e70

License: MIT Signed-off-by: Jeromy <why@ipfs.io>

fix add/cat of small files

8ce9963

License: MIT Signed-off-by: Jeromy <why@ipfs.io>

whyrusleeping force-pushed the feat/raw-nodes branch from 55b20ff to 8ce9963 Compare October 18, 2016 22:26

kevina reviewed Oct 18, 2016

View reviewed changes

update HashOnRead validation to properly support cids

9796a03

License: MIT Signed-off-by: Jeromy <why@ipfs.io>

whyrusleeping assigned ghost and Kubuxu Oct 19, 2016

kevina reviewed Oct 19, 2016

View reviewed changes

rename DataSize to FileSize

bb986eb

License: MIT Signed-off-by: Jeromy <why@ipfs.io>

whyrusleeping added this to the ipld integration milestone Oct 19, 2016

whyrusleeping mentioned this pull request Oct 19, 2016

Limit blocks files to 256KiB boundaries #2122

Closed

Kubuxu reviewed Oct 19, 2016

View reviewed changes

whyrusleeping merged commit 6f3ae5d into master Oct 24, 2016

whyrusleeping deleted the feat/raw-nodes branch October 24, 2016 18:59

whyrusleeping removed the status/in-progress In progress label Oct 24, 2016

kevina mentioned this pull request Nov 4, 2016

merkledag: optimize DagService GetLinks for Raw Nodes. #3351

Merged

ghost mentioned this pull request Dec 23, 2016

Code Review v0.4.4...v0.4.5 #3534

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement 'Raw Node' node type #3307

Implement 'Raw Node' node type #3307

whyrusleeping commented Oct 15, 2016

kevina commented Oct 16, 2016

whyrusleeping commented Oct 17, 2016

whyrusleeping commented Oct 18, 2016

kevina commented Oct 18, 2016

kevina commented Oct 18, 2016

whyrusleeping commented Oct 18, 2016

kevina commented Oct 18, 2016 •

edited

Loading

whyrusleeping commented Oct 18, 2016

kevina Oct 18, 2016 •

edited

Loading

whyrusleeping Oct 18, 2016

kevina Oct 18, 2016

kevina commented Oct 18, 2016 •

edited

Loading

whyrusleeping commented Oct 18, 2016

kevina commented Oct 19, 2016

Kubuxu commented Oct 19, 2016 •

edited

Loading

kevina Oct 19, 2016

whyrusleeping Oct 19, 2016

whyrusleeping commented Oct 19, 2016

Kubuxu left a comment

Kubuxu Oct 19, 2016

whyrusleeping Oct 19, 2016

Kubuxu Oct 19, 2016

kevina commented Oct 19, 2016

RichardLitt commented Oct 25, 2016

Implement 'Raw Node' node type #3307

Implement 'Raw Node' node type #3307

Conversation

whyrusleeping commented Oct 15, 2016

kevina commented Oct 16, 2016

whyrusleeping commented Oct 17, 2016

whyrusleeping commented Oct 18, 2016

kevina commented Oct 18, 2016

kevina commented Oct 18, 2016

whyrusleeping commented Oct 18, 2016

kevina commented Oct 18, 2016 • edited Loading

whyrusleeping commented Oct 18, 2016

kevina Oct 18, 2016 • edited Loading

Choose a reason for hiding this comment

whyrusleeping Oct 18, 2016

Choose a reason for hiding this comment

kevina Oct 18, 2016

Choose a reason for hiding this comment

kevina commented Oct 18, 2016 • edited Loading

whyrusleeping commented Oct 18, 2016

kevina commented Oct 19, 2016

Kubuxu commented Oct 19, 2016 • edited Loading

kevina Oct 19, 2016

Choose a reason for hiding this comment

whyrusleeping Oct 19, 2016

Choose a reason for hiding this comment

whyrusleeping commented Oct 19, 2016

Kubuxu left a comment

Choose a reason for hiding this comment

Kubuxu Oct 19, 2016

Choose a reason for hiding this comment

whyrusleeping Oct 19, 2016

Choose a reason for hiding this comment

Kubuxu Oct 19, 2016

Choose a reason for hiding this comment

kevina commented Oct 19, 2016

RichardLitt commented Oct 25, 2016

kevina commented Oct 18, 2016 •

edited

Loading

kevina Oct 18, 2016 •

edited

Loading

kevina commented Oct 18, 2016 •

edited

Loading

Kubuxu commented Oct 19, 2016 •

edited

Loading