Fix unixfs fetch #364

hannahhoward · 2022-03-07T16:45:29Z

Goals

Fix GraphSyncing UnixFS for multiblock files
based on test in #363

Implementation

update go-ipld-prime to master (need to tag)
update go-unixfsnode to master, then further apply fix contained in Add unixfs to UnixFS path selector tail go-unixfsnode#28
modify the go-type for the ipld message v2 format, making extensions a map[string]*datamodel.Node instead of a map[string]datamodel.Node per @mvdan
modify the visitors on both sides of the request execution to read the entire file into ioutil.Discard, which triggers the loading of the files blocks. This causes the responder to send them over the wire and the requestor to verify them.
also, WOW -- did I not commit a bunch of changes to the docs? HA. Oh, right, I probably left them unsaved in VSCode.

For discussion

blocked on tagged go-ipld-prime
blocked on merge of unixfs PR and retagging

support pathing into multiblock unixfs files, properly reading the files inside the visitor so that they go over the wire

willscott · 2022-03-07T16:51:29Z

requestmanager/server.go

@@ -134,6 +137,16 @@ func (rm *RequestManager) requestTask(requestID graphsync.RequestID) executor.Re
 			Root:     cidlink.Link{Cid: ipr.request.Root()},
 			Selector: ipr.request.Selector(),
 			Visitor: func(tp traversal.Progress, node ipld.Node, tr traversal.VisitReason) error {
+				if lbn, ok := node.(datamodel.LargeBytesNode); ok {


this seems like the relevant chunk for the partial traversal. looks good

I think so right? in the meantime it fixes multiblock files as well.

tchardin and others added 3 commits March 3, 2022 10:18

bug: demonstrate reification traversal bug

491b86a

WIP

779a5a0

feat(impl): handle unixfs files properly

60856a4

support pathing into multiblock unixfs files, properly reading the files inside the visitor so that they go over the wire

hannahhoward requested review from rvagg and willscott March 7, 2022 16:45

willscott reviewed Mar 7, 2022

View reviewed changes

willscott approved these changes Mar 7, 2022

View reviewed changes

feat(deps): update to tagged deps

da98e38

hannahhoward merged commit f4afe52 into main Mar 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix unixfs fetch #364

Fix unixfs fetch #364

hannahhoward commented Mar 7, 2022 •

edited

willscott Mar 7, 2022

hannahhoward Mar 7, 2022

Fix unixfs fetch #364

Fix unixfs fetch #364

Conversation

hannahhoward commented Mar 7, 2022 • edited

Goals

Implementation

For discussion

willscott Mar 7, 2022

Choose a reason for hiding this comment

hannahhoward Mar 7, 2022

Choose a reason for hiding this comment

hannahhoward commented Mar 7, 2022 •

edited