[PoC] importer: UnixFS type Raw is never used

[schomatis]

Do not merge.

[schomatis]

As explained in #5106 (comment), the balanced builder is not using the Raw type for the leaf nodes, only the trickle builder does (i.e., a minimal fraction of the code). This seems to be the case for a long time now, all of the hard-coded hashes in the tests are built with this bug. Right now I'm incorporating this in the PR #5118 which is refactoring the balanced builder to avoid upsetting Jenkins, but I'm not sure how to proceed in the long term. The worst part of this (IMO) is that we're saying something we're not doing (actually we're not saying much as there are no UnixFS specs, but the code clearly implies the use of the Raw type for leaves) and I imagine that would be frustrating for someone who would like to implement its own IPFS stack (or at least the Files API part of it).

[schomatis]

/cc @whyrusleeping @Stebalien

[kevina]

@schomatis please see #4693 I think it is related, and had to add some special case code to avoid changing existing hashes of empty files. Any sort of refactoring will need careful testing to make sure we still do things in the same manor so that we don't change existing hashes.

[whyrusleeping]

@schomatis Yeah, the lack of the usage of Raw here is erroneous here. Unfortunately we're committed to it unless we change hashes. With the --raw-leaves option we switch to using the raw node type which skips the protobuf overhead entirely, this is what we're hoping to switch to moving forward.

[schomatis]

@whyrusleeping Could we change the trickle builder behavior (and hashes) to avoid using the Raw UnixFS type altogether then? (To avoid future confusions.)

[whyrusleeping]

nope, can't change hashes.

[schomatis]

@whyrusleeping Then we should write a clear documentation around the .proto UnixFS definition to clarify this. Is there anything else we could do to improve the current situation?

[whyrusleeping]

Then we should write a clear documentation around the .proto UnixFS definition to clarify this

Yeah, more docs are good.

Is there anything else we could do to improve the current situation?

The current situation of not using Raw even though its there? Really just docs for now, and try to push work on unixfs v2 forward.

[schomatis]

The current situation of not using Raw even though its there?

Just to clarify, I think it's a bit more complex than that because we are actually using it: the trickle DAGs use that type for its leaves (although yes, it's a tiny fraction of the total, not sure who uses trickle) and the balanced builder insinuates it in its code but doesn't (that should be partly addressed in #5118).

[Stebalien]

As far as I know, MFS currently uses trickle. And yeah, this all around sucks.

[Stebalien]

@schomatis status? Can we close this?

[schomatis]

Yes, we can't fix this and it has been documented in the code.