Archive size wrong

[Stebalien]

When running ipfs get --archive, the expected size in the progress bar is wrong:

> ipfs get --archive /ipfs/QmPwecoMB9jN9DosznNqcApK63PnrEmMduxp1mn5nqS7Pq
Saving archive to QmPwecoMB9jN9DosznNqcApK63PnrEmMduxp1mn5nqS7Pq.tar
 68.00 KiB / 49.63 KiB [=====================================] 137.02% 0s

That is, the output file is actually 68KiB, 49.63 is the total size one would get when running ipfs get without --archive.

[Stebalien]

Actually, it looks like the size computation is just wrong every which way, we're using the size of the DAG. Worse, we're using it to set the content length.

[overbool]

@Stebalien ipfs get Qm... without --archive prints the protobuf node size. we should unmarshal the data and then computate its size.

[Stebalien]

Really, I have no idea how to fix it. We need to compute the size on the server before we start transferring which we often can't do because we don't actually know it ahead of time. We have the size of the DAG but not the size of the TAR we're actually transferring (also, the DAG's size may be wrong).

I'm not sure if we can fix this properly with the current protocol.

[qiwaa]

Maybe we can store the TAR in ipfs. Then we know the size of it. Does that make sense?

[Stebalien]

That's not going to work for large datasets, unfortunately. We can't afford to materialize the entire tar file before shipping it off to the user.