Relax piece size restrictions to allow uploading pieces of less than 1MiB

[ianconsolata]

Checklist

• This is not a question or a support request. If you have any Curio related questions, please ask in the discussion forum.
• This is not a new feature request. If it is, please file a feature request instead.
• I have searched on the issue tracker and the discussion forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release candidate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.

Curio component

• PoRep Pipeline
• Snap Deals
• Market
• IPNI provider
• Curio UI
• Other

Improvement Suggestion

A few of our partners at FF who use Singularity (including @parkan from the Internet Archive) want to upload pieces that are smaller than 1MiB, and we have SPs who are willing to make deals with us for pieces that small. It seems like it's not currently possible to make deals that small, however, as they get automatically rejected by Curio. Here's the PR where we updated the Singularity code to allow for small pieces: data-preservation-programs/singularity#479

I assume there is a limit / constant somewhere that is imposing this restriction. Would it be possible to update that limit to allow for smaller pieces?

@LexLuthr @magik6k Arkadiy says he has already been chatting with you both about this, and it may already be in progress?

[parkan]

just to provide a bit more context:

• pieces, for several reasons, may end up with TransferSize/file size of =< 512KiB and therefore rejected as incorrectly padded if declared as 1MiB:
  • DAG only pieces for shallow or low cardinatlity datasets (most common case)
  • "remainder" pieces e.g. a dataset that has total size modulo pieece size of =< 512KiB (this is not possible to prevent by underpacking pieces targeting a "remainder budget" without doing multiple packing passes since pieces are packed in parallel and predicting the true TransferSize of a given file range is nontrivial until the graph is built)
  • legitimate cases of small datasets
• it is not possible to receive DC allocations of <1MiB, therefore verified deals for pieces under this size cannot be made (to the best of my understanding, please correct me if some kind of allocation aggregation is possible, however this likely introduces nightmarish complexity given the parallel deal and allocation worlds)
• the only attack vector that this change would create is "pack a full sector of 1MiB pieces with 1 byte content each to juice the DC use", which is trivially detectable

all other cases of overpadding were addressed here: data-preservation-programs/singularity#479

if someone has a closed form solution for packing N bytes of arbitrarily sized and nested directories/files into M pieces such that no piece is < 512KiB with no blocks crossing piece boundaries please also let me know...