PDP - increase root/piece size limit

[jennijuju]

Checklist

• This is not a question or a support request. If you have any Curio related questions, please ask in the discussion forum.
• This is not a new feature request. If it is, please file a feature request instead.
• I have searched on the issue tracker and the discussion forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release candidate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.

Curio component

• PoRep Pipeline
• Snap Deals
• Market
• IPNI provider
• Curio UI
• Other

Improvement Suggestion

today Filecoin hashing libraries are mostly implemented for porep sectors, in which means the largest piece it can handle is 64GiB. with PDP, the data container, aka the proof set size could be much more flexible. It would be great to enable hashing over >64GiB pieces.

Orthogonal to that, curio has an internal piece size limit of 200MiB or so, it limits the size of data could be uploaded & requires data prep overhead (its only chucking but still) - would be great to increase that limit, if not user configurable.

[jennijuju]

cant just increase the limit as the data is loaded in memory; treed is built in memory & work with large data may cause issues.