Object header size limitation

[roman-khimov]

Following on #171 (comment) discussion.

A regular attribute-less header is expected to be around 180 bytes. Session token can add ~200 bytes to that. But the biggest problem we have is split info data which at minimum is ~150 bytes of auxiliary data. It can have a copy of a parent header inside and it can also have a list of object IDs (hashes). Object header limit affects the maximum size of a large object. 1K of hashes is just 2G of object.

Attributes can be arbitrary, but we may remember FilePath and FileName as some commonly used. 256 bytes is a must to have for them (traditional Windows MAX_PATH is 260), but given the UTF-8 encoding we may even consider 4096 (regular PATH_MAX).

Given all of that 1K size limit can't be considered at all, 4K also seems to be too limiting (~6G "large" objects and really constrained attributes). To have things at least somewhat balanced I'd say we need a minimum of 16K headers. This leaves some space for attributes and allows to have 30G objects.

CC @AnnaShaleva, @notimetoname, @cthulhu-rider.

[cthulhu-rider]

taking into account #263, seems like 4K could be enough. If anything, the limit can be raised compatible

or just stock 16K and be sure?

[roman-khimov]

Four PATH_MAX seems to be more viable to me, so 16K. At least something that can last.

[cthulhu-rider]

im ok with 16K

[cthulhu-rider]

btw, does the limit apply specifically to header field or to all fields except the payload?

[roman-khimov]

I'd think of HEAD reply, but it'd be less convenient to check this way. OID is known to be small, signatures are short as well, the only thing that can be problematic is Header. @carpawell?

[carpawell]

I think limiting only Header field is OK. Also, if we add more fields here, the same header (the same size) may surprisingly become invalid for the new objects but still be OK for the old ones.

Another question: what about the big objects? Parent header may be OK, but the child parts that have it as a part of their headers may become invalid at the last split step. Do we want to do smth about it?

[roman-khimov]

Do we want to do smth about it?

No.