ADR 001: Addressing the 13-bit Limit via Extended Containers

[quickwritereader]

ADR 001: Addressing the 13-bit Limit via Extended Containers

Status

Proposed

Context

The standard PackOS header is a compact 16-bit structure featuring a 3-bit Tag and a 13-bit Delta. This 13-bit delta imposes a physical addressing limit of 8,192 bytes (). To support large datasets and general-purpose storage, we require a mechanism to link segments using absolute 32-bit offsets.

Decision

We will utilize Type 2 (TypeExtendedTagContainer) as a structural escape hatch. The encoder employs an Adjustable Oversize Limit Detector to decide when to pivot from 13-bit deltas to absolute 32-bit (4-byte) offsets.

1. Adjustable Encoding Triggers

• Default Pivot (4 KB): By default, the encoder triggers TypeExtendedTagContainer when a segment reaches 4 KB.

• Granular Access Strategy: A 4 KB pivot ensures that root-level arguments and types remain directly accessible within the primary 13-bit range. This allows decoders to read the argument count and basic metadata via high-speed deltas without being forced into segment-hopping immediately.

• The "Extreme" Trade-offs (BFS vs. DFS):

• Low Threshold (BFS-like Redirection): Forcing a very low trigger (e.g., 32 bytes) results in a BFS-style "header-heavy" structure. While this makes determining argument counts and types fast (as they are all clustered in the root), it makes reading the actual values difficult due to heavy fragmentation. Furthermore, the 8-byte overhead of TypeExtendedTagContainer can create a cyclic overhead problem, where the metadata itself triggers further chunking, hurting the very accessibility it intended to provide.

• High Threshold (8 KB Limit): Pushing to the 8 KB maximum maximizes payload density but risks "pushing" the root arguments into a second segment. This makes it difficult to retrieve even the basic argument count without reading the next packet.

• Minimal Threshold Recommendation: To avoid cyclic overhead, a minimal threshold (e.g., >32 bytes) is recommended.

2. Structural Layout
When triggered, the payload begins with an 4/8-byte management block:

• • Self-Offset (4 bytes):* Absolute 32-bit address for validation (optional)
• Continuation Offset (4 bytes): * Next Link: Absolute 32-bit offset to the next segment.
• End Marker (0xFfFfFfFf): Signals the terminal segment.

Consequences

• Pros: Lifts the 8 KB limit to 4 GB; optimizes for "Fast-Root" metadata access.
• Cons: Improperly tuned thresholds can cause "TypeContainer Bloat," where the structural overhead consumes more space than the data.

---

[heartical]

#10

Bless you ❤️

[quickwritereader]

modifications to ADR:

• first extended container of the spesific array/container will be placeholder. while the real data will be chunked from the second for that spesific array/container. this way we lose 2+4 bytes, but gain more room for elements in the same level.
• segment concat order are relaxed. this way different orders could be used based on how segments and its meta triplet data stored/streamed