Downloading Recent Committed Ledger Entries

[Amaury Chamayou (achamayou)]

Proposal: Downloading Recent Committed Ledger Entries

Problem

The existing GET and HEAD /node/ledger_chunk endpoints expose only ledger
files whose on-disk names end in .committed. This is safe, but it can lag the
service's commit point: the current ledger file may contain a committed prefix
followed by uncommitted entries, and it is not renamed until the file is
complete and all of its entries are committed.

Clients that need the latest committed history must currently force a chunk
boundary and wait for the new .committed file. The API could instead
materialize a valid, immutable ledger chunk from the committed prefix without
reading or exposing any uncommitted entry. These synthetic resources use a
.committed_prefix suffix so they cannot be confused with canonical physical
.committed files.

Proposed API

Extend the existing locator with an opt-in query parameter:

GET /node/ledger_chunk?since=<seqno>&include_committed_prefix=true
HEAD /node/ledger_chunk?since=<seqno>&include_committed_prefix=true

Without include_committed_prefix=true, behavior remains unchanged.

With the option enabled, the locator first looks for a normal .committed
file, as it does today. If none is available, but the requested sequence number
is locally available and no later than the node's committed index, it selects
an exact committed range and returns:

307 Temporary Redirect
Location: /node/ledger_chunk/committed_prefix/ledger_101-140.committed_prefix
Cache-Control: no-store

Add an exact-range endpoint:

GET /node/ledger_chunk/committed_prefix/ledger_<start>-<end>.committed_prefix
HEAD /node/ledger_chunk/committed_prefix/ledger_<start>-<end>.committed_prefix

This endpoint:

• strictly validates the synthetic .committed_prefix chunk name;
• returns 404 unless every requested entry is locally available and committed;
• returns a complete ledger representation: header, unchanged transaction
  bytes, and positions table;
• sets x-ms-ccf-ledger-chunk-name to the .committed_prefix name; and
• sets x-ms-ccf-ledger-chunk-kind: committed-prefix and
  Cache-Control: no-store; and
• supports the existing Range, ETag, If-None-Match, Want-Repr-Digest,
  and Repr-Digest behavior.

The exact URL is important. Although the commit index may advance, the bytes
identified by ledger_<start>-<end>.committed_prefix never change. A client
can therefore retry or resume that download safely. Cache-Control: no-store
prevents HTTP caches from retaining the response, while the suffix and
chunk-kind header tell application clients that this is not a canonical
physical chunk and should not be archived or installed as a .committed file.
no-cache is not sufficient because it permits storage and only requires
revalidation.

The locator should use 307 Temporary Redirect, rather than the existing 308
for this case, because the selected end may advance between requests. Its
redirects and 404 responses should include Cache-Control: no-store so that
a cache cannot pin an older range or cache the temporary absence of a range.
Current node redirect behavior should otherwise remain: a backup that cannot
locate a recent native chunk redirects to the primary, while any node that can
validate and reconstruct an exact range may serve the exact-range request.

Note: This intentionally differs from discovery of a physical
.committed file, which continues to use 308 Permanent Redirect and a
.committed path. Synthetic committed prefixes use 307 Temporary Redirect
and a .committed_prefix path because the range selected for the same since
value may end later as the commit index advances.

Why More Than One Such Chunk Can Exist

The synthetic chunks are immutable views, not aliases for a mutable physical
file. For example:

1. The last physical .committed file ends at sequence number 100.
2. The active file starts at 101, contains entries through 180, and the commit
   index is 140. The API emits ledger_101-140.committed_prefix.
3. Later, the same active file contains entries through 220 and the commit
   index is 165. The next request starts at 141, so the API emits
   ledger_141-165.committed_prefix.

Both synthetic chunks remain valid even though they came from the same
in-progress physical file. Extending or replacing the first response would
break stable names, resumable downloads, and incremental backup.

There can also be two contiguous prefixes available at the same time because
physical chunking can run ahead of the commit watermark. For example:

1. The last canonical file is ledger_1-100.committed.
2. Replicated chunk boundaries have already produced the completed but
   unsuffixed files ledger_101-150 and ledger_151-200, plus the active file
   ledger_201.
3. The commit watermark catches up to 200 before those completed files have
   been published with .committed names.
4. The API can expose both ledger_101-150.committed_prefix and
   ledger_151-200.committed_prefix.

These two resources are distinct, contiguous, and contain only committed
entries. The implementation should preserve the existing physical boundaries
rather than collapse them into one response, and must iterate all eligible
unsuffixed files rather than assuming there is only one.

Implementation Sketch

1. Add a host-ledger operation that takes a start sequence number and returns
   an immutable {start, end, bytes} result. Under the ledger state lock, it
   should snapshot the committed index, reject a start beyond it, copy only
   entries at or below it, and choose an end at a valid signature boundary.
2. Serialize the copied entries as a normal completed ledger chunk. Reuse the
   layout already produced by LedgerFile::complete() and
   python/src/ccf/split_ledger.py: an offset header, raw entries, and 32-bit
   entry positions. Do not alter transaction bytes.
3. Expose this operation through AbstractReadLedgerSubsystemInterface and
   ReadLedgerSubsystem. Return owned bytes so the ledger lock is released
   before HTTP range handling and digest calculation.
4. Extend the locator and add the exact-range handlers in
   src/node/rpc/file_serving_handlers.h. Preserve the opt-in parameter across
   node redirects, use non-cacheable 307 responses for synthetic discovery,
   add the chunk-kind and cache headers, and factor the existing file response
   logic so it can also serve an in-memory buffer.
5. Add host-ledger tests for committed prefixes, uncommitted suffix exclusion,
   multiple consecutive synthetic chunks, and ranges spanning active files.
   Add end-to-end coverage for redirects, cache headers on redirects and
   404s, HEAD, byte ranges, digests, and parsing the downloaded result with
   ccf.ledger.LedgerChunk.
6. Add .committed_prefix support to filename range parsing and direct
   LedgerChunk reads, while ensuring committed-only directory discovery,
   read-only ledger directories, recovery, and cleanup continue to recognize
   only canonical .committed files. Promotion of a prefix for recovery, if
   ever needed, should be an explicit validation and conversion step.
7. Add a benchmark-labelled performance test covering representative prefix
   sizes such as 64 KB, 1 MB, 5 MB (the default ledger chunk size), and 50 MB.
   After a warm-up, repeat each case enough times to report the average
   materialization and HTTP response cost, throughput, and peak memory. Compare
   against serving an equivalent physical .committed file so the additional
   cost of prefix construction is visible.

[taicchoumsft]

Thanks Amaury, this would really help us index fresher data. Just a few questions:

1. Will turning this on slow the node down?

We use 100MB ledger chunks in ACL/MST and poll for new committed chunks every 15 seconds. That's a lot bigger than the 5MB default, so just want to confirm the cost for us.

The write-up says the node copies the entries while holding the ledger state lock. Is that the same lock the node needs to commit new transactions? If so, roughly how long is it held for a large prefix like ours?

I think this probably also depends on how the synchronizer should poll, which way were you recommending:

• If the client moves since forward past each prefix, each request only copies a small amount of new data, but we end up with lots of temporary small files.

• If the client keeps asking from the last .committed boundary, we will only have one prefix file, but it seems that the node will re-copy more and more data each time — up to ~100MB for us, every 15 seconds.

Which way should we go with for our case? Also, should the node put a size cap on how much it materializes at once, so a client like ledger synchronizer can't accidentally make it do a lot of work?

2. When is it safe to delete a .committed_prefix file?

Our setup has two parts: one component (ledger synchronizer) downloads chunks to a file share, and a separate one indexes from that share on a slightly slower schedule. So .committed-prefix files need to sit on the share in a separate folder for a while. We're not archiving them or treating them as real .committed files, but at least temporarily, they can't be throwaway either.

Our assumption is: we can delete a prefix covering entries 101–140 once we've downloaded real .committed files that cover that range. The covering file might be ledger_101-250.committed, so it's about the range being covered, not the filenames lining up. Is this the right assumption and will it hold under node restarts or other circumstances?

We can participate and give real numbers or benchmark our setup at 100MB chunks and production write rates.

Thanks so much Amaury!

[Amaury Chamayou (achamayou)]

taicchoumsft

1. Yes, and the last implementation item is to find out by how much. Exposing, adding the framing (offset table) and serving these files will have a cost. The ledger state lock is only used for ledger file operations, it is not on the critical path to commit new transactions. Because we know we only write after the commit watermark, I suspect we can have the logic done in such a way that it's not held very long.

The number and size of prefix files will be decided by the node, same as today for committed files. Let us run some numbers before we go too deep into this.

2. Any time. Once you have a full committed file that covers the same range, it definitely it no longer useful, but even before that, it's something that gets served on the fly by the node, it's not necessary for persistence or the functioning of the network.

[Amaury Chamayou (achamayou)]

taicchoumsft draft implementation in #8214, I probably won't have time to review in depth until next week, but if you want to take an early look, please do.