Introduce Offset Index

[sijie]

FEATURE REQUEST

1. Please describe the feature you are requesting.

Currently in a ledger, we indexed entries by entry id. It would be good to have an index by offsets. This allows supporting APIs like:

• readEntries(long startEntryId, int maxBytes)
• readEntries(long startOffset, long endOffset)

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

nice-to-have

3. Provide any additional detail on your proposed use case for this feature.

entry(/request) oriented api is not very good friendly to resource-usage when do prefetching or batching reads. offset oriented api is much better for estimating resource usage.

[eolivelli]

It would be great for stagare usecase, like reading a batch of sequential entries woth random access pattern (opposite to tailing reads)

[eolivelli]

A new wire protocol rpc would great as well

[Tielem]

FEATURE REQUEST

1. Please describe the feature you are requesting.

We create continuous streams of growing data, eg ledgers with entities.
However, we also require random access in the underlying data stream.

Proposed API's:

• getLastAddConfirmedByteOffset(): long

Returns the byte offset of the last byte of the last entry committed to the ledger.

• readBytes(long startOffset, long endOffset): Enumerator<Byte[]>

Reads all bytes between startOffset and endOffset (inclusive), returned per stored entry.
In case the endOffset is beyond the end of the ledger, the behavior should be the same as readEntries.

• readBytes(long startOffset): Enumerator<Byte[]>

Reads all bytes from startOffset to current confirmed end of ledger, returned per stored entry.

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

Currently using a different solution to tackle this use case, with less durability/scalability/etc.
It would make our future architecture simpler and better overall.

3. Provide any additional detail on your proposed use case for this feature.

• Uncommitted API's

While not needed for our use-case, to be API complete, uncommitted API's might be good to have.

• Polling API's

While not immediately needed for our use-case and we can tackle this with other polling mechanisms, it would be useful if we can open read binary from a ledger.
Starting at a given startOffset, keep receiving byte[] until either the handler is closed or endOffset is met.