-
Notifications
You must be signed in to change notification settings - Fork 249
Description
This is the next step in the blocking requests work. There's a few options, but the goal is to lock down a stable API with a working solution, such that we can potentially tear out the backend (signature/receipt management and caching).
We could try to access receipts as we currently do, through a historical query, but this is inherently async (would need to be polled from the task system), adds contention to an already complex system (many potential users of this single "historical query" cache), and is more general than needed (surely most receipts are produced from a signature that's still in, or just-been-flushed-from, memory, rather than arbitrary ledger scanning?).
Drilling into the last point, I think this is the sensible approach:
- Add a global hook on the signatures table(s), to store the most-recently committed signatures.
- That fires within
Aft::commit(), when it callsstore->compact(idx);(itself eventually callingmap->post_compact();). - Immediately after this,
Aft::commit()callstrigger_callbacks(), telling the deferred session that their transaction was committed. - Those callbacks can go fetch the signatures from wherever they were temporarily stashed. They should do this inline, while there's a reasonable understanding that recent ones are definitely still available, even if they defer actual IO work to later.
- Somewhere this callback invokes a peer of
default_respond_on_commit_func()(called something likereplace_response_body_with_receipt_on_commit_func()), which either replaces the body with the receipt (constructed from above stash) or an error (for commit failures). Then the existing response flow proceeds, writes this back to the client.
An important point here is that we need to stash multiple signatures, not just 1, because commit can advance in chunks/batches. If a backup hears about commit advancing from 10 to 100 in a single step, and has signatures every 10 transactions, it needs to fire hooks and callbacks for all those intermediate entries. I think storing ~10s of sigs in this system is fine, and gives us some multiplier on the sig-threshold for "how large a commit-chunk will still get a receipt-on-commit response". That's best effort, and we'll need an explicit error path for these cache misses (potentially patched by a better future implementation!). In practice, it should be easy to configure this so that those situations are a client-timeout or similar known-extreme.