feat(web): one chronology over four stores, with an injection-safe au… - #722
Merged
Conversation
…dit export (#703) Four stores record what keel has done and none of them knew about the others: the engine's JSONL log, the `orders` book, the `transactions` ledger, and the attestation tables. Merging them is only an improvement if the merge keeps them APART -- otherwise it is four tables with their labels removed. So provenance is a first-class field from a closed vocabulary, decided once in the service and never inferred from a row's shape by a renderer. A live fill is `venue-reported`. A paper fill is `simulated` -- the paper trader wrote it with no venue involved, and one word covering both would put synthetic and real evidence in one bucket, which is the one thing four separate tables at least never did. A `transactions` row is `imported-ledger`: the venue produced the CSV, but nothing verified the line on the way in, and that is not the same claim as a venue report. An attestation is `human-attested`, and the name of whoever swore to it is in the summary, because "who signed this" is the first question an auditor asks of one. A cycle is `engine-log`: the agent's own account, which nothing outside the process confirmed and nobody signed. `simulated` is the only provenance styled WARN. Not because synthetic evidence is worse -- an imported ledger line is not worse evidence than a venue report, it is different -- but because a paper fill sitting in a chronology beside real ones is the single row a reader must not skim past. THE EXPORT IS A SECURITY SURFACE, AND WAS TREATED AS ONE FROM THE FIRST LINE. This file is meant to be opened in Excel or Google Sheets by an auditor or a tax preparer, and both EXECUTE a cell whose text begins `=`, `+`, `-`, `@`, tab or carriage return. Several columns carry text keel did not write: a transaction's `notes` and `coinbase_id` come out of a venue's own CSV export, an attestation's `source` and `attested_by` were typed by a person. `csv_safe` applies OWASP's defence -- prefix with a single quote so the cell is inert text -- to EVERY cell rather than to a list of the risky ones, because a maintained list of which columns are attacker-influenced is exactly the thing that rots. Writing the end-to-end test for that found my own example was wrong: a hostile `notes` value is diluted by accident, because `summary` prefixes it with "deposit USD -- " and the cell no longer starts with a trigger. The real path is a field that lands at the START of a cell by itself, so the test drives a hostile `coinbase_id` into `reference`. Relying on the accidental prefix is exactly what applying the defence everywhere avoids. THE CSV ROUTE IS DELIBERATELY NOT AN `ApiRoute`. Everything in `API_ROUTES` is wrapped in the JSON envelope by `respond`, and every pin in `test_api.py` is parametrised over that table. A `text/csv` route inside it would force each of those to grow an exception, and an exception carved into a security pin is how the pin stops meaning anything. It gets its own handler branch -- inside the same loopback-plus-session admission, because an export of the whole audit trail must not be reachable more easily than the page it came from -- and its own header suite: `nosniff` (which matters MORE for a file a browser is told to save), `attachment` with a server-generated dated filename, `no-store`. The filename is never echoed from the query string: that would be the header-injection twin of the formula injection the body is defended against, and there is a test for it. NOTHING HERE IS TAMPER-EVIDENT, AND THE COLUMN SAYS SO. #703 asked each row to carry its hash. None of the four stores hashes its rows -- `orders`, `transactions` and both attestation tables have no hash column between them -- and the only hash-chained store in this codebase is the research trials ledger, which records experiments rather than trading activity. Folding it in to borrow its hashes would be provenance laundering, in the export whose entire purpose is provenance. So `row_hash` reads NOT RECORDED, never blank: blank invites the reader to assume the check passed. #721 files the engine work, and names the part that issue cannot pre-empt -- `orders` rows are UPDATED as a venue reports fills, so a chain over them breaks on every legitimate update. The chips come from the SCOPED window and are stored on the report, not derived from the rows on screen. Derived from those, pressing Flows would delete the Trades chip and leave no way back except knowing the empty string means all -- and a kind whose only rows fell past the cap would vanish from the bar while still being in the window. That bug was written, caught by its own test, and fixed here; it is the same one already fixed once on the Orders status tabs. The merged read is capped, because three of the four stores are unbounded and one request would otherwise cost the size of the deployment's whole history. The counts still describe the window, so a reader can see there is more. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…dit export (#703)
Four stores record what keel has done and none of them knew about the others: the engine's JSONL log, the
ordersbook, thetransactionsledger, and the attestation tables. Merging them is only an improvement if the merge keeps them APART -- otherwise it is four tables with their labels removed.So provenance is a first-class field from a closed vocabulary, decided once in the service and never inferred from a row's shape by a renderer. A live fill is
venue-reported. A paper fill issimulated-- the paper trader wrote it with no venue involved, and one word covering both would put synthetic and real evidence in one bucket, which is the one thing four separate tables at least never did. Atransactionsrow isimported-ledger: the venue produced the CSV, but nothing verified the line on the way in, and that is not the same claim as a venue report. An attestation ishuman-attested, and the name of whoever swore to it is in the summary, because "who signed this" is the first question an auditor asks of one. A cycle isengine-log: the agent's own account, which nothing outside the process confirmed and nobody signed.simulatedis the only provenance styled WARN. Not because synthetic evidence is worse -- an imported ledger line is not worse evidence than a venue report, it is different -- but because a paper fill sitting in a chronology beside real ones is the single row a reader must not skim past.THE EXPORT IS A SECURITY SURFACE, AND WAS TREATED AS ONE FROM THE FIRST LINE. This file is meant to be opened in Excel or Google Sheets by an auditor or a tax preparer, and both EXECUTE a cell whose text begins
=,+,-,@, tab or carriage return. Several columns carry text keel did not write: a transaction'snotesandcoinbase_idcome out of a venue's own CSV export, an attestation'ssourceandattested_bywere typed by a person.csv_safeapplies OWASP's defence -- prefix with a single quote so the cell is inert text -- to EVERY cell rather than to a list of the risky ones, because a maintained list of which columns are attacker-influenced is exactly the thing that rots.Writing the end-to-end test for that found my own example was wrong: a hostile
notesvalue is diluted by accident, becausesummaryprefixes it with "deposit USD -- " and the cell no longer starts with a trigger. The real path is a field that lands at the START of a cell by itself, so the test drives a hostilecoinbase_idintoreference. Relying on the accidental prefix is exactly what applying the defence everywhere avoids.THE CSV ROUTE IS DELIBERATELY NOT AN
ApiRoute. Everything inAPI_ROUTESis wrapped in the JSON envelope byrespond, and every pin intest_api.pyis parametrised over that table. Atext/csvroute inside it would force each of those to grow an exception, and an exception carved into a security pin is how the pin stops meaning anything. It gets its own handler branch -- inside the same loopback-plus-session admission, because an export of the whole audit trail must not be reachable more easily than the page it came from -- and its own header suite:nosniff(which matters MORE for a file a browser is told to save),attachmentwith a server-generated dated filename,no-store. The filename is never echoed from the query string: that would be the header-injection twin of the formula injection the body is defended against, and there is a test for it.NOTHING HERE IS TAMPER-EVIDENT, AND THE COLUMN SAYS SO. #703 asked each row to carry its hash. None of the four stores hashes its rows --
orders,transactionsand both attestation tables have no hash column between them -- and the only hash-chained store in this codebase is the research trials ledger, which records experiments rather than trading activity. Folding it in to borrow its hashes would be provenance laundering, in the export whose entire purpose is provenance. Sorow_hashreads NOT RECORDED, never blank: blank invites the reader to assume the check passed. #721 files the engine work, and names the part that issue cannot pre-empt --ordersrows are UPDATED as a venue reports fills, so a chain over them breaks on every legitimate update.The chips come from the SCOPED window and are stored on the report, not derived from the rows on screen. Derived from those, pressing Flows would delete the Trades chip and leave no way back except knowing the empty string means all -- and a kind whose only rows fell past the cap would vanish from the bar while still being in the window. That bug was written, caught by its own test, and fixed here; it is the same one already fixed once on the Orders status tabs.
The merged read is capped, because three of the four stores are unbounded and one request would otherwise cost the size of the deployment's whole history. The counts still describe the window, so a reader can see there is more.
What & why
Tests-first evidence
Gates (all must pass)
uv run ruff checkcleanuv run mypycleanuv run pytest -qgreenScope check
leave checked only if true, and if so: cite the source and open the discussion
BEFORE review (CONTRIBUTING.md, "Governance: rulings vs. machinery").