fix: retire the per-id ledger on a capped close; expose range-end incarnation; pair range-end marks in the grader - #175
Merged
Merged
Conversation
strategy_closed_trade_entry_incarnation bounded trade_index by trade_count() == trades_.size(), but pf_report_t::trades is trades_ followed by range_end_trades_ (fill_trades_section), so every range-end row — the position still open after the final bar, ABI v3's open_at_end — read as out of range and returned 0. run_strategy.py then wrote an empty "Engine entry incarnation" for the row, the grader failed closed on the identity gate and the verifier ladder rejected the TV-identical trim candidate for ena-grid (XAUUSD 1D), while the engine's per-lot range-end rows themselves matched TV (xau-grid 6/6, silicon 9/9 rows). Round-4b F3. The accessor now indexes the report's row space (report_trade_count / get_report_trade on BacktestEngine); trade_count() / get_trade() stay the Pine-visible closed trades. No struct changed, PF_ABI_VERSION stays 3. Tests: the F3 discrimination lives in the C++ tests — test_c_abi_setters pins a range-end row at index trade_count() (returns 0 on 650b0cb) and test_range_end_close pin I runs a real open-at-end position and reads the row's incarnation through the C ABI (range_end_inc == 0 on 650b0cb). test_run_strategy_range_end adds a CSV-writer companion only: given a populated entry_incarnation on an open_at_end row, the writer carries it to the Entry line; it stubs the engine, never calls the accessor and passes on the unpatched baseline, so it is not cited as the F3 pin. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXAE7TVTKZYbmtenHDHVJo
A default-FIFO strategy.close(id) resolves its target as
min(id_unclosed_qty_[id], avail). When avail capped the target, only the
target was debited and the remainder was carried; the next entry under
the same id re-credited on top of the carry (execute_market_entry /
apply_entry fill: id_unclosed_qty_[id] += qty), so the next close(id)
over-closed by exactly the carry. TradingView retires every entry under
the id on a close(id) ("all entries with the ID are exited at once"): a
capped fill still retires the id. Round-4b F1.
Evidence: 3commas xlm-grid 2025-05-08 10:45 TP_L35 closes 0.0987 of an
L35 lot of 0.1043 on both engine and TV; on 2026-02-06 14:30 the engine
then closed 0.1099 = 0.1043 + 0.0056 carried while TV closed 0.1043
(nvdax: engine 0.0972 vs TV 0.0926). A Python port of the two ledger
rules reproduced the engine's closed qty byte-for-byte on 3/3 grid bots.
Both consumption paths now erase the id's ledger whole: the POOC flush's
sole-call branch (flush_active_same_bar_close) and
compute_close_target_qty's default-FIFO branch (immediate / deferred
closes). The latter reports the retired remainder so the two paths that
undo an unfilled close — the COOF reissue re-credit in strategy_close and
suppress_declined_reversal_close_legs (PendingOrder::
suppressed_close_retired_ledger_qty) — restore the exact pre-call
balance; suppressed_close_consumed_ledger_qty keeps its target value,
which the short-seed collision cohort reads as the materialized qty.
Deliberate partials (explicit qty / qty_percent, strategy.exit legs) never
touch the ledger and keep their semantics. The never-decaying close
reservations (F2) are out of scope and noted where F1 touches them.
The whole-retire is scoped to the cap the evidence pins: the position or
a PRIOR-BAR (persistent) reservation. A tokenized sole call whose
shortfall comes SOLELY from this bar's other pending callsites
(SameBarCloseCallsite::retire_ledger_whole == false, decided at admission
in enqueue_same_bar_close) keeps the pre-F1 debit-by-target rule, so it
stays continuous with the zero-target A/A->B oracle, which performs no
cleanup for that case; that rule is unpinned against TV. Token 0 has no
same-bar pending reservation and always retires whole.
Test: test_close_id_retires_ledger — A/A2: pyramided grid under POOC,
close(L35) capped 0.1043 -> 0.0987 by a standing reservation on L36,
re-entry L35 0.1043, next close(L35) closes exactly 0.1043 (0.1099 on the
unpatched 650b0cb engine, verified); the ledger is absent right after the
capped fill (was 0.0056). B: a deliberate partial close keeps its
remainder on both the POOC and the next-bar path. C: the deferred path —
a position-capped close(L) co-queued after a declined reversal is
suppressed and the ledger comes back at its exact pre-call balance
(target + retired), so the follow-up close(L) closes 100, not 60 (fails
3 checks with the retired term dropped). D: the COOF C-cursor fill-recalc
path under POOC — the immediate re-credit restores target + retired
(fails 4 checks with that term dropped). E: a tokenized sole call capped
solely by a same-bar sibling site keeps its carry (fails under an
unconditional whole-retire). No existing test pinned the carry.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BXAE7TVTKZYbmtenHDHVJo
… grade every tape bounded A lot still open at the range's end is marked twice. TradingView's browser export writes its exit row with Signal "Open", priced at the range end's last close and netted of commission on both legs to the cent (3commas-xlm-grid-bot #615: qty 0.0912, entry 2189.13, mark 2261.44 -> 6.3511, TV 6.35); the engine, fed to TradingView's range end, books the same lot as its range-end close (open_at_end, 650b0cb) -- 6.351137 for that lot. The ws-report-v1 export prints the same row unmarked, as a closed trade at the last bar's close, which is exactly the engine's row. Under v4 (650b0cb) the tape's spelling picked a regime: an unmarked tape was bounded at TV's range end and the rows written; a marked tape (227 of the 396 scraped ETH tapes) was measured as the baseline had measured it -- feed unbounded three days past the range, the engine's marks withheld -- because the canonical grader paired the baseline's post-range closes with TV's Open rows on entry time, and the bounded run (v2) moved pnlP90 on the three 3commas grid bots (0.5536 -> 0.6675 xlm, 0.5253 -> 0.6403 btcusdt-p, 0.1814 -> 0.9310 nvdax), which the hard gate counts as a regression. Round 3's instrumentation of the grader on spark (diag/r4, 2026-09-02; every recorded engine_verify.json number reproduced to 4 decimals) put the cause in the grader, not the engine. On the grid bots pnlP90 is not a percentile: the fragmented-FIFO schedule path scores ONE aggregate, |sum(tv) - sum(eng)| / |sum(tv)| over the raw rows, and the marks went into it -- 22 TV Open rows at cents against 22 engine rows exact (21 of 22 agree to the cent; the 22nd, tv#614, is a FIFO lot-size split, per- unit P&L identical). The baseline's 0.5536 was itself an artifact of the unbounded feed: the engine's four post-range closes (+20.87, at prices 2-4% above the mark) over-earned TV's three surviving marks (+19.56) and cancelled a third of the real interior shortfall (+3.82 on 432.70 = 0.8822%), while 19 of 22 Open rows fell out of the window. The interior sums are identical between the bounded and unbounded runs (428.8824 vs 428.8825): the engine is byte-equivalent to the range end. The grader now pairs the two marks of one lot (verify_corpus.py pair_range_end_marks) before anything else looks at the rows. The engine says which rows are marks: engine_trades.csv gains a trailing column, "Engine range-end", "open" on the exit row of an open_at_end trade and empty elsewhere (write_engine_trades_csv); TV says so with Signal "Open". Marks pair with marks by the alignment every other trade gets (align_by_time: same direction, entry within the window and the price gate, closest entry first). A pair is a matched trade -- it counts on both sides, it covers, its entry is gated -- and its exit and P&L are compared nowhere that is gated: the rows leave both raw lists, so no consolidation, alignment, trim window, schedule aggregate or percentile (gated or report-only) sees a mark's cent-rounded P&L or its exit; the P&L agreement is reported at TradingView's precision instead (open_mark_pnl_cent_exact / open_mark_pnl_max_abs_usd). A pair still anchors the common window, so a TV trade closed between the last interior match and the open lots stays in it, unmatched, as before. Unpaired marks are ordinary rows: a TV Open row the engine closed for real earlier keeps its exit/pnl deltas, one the engine never opened leaves the coverage denominator as before, and an engine range-end row for a lot the tape closed is matched by entry and gated like any close. A tape without Open rows, or an engine CSV without a marked row, pairs nothing and grades as it did; the ws-report-v1 row pairs as the closed trade it is. The former same-bar proxy (TV Open row + engine exit on the same bar) is gone: the marker is the rule, and an engine close on the last bar without it is a fill against a mark, gated. The distinct-entry identity gate sees the marks as it saw them before pairing existed: the keys TradingView proves to hold two physical entries (distinct Signals at one time/price/direction) are read from the tape BEFORE the marks pair off -- an Open row's entry is a fill with a Signal like any other -- and distinct_entry_fill_mismatches runs over the gate pools with the pairs added back, so the engine's range-end row at a proven key must carry an entry incarnation exactly as any close's entry must. Without that, a key proven by an A lot closed and a B lot still open lost its proof when the B lot paired off, and the engine's unidentified rows passed (strong -> excellent); the same tape graded unpaired still refused it. With the marks paired by the grader, the spelling no longer picks a regime: run_strategy.py bounds every tape at TradingView's range end (_load_tv_range_end: metrics.json wsProvenance.requestedRange.to / to as 00:00 UTC, the tape's last row as the fallback, a probe's own ohlcv_end_ms kept, no range and no rows left as given) and writes the range-end rows marked. _withhold_open_at_end_trades is gone with the regime; the log line names the bound, its source and how the tape spells its row. On the diag set this grades alexgrover / aborkan89 (one Open row each) 100% at every p90 = 0 and pairs the grid bots' 22/22/28 lots exactly; their pnlP90 becomes the interior number (xlm 0.8822%, btcusdt-p 0.8458%, nvdax 1.4481% -> strong), above the baseline's recorded artifact, which the hard gate compares against verbatim -- that transition is the operator's call, recorded in the campaign, not something a pairing rule can paper over. Contract note: the registry's projection of engine_trades.csv (pineforge-workflow campaign/src/lib-results.mjs parseEngineTrades and airflow/pineforge_airflow/registry.py parse_engine_trades) pins the header and the field count per line. Both parsers now accept this 12-column header alongside the 11-column one engines before it wrote (the header fixes the width of every row), project the cell as rangeEnd (blank -> null, "open" -> "open", anything else a parse error) and carry rangeEnd null on the 11-column shape, so trade sets already recorded and the sets this engine records both project; the runner and Airflow ship with that parser before this engine is measured. The lab's loaders (source_trade_provenance load_verifier_engine_trades, verify-engine-local load_trades) and the grader read by column name and take the column as is. Tests: test_run_strategy_range_end.py -- the writer's column (header, values on exit rows only, 12 fields per line, the grader reads it), one regime for marked and unmarked tapes (same bytes out), the fallback and no-range paths; test_verify_corpus_open_mark.py -- the pair (matched, entry gated, exit/pnl not, P&L at cents), pairing on the lot not the bar, the unpaired shapes on both sides, a legacy CSV without the column, the ws tape, a one-lot tape, the grid-bot schedule aggregate unchanged by exact and by plainly wrong marks, and the identity gate: an A/B-signal key with the B lot open stays proven and the paired mark is identity-checked (refused without an incarnation, certified with one, the same verdict the tape gets unpaired). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BXAE7TVTKZYbmtenHDHVJo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three fixes for pyramided strategies and the browser-export tapes, gated together against a like-for-like baseline.
strategy.close(id)retires the id's ledger. When the fill was capped by the position or a prior-bar reservation, the engine debited only the filled qty and re-credited the remainder on the next entry under that id; TradingView retires the id. xlm-grid: the 2026-02-06 14:30 close was 0.1099 = 0.1043 + 0.0056 carried (TV 0.1043) — the whole interior divergence of that probe. Same-bar-pending caps keep the old debit (unpinned against TV, documented).strategy_closed_trade_*accessors were bounded bytrade_count()and read blank identities for the appended range-end rows, so the grader failed closed and the ena-grid (XAUUSD-1D) ladder rejected the TV-identical candidate. Now 25/25 matched, P&L equal to the cent.engine_trades.csvgains a 12th columnEngine range-end(openon a range-end exit; registry parsers accept both widths, pineforge-workflow#19), the range-end regime applies to every tape with a range (feed bounded at TradingView's range end, rows written), andverify_corpus.pypairs engine marks with TradingViewOpenrows per lot (matched, exit/pnl ungated, cent-exact, outside the schedule aggregates). Browser-export lanes (AAPL, EURUSD, 227 ETH tapes) are scored on their marks for the first time.Gate (like-for-like)
Because the harness scripts changed, the baseline was re-measured as 650b0cb's engine core + these scripts (
base-round4-harness-20260903→ snapshot1a633cec, itself +81 / −2 tiers vs the active baseline: AAPL +47, EURUSD +33; the only "worse" are the three grid bots at their honest values). Against that pin: PASS, hard 704 / 0 regressions, target +1 / −0; verdict0056629e.Test plan
-UNDEBUG), python 102/102gate_candidatePASS recorded🤖 Generated with Claude Code
https://claude.ai/code/session_01BXAE7TVTKZYbmtenHDHVJo