Test Catalogue

🧪 Test Catalogue

📌 Guide, not policy. tests/README.md, the expected files in tests/expected/ and .github/excel-evidence-policy.json govern; where this page differs, the repository wins.

The tests of K-SACCR at v1.0.2: the 28 catalogue IDs T01–T28 ported from the prototype workbook's TestCatalogue sheet (#59), the illustrative cases for client clearing, CCP trade exposures and settled-to-market trades added in 1.0.2, the eight Excel suites with their case and check counts, how to run them, and the TestCatalogue sheet in the workbook. Only T01–T14 are published values; everything else is illustrative and validates nothing (see Validation Approach).

---

🏷️ The 28 catalogue IDs

Inputs come from the template's NettingSets and Trades, valued on 2026-09-30. The cre99-example-* cases are in USD, unscaled: CRE99 prints USD thousands, so their amounts and expected values are the printed figures × 1,000, with a tolerance of half a printed unit (500 USD) for amounts. The other cases keep the template's figures, labelled EUR.

Test	Case and regime	Quantity (trade)	Expected	Tolerance (abs / rel)	Class
T01	cre99-example-1.bcbs	adjusted_notional (EX1-T1)	78,694,000 USD	500 / 0	published
T02	cre99-example-1.bcbs	supervisory_delta (EX1-T3)	-0.2694	0.00005 / 0	published
T03	cre99-example-1.bcbs	add_on.interest_rate	347,000 USD	500 / 0	published
T04	cre99-example-1.bcbs	exposure_value	569,000 USD	500 / 0	published
T05	cre99-example-2.bcbs	multiplier	0.965	0.0005 / 0	published
T06	cre99-example-2.bcbs	exposure_value	381,000 USD	500 / 0	published
T07	cre99-example-3.bcbs	exposure_value	5,406,000 USD	500 / 0	published
T08	cre99-example-4.bcbs	replacement_cost	40,000 USD	500 / 0	published
T09	cre99-example-4.bcbs	aggregate_add_on	629,000 USD	500 / 0	published
T10	cre99-example-4.bcbs	exposure_value	936,000 USD	500 / 0	published
T11	cre99-example-5.bcbs	margin_period_of_risk	14 business days	0 / 0	published
T12	cre99-example-5.bcbs	aggregate_add_on	1,401,000 USD	500 / 0	published
T13	cre99-example-5.bcbs	multiplier	0.958	0.0005 / 0	published
T14	cre99-example-5.bcbs	exposure_value	1,879,000 USD	500 / 0	published
T15	cre99-example-5.crr	exposure_value	1,879,000 USD	500 / 0	illustrative
T16	illustrative-posted-vm-cap.bcbs	exposure_value	40774849.34860364 EUR	1e-9 / 1e-9	illustrative
T17	illustrative-posted-vm-cap.crr	exposure_value	40774849.34860364 EUR	1e-9 / 1e-9	illustrative
T18	illustrative-posted-vm-cap.crr	cap_applied	Y	text	illustrative
T19	illustrative-swaption-forward-1pct.crr	exposure_value	760.4939328393385 EUR	1e-9 / 1e-9	illustrative
T20	illustrative-negative-ir-forward.crr	lambda_shift (NIR-C)	0.003	1e-12 / 0	illustrative
T21	illustrative-negative-ir-forward.crr	supervisory_delta (NIR-C)	-0.9941752720429311	1e-9 / 1e-9	illustrative
T22	illustrative-negative-ir-forward.bcbs	trade_status (NIR-C)	EXCLUDED	text	illustrative
T23	illustrative-negative-commodity-price.crr	lambda_shift (NCO-1)	5.5 price	1e-12 / 0	illustrative
T24	illustrative-other-risks.crr	add_on.other	880 EUR	1e-9 / 0	illustrative
T25	illustrative-other-risks.crr	exposure_value	1,232 EUR	1e-9 / 0	illustrative
T26	illustrative-other-risks.bcbs	trade_status (OT-1)	EXCLUDED	text	illustrative
T27	illustrative-climatic-commodity.crr	hedging_set (CL-1)	CO|CLIMATIC|STD	text	illustrative
T28	illustrative-credit-quality-step.crr	supervisory_factor (B03)	0.0042	1e-12 / 0	illustrative

A value matches when |actual − expected| ≤ max(absolute, relative × |expected|).

Notes from the repository:

• published values were checked against the CRE99 text on 2026-10-09. T02 is printed as −0.2694; the prototype catalogue had −0.27.
• CRE99 Example 3 prints a residual maturity of 9 months for trade 1, so its case runs with DaysPerYear 360 and 270 days, exactly 0.75 years.
• T16 was revised in #39 (#130) to the cap with C = NICA only; T16 and T17 therefore expect the same capped EAD in both regimes.
• T22 and T26 read trades NIR-B and OT-X in the workbook; the fixtures hold the same inputs as NIR-C and OT-1. T28 needs only trade B03 of NS-GAMMA. Netting sets duplicated per regime in the workbook (BCBS-EX5/CRR-EX5, BCBS-VMCAP/CRR-VMCAP, BCBS-NEGIR/CRR-NEGIR) are one fixture with an expected file per regime.
• illustrative values of T15–T28 came from a Python implementation that is not in the repository or from hand calculations; they stay illustrative unless an external implementation registered as a source reproduces them (#100).

Four more expected files carry no catalogue ID: illustrative-ir-sum-of-absolutes.bcbs (the CRE52.57(5) sum of absolutes), illustrative-ir-swap-unmargined.bcbs and .crr, and illustrative-partly-rejected.crr (an INCOMPLETE netting set).

Thirteen expected files added in 1.0.2, all illustrative, with values from the separate Python derivations in tests/derivations/, which share no code with the engine:

Fixture	Regimes	Shows	Derivation
illustrative-clearing-member-listed	CRR, BCBS	A listed equity index future and call in a margined CM netting set, daily remargining: MPOR 5 in both regimes (#359)	clearing_listed.py
illustrative-clearing-client-listed	CRR, BCBS	The same as CLIENT, weekly remargining: MPOR 9 under CRR, 14 under BCBS (#359)	clearing_listed.py
illustrative-ccp-trade-exposure	CRR, BCBS	A margined CCP netting set facing a QCCP, daily remargining: MPOR 10, the same EAD in both regimes (#112)	ccp_trade_exposure.py
illustrative-ccp-retained-vm	CRR, BCBS	The same set with retained VM: trade margined maturity factors 0.3, 1.0592 and 1.5 (#112)	ccp_trade_exposure.py
illustrative-ccp-non-qualifying	CRR	The first set with QCCP = N: the same EAD (#112)	ccp_trade_exposure.py
illustrative-settled-to-market-daily	CRR, BCBS	An equity index future and call reset the next business day: MF 0.2 (#360)	settled_to_market.py
illustrative-settled-to-market-periodic	CRR, BCBS	A five-year swap, an equity call and a Brent forward reset in 30 days; the swap keeps its bucket by E, the forward's reset falls on its maturity (#360)	settled_to_market.py

They check, depending on the case, MPOR, RC, deltas, add-ons, multiplier, EAD, cap flag, status, and the trade quantities remaining_maturity, maturity_factor and maturity_factor_margined. Together with the 16 catalogue files and the four above this makes 33 expected files with 186 checks, which is exactly what TEST_Cases runs.

---

🧰 The eight suites

From .github/excel-evidence-policy.json; counts are those of 64-bit Excel.

#	Suite	Entry point	Covers	Cases	Checks
1	TEST_Harness	TEST_Harness.RunTests	Worksheet-function regression with a completeness and cleanup verdict (checks = assertions)	4	6
2	TEST_Formulas	TEST_Formulas.RunFormulaTests	Domain and Excel-error contract of every SACCR_* function	9	82
3	TEST_MainState	TEST_MainState.RunMainStateTests	Excel state, stale-result detection, failure paths of the macros, output-table names	29	501
4	TEST_Cases	TEST_Cases.RunCaseTests	The numerical cases generated from tests/fixtures/ and tests/expected/	33	186
5	TEST_InputValidation	TEST_InputValidation.RunInputValidationTests	Rejection of malformed inputs and the input conventions	177	308
6	TEST_Aggregation	TEST_Aggregation.RunAggregationTests	Row-order independence, offsets and a mixed clearing population	12	15
7	TEST_Invariants	TEST_Invariants.RunInvariantTests	Relations every result must satisfy	4	23
8	TEST_Reconcile	TEST_Reconcile.RunReconcileTests	Netting-set and trade-leg reconciliation, and the evidence export	20	162
	Total			288	1283

On 32-bit Excel TEST_Formulas skips two LongLong checks (80). Suites 4–7 need TEST_CaseRunner; suites 2–8 run the real engine or write to sheets, so use a development workbook. The harness's four cases are replacement-cost.exact, maturity-factor.tolerance, cdo-delta.invalid-tranche (an expected #NUM!) and option-delta.repeatability.

What each suite checks, in short:

• TEST_MainState — the macros put calculation mode, events and screen updating back; operation and cleanup failures are raised separately and withdraw the failed run's results; results read out of date after an input edit (also after the Params tables moved) and not after formatting; a run in manual calculation recalculates first; every output table is found through its header name (layout.output-header-names, #253); RunSACCR_Silent restores its silent flag. The suite sets M_Main.gQuietLog for its run, as Test_RunAll does, so the macros' run messages stay out of its report (#303).
• TEST_Cases — each case writes its fixture into NettingSets, Trades and Params, runs the engine, compares, and restores. Results are counted per reference class.
• TEST_InputValidation — missing and duplicate IDs, unreadable dates and codes, error values, broken layout names, out-of-range values, invalid netting-set terms, malformed economic trades, and the date, IR bucket, FX pair and number-format conventions, plus REGULATORY / WHAT-IF. From 1.0.2 also the client-clearing floors with the large flag, disputes and overrides, role MULTI, the INFO lines (netting-set-clearing-*), the CCP inputs, floors, horizon, risk-weight texts and per-regime subtotal (netting-set-ccp-*, netting-set-qccp-on-role-n, netting-set-retained-vm-on-role-cm), and the settled-to-market rules (stm-*, settlement-*, legs-settlement-*, legs-reset-differ).
• TEST_Aggregation — the demo portfolio in original, reversed and rotated order gives the same Results; two opposite swaps in one bucket offset exactly; nothing offsets across currencies, IR and inflation, netting sets, entities, or standard and basis hedging sets. From 1.0.2, aggregation-mixed-clearing-population (#361) runs an ordinary, a CM, a CLIENT and a CCP netting set holding the same swap, and a CCP set without its QCCP status, under CRR: each VALID set takes its role's MPOR (10, 5, 5 and 10), TOTAL CRR sums the four and reads 4 of 5 netting set(s) VALID; 1 withheld, and the CCP line on Checks names one of two CCP sets VALID with its EAD.
• TEST_Invariants — recomputes add-on, RC, multiplier, PFE, EAD, cap and the TOTAL rows from Results; a second run writes identical outputs; IRBucketOffset = FALSE makes a CRR set with IR trades INVALID; deliberately corrupted copies are caught.
• TEST_Reconcile — CORE_Reconcile with synthetic figures (tolerances, scale, missing, duplicates, regimes, withheld, excluded, unusable rows), the two reconciliation sheets and their status, CSV field encoding, and an export that is byte-identical when written twice and refused after an input, bank-figure or output edit. Like TEST_MainState, it sets M_Main.gQuietLog for its run (#303).

A failure-injection mode proves a case-runner suite catches a wrong value: TEST_CaseRunner.InjectFailure: TEST_Cases.RunCaseTests must end with RESULT=FAIL; TEST_Harness.RunTestsWithInjectedFailure does the same for the harness. Injected runs are never evidence logs.

TEST_Benchmark.RunBenchmark (#292) is not a suite: it times the engine on large synthetic portfolios, needs TEST_CaseRunner, and neither Test_RunAll nor the Excel evidence includes it. Its results are under Getting Started › Portfolio size.

---

▶️ Test_RunAll

In the Immediate window of a development workbook:

Test_RunAll

It runs the eight suites in the order above and prints one line per suite, the failure lines of a suite that failed, and a final RESULT= line. An error raised by a suite counts as its failure and the next suite still runs. The run on the development branch just before the certification of 1.0.2:

SACCR TEST SUMMARY 2026-10-22 22:59:09
  PASS  TEST_Harness          completeness=COMPLETE; cases=4; assertions=6; failures=0; cleanup=PASS  (0,0 s)
  PASS  TEST_Formulas         cases=9; checks=82; failures=0  (0,1 s)
  PASS  TEST_MainState        cases=29; checks=501; failures=0; caller_state=RESTORED  (28,9 s)
  PASS  TEST_Cases            cases=33; checks=186; failures=0; restore=PASS  (5,3 s)
  PASS  TEST_InputValidation  cases=177; checks=308; failures=0; restore=PASS  (26,7 s)
  PASS  TEST_Aggregation      cases=12; checks=15; failures=0; restore=PASS  (2,9 s)
  PASS  TEST_Invariants       cases=4; checks=23; failures=0; restore=PASS  (3,5 s)
  PASS  TEST_Reconcile        cases=20; checks=162; failures=0  (11,8 s)
RESULT=PASS; suites=8; passed=8; failed=0; seconds=79,2

The certification itself ran each of the eight suites on its own, at the tagged commit cfb88e5, all passing with the same counts (288 cases, 1,283 checks; see Open Items and Limitations). The summary is an index: the Excel evidence record needs each suite's own full log (see Validation Approach).

---

📋 The TestCatalogue sheet

The workbook's TestCatalogue sheet keeps the 28 catalogue tests as formulas over the demonstration portfolio: each row reads one output cell by netting-set or trade ID and column header (INDEX/MATCH on Results or TradeCalc), compares it with the expected value and tolerance (abs, rel or text), and shows Engine, Difference and Result. Columns: Test · Area · Description · Source · Source status · Regime · Sheet · Reference · Metric · Expected · Tolerance type · Tolerance · Engine · Difference · Result · Note. E36 summarises (28 passed, 0 failed of 28 on the recorded run). It carries the Run SA-CCR, Validate inputs and Clear outputs buttons.

It differs from the versioned cases above in units and one tolerance:

	TestCatalogue sheet	tests/expected/ (run by TEST_Cases)
CRE99 amounts	USD thousands, as the demo portfolio enters them	USD unscaled (× 1,000)
T02	−0.2694, tolerance 0.00005 (since #267)	−0.2694, tolerance 0.00005
T07	5,406, tolerance 2: the demo's 9-month leg is 274 days, 0.7507 years, as no date gives 0.75 years at 365 days, so the demo shows 5,404.62, 1.38 below; an input difference, not engine rounding (its Note says so from 1.0.2)	5,406,000, tolerance 500 with exactly 0.75 years (DaysPerYear 360)
Authority	A workbook view; values shown are from the last run	The versioned reference, validated by tools/check.py and run in Excel

Its source-status labels are "Official published value (rounded)" (T01–T14), "Illustrative - separate unreviewed calculation", "Illustrative - unreviewed hand calculation" and "Engineering scenario - not authoritative" (#274); the repository's classes are only published (T01–T14) and illustrative (the rest).

Screenshot: TestCatalogue after the demonstration run

---

➕ Adding a case

Add a synthetic fixture under tests/fixtures/ and its expected file(s) under tests/expected/ in the format of TEST_CASES.md, with regime, units, locator, reference class and tolerance; then run python tools/generate_case_tests.py, import the regenerated TEST_Cases.bas, and update the evidence policy and README counts in the same pull request (the excel-evidence-policy gate checks them). Never turn current engine output into a published expectation.

---

🔗 Related

• Validation Approach — what a PASS does and does not show
• Worked Examples — the CRE99 examples behind T01–T14
• Worksheet Functions — FormulaTest
• Getting Started — building a development workbook
• Open Items and Limitations · Home