Validation Approach

✅ Validation Approach

📌 Guide, not policy. docs/EXCEL_EVIDENCE.md, TEST_CASES.md and the methodology › Assumptions and scope govern; where this page differs, the repository wins.

A green check mark can mean many different things. This page separates them: which evidence exists, what each kind supports and does not, the policy accepted for v1.0.0 and kept in v1.0.1 and v1.0.2, how a run in Excel is recorded and validated against an exact commit, and how to reconcile your own figures step by step.

---

🧭 Evidence classes and what each supports

Evidence	What it supports	What it does not establish
Static checks (python tools/check.py, CI job Repository integrity)	Source layout, VBA file integrity, jumps, conditional compilation, public API, test-case file format, the generated case module, the evidence policy, changelog, whitespace	That the VBA compiles, runs in Excel, or computes any SA-CCR number correctly
Compile and import in Excel	The recorded host accepts the code of that commit	Numerical correctness
published cases (T01–T14, CRE99)	The specific Basel quantities, within the tolerance of the printed rounding	All Basel rules; any CRR rule
illustrative cases (T15–T28, four more, and from 1.0.2 the clearing, CCP and settled-to-market cases from tests/derivations/)	Regression: the behaviour does not change unnoticed	Validation of any kind
Invariants, aggregation and input-validation suites	Internal consistency, row-order independence, rejection of bad inputs	That the regulatory interpretation is right
FormulaTest	Excel's own formulas and the VBA functions agree on the inputs shown	Independent review of the rule itself
Reconciliation with a bank's figures	Where two implementations agree or differ, figure by figure	Independent validation of either (#100)
REGULATORY run class	The calibration equals parameter set CRE52-CRR-2026-10-09	Valid inputs, regulatory approval or certification

The repository's reference classes for expected values are:

Class	Meaning	In v1.0.2
published	Printed in a registered source (CRE99 / BCBS 279 Annex 4a), checked against the text	T01–T14
independent	Computed by an external implementation by another author, registered as a source with an ID starting IMPL-, from the same inputs (for example the R package SACCR of #352)	None: no external implementation is registered as a source
illustrative	Shows a format or a behaviour; not a validated figure	Every other expected value, including the clearing, CCP and settled-to-market values of 1.0.2, derived by the scripts in tests/derivations/

From 1.0.2 independent review is no longer required anywhere: the project has no separate reviewer: the code under test and its expected values come from the same hand. A value derived within the project from a regulation stays illustrative, however it was derived; an independent value can come only from an external implementation registered as a source (#410). Acceptance criteria ask for published values where a registered source prints them, and otherwise for illustrative values from a derivation separate from the code under test. Regression, boundary and rejection tests are evidence that the software behaves as specified, not that a figure is correct.

---

📜 Accepted policy (v1.0.0, kept in v1.0.1 and v1.0.2)

The policy (#44):

• v1.0.2, like v1.0.0 and v1.0.1, validates rules only with the published values of CRE99, checked against the registered text: T01–T14.
• There are no independent values; every other case stays illustrative and never counts as validation. That includes the client-clearing, CCP and settled-to-market rules added in 1.0.2: they are implemented and tested on illustrative values only.
• Because the published examples are Basel examples, they substantiate the Basel cases and the calculation paths they exercise, not the whole framework. CRR rules are implemented and tested against illustrative values, but not numerically validated in v1.0.2, and the release says so; CRR provisions were checked against the registered sources.
• A later release cross-checks results against a second source: #100 (milestone v1.3.0).

The completeness policy in TEST_CASES.md adds: a rule is validated for a regime only when a published or independent case passes for it in that regime; each implemented rule also needs a boundary case and each public input domain an invalid case; every recorded regime difference has a case that shows it.

---

🧾 The Excel evidence record

Static checks never stand in for Excel. A run in Excel is recorded in an evidence bundle bound to one exact commit and checked offline by tools/check_excel_evidence.py. Execution is manual: an operator runs Excel interactively; there is no automated Excel job, and the procedure never changes macro security or Trust Center settings.

What a bundle binds

Item	Bound to
Candidate	The full 40-character commit SHA; every log starts with CANDIDATE=<sha>
Sources	SHA-256 of every .bas, .cls, .frm (and .frx) under src/ and tests/ and of the template, as stored in Git (LF) — print them with --inventory, never with Get-FileHash
Policy	.github/excel-evidence-policy.json at that commit: the harness cases and assertions, every suite with its header, case count, check count per bitness, restoration field and first case
Environment	Excel version and build, Office bitness, Windows version, VBA runtime, the four references, macro policy and VBA-project access as found, trust_changes: false
Logs	SHA-256 of each cited log: one log per suite, each run on its own; the optional Test_RunAll summary must agree with them

Procedure (summary)

1. Check out the exact commit; git status clean.
2. Create the bundle folder outside the checkout; start session.log (UTF-8) with the SHA and start time.
3. Record the environment without changing it (About Excel, winver, Trust Center settings).
4. Build a development workbook from that checkout (by hand, or with Sync-SACCR-VBA.ps1 -Mode Development -IncludeSheets only if VBA-project access was already on); compile.
5. For the harness and each suite in policy order: clear the Immediate window, run the entry point, copy the whole output into its own log (harness.log, TEST_Formulas.log, …) after a typed CANDIDATE= line. Keep a failing report as it is. Optionally run Test_RunAll last into summary.log. TEST_Benchmark is not a suite of the policy: its output is not part of the record.
6. Note the harness CLEANUP= line and close the workbook.
7. Hash the logs; write host.json from the record template.
8. Validate:

python tools/check_excel_evidence.py --candidate-sha FULL_SHA \
  --evidence ../saccr-evidence-SHORTSHA/host.json \
  --output test-results/excel-evidence.json --summary test-results/excel-evidence.md

Outcomes

Outcome	Meaning
PASS	Complete, consistent record; every stage and the expected-error result passed
IMPORT_FAILED · COMPILE_FAILED	Import or compilation failed
TEST_FAILED	The harness or a suite reported a failure
CLEANUP_FAILED	Harness or operator cleanup failed
EXECUTION_TIMEOUT	A stage was abandoned as hung
INCOMPLETE	At least one stage did not run
UNAVAILABLE	No Excel run was attempted
EVIDENCE_INVALID	Missing, malformed, contradictory or stale-source record or logs

A record can carry several outcomes. Exit code 0 is a consistent pass, 1 a non-passing or invalid record, 2 a command or output error.

⚠️ The ~200-line Immediate-window limit. The VBE Immediate window keeps only about the last 200 lines, and the suites print nowhere else. A run that prints more loses its first lines — the header and first CASE lines — and its log fails validation (EVIDENCE_INVALID). TEST_InputValidation prints 183 lines on a pass (139 in 1.0.1), so a failing run with more than about 15 FAILURE= lines overflows. TEST_MainState and TEST_Reconcile run the workbook macros, whose run messages would overflow the window on their own; both set M_Main.gQuietLog for their run, as Test_RunAll does (#303). Before copying, check the first line in the window is the suite's header; if not, keep the partial copy, note the truncation in session.log and expect EVIDENCE_INVALID, never a pass.

Limits of the record

Digests are identity controls, not proof of what a person did; the CANDIDATE= line is typed by the operator. A pass covers the recorded host only, and validates SA-CCR numbers only as far as the cases do. The Test_RunAll summary is an index checked against the suites' logs, never a replacement. Keep logs free of credentials and client data.

A separate source round trip (tools/Export-SACCR-VBA.ps1 and tools/compare_vba_export.py, #34) checks that a built workbook holds exactly the source of one commit.

---

📍 Where the evidence stands (release 1.0.2)

Release 1.0.2 was certified in Excel at cfb88e5, the commit tagged v1.0.2, on Excel for Microsoft 365, Version 2608 (Build 16.0.20326.20072), 64-bit, Windows 11 Enterprise 25H2 x64: compile PASS; all eight suites PASS, each run on its own (288 cases, 1,283 checks); the evidence validator PASS; source round trips PASS with binding=VERIFIED for the release workbook (24 components) and the certified development workbook (35 components). One deviation is declared: a first attempt failed TEST_MainState on a recalculation interrupted in the Excel session; it is kept in the evidence, and a second attempt on a fresh workbook from the template passed every suite, with no code changed in between. The tag message and the release notes record the certification; see Open Items and Limitations.

Release 1.0.1 was certified at 31222d1 (230 cases, 1,033 checks) with no deviation (#308).

Release 1.0.0 (87194e8, #46) declared one deviation: its evidence validator rejected the record only because the policy at that tag misnamed TEST_InputValidation's first case; 1.0.1 ships the corrected policy.

---

🔁 Reconcile progressively

When your figures differ from K-SACCR's, compare from the top down and stop at the first level that differs:

#	Compare	Where in K-SACCR
1	Population and units: same netting sets, same trades, same regime, same reporting currency and scale	Results (count, status), TradeCalc, Checks, BankNettingSets scale
2	Status: excluded trades and withheld netting sets	Checks; Results status column
3	EAD	Reconciliation (EAD)
4	RC and PFE; then the multiplier (V, C)	Reconciliation (RC, PFE, Multiplier)
5	Aggregate add-on, then per asset class	Reconciliation (AddOn aggregate, AddOn IR … AddOn OT)
6	Hedging sets and buckets	HedgingSets, Buckets
7	Trade inputs and intermediates: notional, MtM, M, SD, d, delta, MF, effective notional, stand-alone add-on	TradeReconciliation

Compare economic-trade counts with economic-trade counts and leg counts with leg counts. Use the Economic trade ID column of TradeReconciliation to group legs as your bank groups its trades. A margined set has two calculations: check the margined columns and the cap columns (Z–AC) of Results. See Reconciliation and Evidence.

---

🔗 Related

• Test Catalogue — T01–T28 and the eight suites
• Worked Examples · Regulatory Sources — the published references
• Reconciliation and Evidence — bank comparison and the CSV
• Open Items and Limitations · Home