Skip to content

Platform audit: model correctness, comparison integrity, website UX, and UK-HANK roadmap #69

Closed
Closed
Platform audit: model correctness, comparison integrity, website UX, and UK-HANK roadmap#69

Description

@vahid-ahmadi
vahid-ahmadi
opened on Jul 20, 2026

Summary

MacroMod has a strong transparent-model foundation, but it is not yet one coherent modelling platform. It currently exposes five distinct models through a partially shared CLI/MCP layer, while parts of the website and README imply stronger reform support, comparability, accessibility, and validation than the implementations support.

The recommended direction is:

Question in, appropriate model out — with evidence, assumptions and limitations attached.

Before adding another large structural model, fix four foundations:

  1. A single capability and provenance registry.
  2. Truthful model-specific access and status labels.
  3. A result schema that distinguishes genuinely comparable quantities.
  4. End-to-end benchmark tests that run the actual models.

This issue consolidates the full repository, implementation, comparison, website, validation, additional-model, and UK-HANK audit performed on 20 July 2026.

P0: correctness and trust

SVAR estimation window

  • Correct the website claim that production estimation runs through 2025Q1 and is re-estimated on the current vintage.
  • Either update the site to the actual hosted behaviour or introduce an explicit current-estimation mode.
  • Add estimation_start, estimation_end, data_edge, and forecast_origin to every SVAR result.

Evidence:

  • svar/index.html:108-112, 142-148, 337-338
  • integration/src/policyengine_macro/core.py:643-650 fixes estimation through 2023Q2.
  • integration/src/policyengine_macro/mcp_server.py:231-238 already describes the hosted behaviour more accurately.

FRB-US installation

  • Make the documented policyengine-macro[models] installation produce a usable FRB-US model.
  • Package model.xml and LONGBASE.TXT as runtime package data, or remove FRB-US from the generic extra and document a separate supported installation.
  • Add a clean-environment test that performs the documented install and runs one FRB-US solve.

Evidence:

  • integration/pyproject.toml:22-33
  • integration/src/policyengine_macro/core.py:310-333
  • Modal succeeds only because it clones and editable-installs the repository in integration/modal_app.py.

Product positioning

  • Remove the README claim that every model scores the same reform objects and reports directly comparable quantities.
  • Replace it with capability-specific language.
  • Update the OBR bridge documentation: the bridge now exists, but is limited.
  • Remove the US label from OG-UK/PSL OLG unless a separate OG-USA implementation is actually integrated.
  • Remove claims that the current OBR adapter returns borrowing effects; it currently returns GDP, consumption, and business-investment effects.

Suggested replacement:

Each model answers a different class of economic question. MacroMod provides a common way to discover, run and interpret them while preserving each model's assumptions, evidence and outputs.

Relevant files:

  • README.md:16-24
  • models/index.html:54-57
  • docs/index.html:88-109
  • integration/src/policyengine_macro/core.py:1857-2001, 2225-2229

Mobile website

  • Fix content wider than the viewport at 390px.
  • Remove reliance on body { overflow-x: hidden; } as a way to conceal broken reflow.
  • Make Contact and GitHub reachable from the mobile header.
  • Increase small navigation touch targets to approximately 44px.
  • Fix mobile validation headings, body text, tables, and tabs being clipped.

Evidence:

  • style.css:166, 300, 322
  • Homepage and validation fail at a 390px audit viewport.

Model-by-model status

PolicyEngine microsimulation

Best for deterministic household tax-benefit calculations and population distributional scoring.

Current gaps:

  • Replace “Exact household arithmetic” with “Deterministic household calculations checked rule by rule against implemented UK and US legislation.”
  • Publish statute coverage by policy area and last review date.
  • Separate household-rule validation from population-estimate validation.
  • Report survey vintage, weights, calibration, package versions, sampling/imputation uncertainty, and official costing comparisons.
  • Clarify that UK population microdata are restricted/private and therefore not fully reproducible from the public repository.
  • Keep macro feedback explicitly out of scope for the base microsimulation.

Recommended status: Production-ready for selected household applications.

OBR macro model

Best for selected UK fiscal and macroeconomic scenarios over several years.

Current gaps:

  • Describe the PolicyEngine-to-OBR bridge as a documented scenario translation, not universal reform scoring.
  • Explain that reforms with the same static cost can have different incidence, labour-supply, investment, financing, and fiscal-feedback effects.
  • Add borrowing/fiscal outputs before promising them.
  • Expand the curated scenario catalogue beyond CGG, TCPRO, and CGIPS.
  • Add unit and range validation for every scenario.
  • Elevate known inert equations, missing source series, and weak closures into a visible model-status panel.
  • Distinguish anchored replication from free-running performance.

Recommended status: Validated for selected scenarios.

Bank of England structural VAR

Best for empirical shocks, forecasts, forecast revisions, and posterior uncertainty.

Current gaps:

  • Replace “what is driving the economy now” with “at the latest available model-data vintage.”
  • Expose the estimation endpoint separately from the latest conditioning observation.
  • Do not imply that identified historical shocks directly evaluate statutory reforms.
  • Expose hosted versus package-only capabilities, including decompositions and revision narratives.
  • Monitor accepted posterior draws/effective sample size and return warnings when inference is weak.

Recommended status: Validated replication for selected outputs.

FRB-US

Best current software-replication evidence and useful for US impulse-response analysis.

Current gaps:

  • Separate solver replication, add-factor tracking, external multiplier comparison, and substantive validation.
  • Explain that machine-precision agreement with pyfrbus proves implementation agreement, not forecasting or causal accuracy.
  • Keep model-consistent expectations and anticipated-policy scenarios explicitly out of scope.
  • Replace expert-facing raw add-error exposure with a curated scenario catalogue where possible.
  • Attach model version, LONGBASE vintage, policy rule, expectations mode, and run timestamp to every result.

Recommended status: Validated replication with important scope limits.

OG-UK / PSL OLG

Best for long-run behavioural and general-equilibrium mechanisms.

Current gaps:

  • Label calibration targets as calibration rather than validation.
  • Publish an achieved-versus-target moment reconciliation.
  • Run and publish at least one independent UK reform benchmark.
  • Re-estimate or clearly qualify inherited non-UK behavioural profiles.
  • Resolve the PolicyEngine package/statute-version conflict.
  • Distinguish suite steady-state scoring from the underlying transition-path and multisector capabilities.
  • Keep the model UK-only until OG-USA exists as a separate integrated member.
  • Treat slow custom solves as local or asynchronous jobs.

Recommended status: Research prototype / experimental.

OG-to-microsimulation overlay

  • Rename “dynamic score” to Experimental steady-state earnings overlay.
  • Explain that the long-run wage ratio is applied flat from the first year.
  • Explain that only employment income is scaled.
  • Explain that aggregate hours, self-employment, pensions, price effects, and heterogeneous wage incidence are not applied.
  • Validate the overlay against a predeclared benchmark before putting it in headline comparisons.

Evidence:

  • integration/src/policyengine_macro/assumptions.py:60-80, 101-118, 169-179
  • integration/src/policyengine_macro/core.py:2033-2139

Build a capability registry

The website, CLI, MCP service, and documentation should read from one registry.

For every model store:

model_id
display_name
model_class
geography
supported_question_types
supported_inputs
supported_outputs
unsupported_questions
time_horizon
data_vintage
estimation_sample
runtime
access_method
uncertainty_method
validation_status
limitations
package_version
source_commit
  • Generate website model cards from the registry.
  • Generate MCP tool descriptions/status from the registry.
  • Generate CLI model-status output from the registry.
  • Add recommend_model, get_model_status, and list_example_questions.
  • Make routing deterministic and capability-based before adding unrestricted AI routing.

Common result and comparison contract

Current ScoreResult lacks enough metadata to support trustworthy comparison.

Add:

analysis_type
model_id
model_version
package_version
country
baseline
baseline_vintage
data_vintage
estimation_sample
run_at
scenario
horizon
validation_status
warnings
provenance

Every quantity should include:

name
value
units
price_basis
time_basis
geography
baseline_definition
uncertainty
comparability_status

Use three comparability states:

  • Comparable

  • Related but not like-for-like

  • Not comparable

  • Do not put annual microsimulation revenue, cumulative OBR GDP, and OG steady-state GDP into one undifferentiated numerical table.

  • Refuse invalid comparisons.

  • For related results, explain differences in horizon, baseline, mechanism, units, and uncertainty.

  • Never average or rank structurally different model answers.

Suggested comparison note:

These results describe different horizons and mechanisms. They should be interpreted together, not averaged or ranked.

Website and information architecture

Navigation and routes

  • Give Compare a clear /compare/ route, or consistently call /docs/ Documentation.
  • Avoid using different labels for the same destination.
  • Preserve all global destinations on mobile.

Models

  • Add to each question/model route: best for, cannot answer, hosted/local, runtime, data vintage, uncertainty, and status.
  • Use capability-specific hosted/package/local labels rather than a single model-level access badge.

Validation

  • Replace “See how each model is validated” with “See the evidence for each model.”
  • Preserve the distinction among replication, calibration, statutory checking, and absence of ground truth.
  • Give horizontally scrollable mobile model tabs an explicit cue, or wrap/use a selector.
  • Convert dense comparison tables into compact evidence cards on narrow screens.

Papers

  • Correct PSL OLG geography.
  • Preserve the strong current separation among replication papers, calibrated models, and statutory evidence.

Accessibility verification

Existing strengths include skip links, visible focus styling, semantic captions, and scoped table headers.

Still required:

  • Keyboard-only navigation audit.
  • Screen-reader label/state audit.
  • 200% zoom and reflow testing.
  • Touch-target review.
  • Focus behaviour for sticky and scroll-selected tabs.
  • Responsive verification at 320px, 390px, tablet, and desktop widths.

Testing and reproducibility

Audit result:

  • Plain pytest could not collect because the integration package and MCP dependency were not installed in the audit environment.
  • With PYTHONPATH=src, the executable subset returned 87 passed, 39 skipped, and two environment-related failures.
  • PR CI skips real model solves.
  • The nightly contract suite does not run a complete benchmark for every model.

Actions:

  • Document a one-command developer setup that makes pytest collect.
  • Pin every model dependency to a release or commit SHA.
  • Return exact model SHAs in deployed provenance.
  • Add a full documented-install smoke matrix.
  • Add immutable benchmark fixtures.
  • Run at least one real benchmark per model nightly.
  • Keep private-data tests separate, but fail clearly when a required protected production contract cannot run.

Minimum benchmark matrix:

Model Required benchmark
PolicyEngine Selected household statute cases plus population aggregate reconciliation
OBR Anchored forecast, free-running result, and one scenario for each supported closure
SVAR Fixed-vintage estimation, forecast bands, and shock-identification benchmark
FRB-US Clean wheel installation and selected pyfrbus impulse-response comparison
OG-UK Achieved calibration targets, solver residuals, and one published external reform benchmark

Recommended models to add

1. BoE SC-MIDAS nowcaster

Priority: first.

Why:

  • Fills the within-quarter UK GDP nowcasting gap.
  • Fast enough for hosted use.
  • Interpretable release-cycle weighting.
  • Supports meaningful pseudo-real-time validation.

Source: https://www.bankofengland.co.uk/macro-technical-paper/2025/nowcasting-gdp-at-the-bank-of-england-a-staggered-combination-midas-approach

Implementation:

  • Start with an open-data profile using ONS and BoE series.
  • Make commercial PMI inputs optional.
  • Implement release calendars and real-time vintages.
  • Validate pseudo-real-time against simple benchmarks.
  • Return nowcast date, information set, vintage, uncertainty, and revision history.

2. ONS input-output model

Priority: second.

Why:

  • Adds sector, supply-chain, GVA, imports, and public-investment analysis.
  • Official source tables.
  • Transparent matrix implementation.
  • Fast hosted execution.

Source: https://www.ons.gov.uk/economy/nationalaccounts/supplyandusetables/methodologies/inputoutputanalysistooluserguide

  • Implement Type-1 direct and indirect effects.
  • Clearly state fixed production relationships and capacity assumptions.
  • State that household-induced Type-2 effects are excluded.
  • Version every source table.

3. BoE estimated UK DSGE

Priority: investigate after the two smaller additions.

Source: https://www.bankofengland.co.uk/-/media/boe/files/macro-technical-paper/2025/decompositions-forecasts-and-scenarios-from-an-estimated-dsge-model-for-the-uk-economy.pdf

Why:

  • Medium-term monetary and energy scenarios.

  • Structural decompositions.

  • Natural comparator for the SVAR and UK-HANK.

  • Ask the authors for code, calibration, estimation data, environment, licence, and benchmark outputs.

  • Choose official-code wrapper versus clearly independent implementation.

  • Do not claim exact replication without code and benchmark agreement.

4. UK-HANK

Priority: strategic research programme, not a quick model addition.

The Bank of England published A UK-HANK model, Macro Technical Paper No. 7, on 27 March 2026. It is not an OBR model.

Source: https://www.bankofengland.co.uk/macro-technical-paper/2026/a-uk-hank-model

Why:

  • Monetary transmission by renters, mortgagors, and outright owners.
  • Distributional effects of interest rates and house prices.
  • Household balance-sheet channels.
  • Fiscal-monetary interactions.
  • Better macro/distribution integration than the current wage overlay.

5. SimPaths

Source: https://github.com/centreformicrosimulation/SimPaths

Why:

  • Dynamic life-course employment, household formation, health, and wealth.
  • Useful for demographic and intergenerational questions.

Constraint: Java/Stata stack and restricted inputs. Start with local/asynchronous or precomputed scenarios.

Deprioritise NiGEM

Source: https://niesr.ac.uk/access-nigem

NiGEM is useful for global spillovers but commercially licensed. It does not fit an open hosted platform without a commercial agreement.

UK-HANK data feasibility

UK panel data exist, but the main calibration microdata cannot be redistributed openly.

The Bank paper uses:

  • ASHE 1993–2016 for the earnings-risk process; ASHE is described as a stable 1% employee panel.
  • Wealth and Assets Survey for income and wealth distribution comparisons.
  • ONS national accounts, effects-of-taxes-and-benefits data, input-output tables, CPIH, and English Housing Survey aggregates.

Sources:

Practical design:

  • Keep the runtime model open.
  • Never commit restricted ASHE/WAS/Understanding Society extracts.
  • Publish estimated parameters and disclosure-approved moments.
  • Publish calibration scripts that can run inside SRS/SecureLab.
  • Use synthetic/test data in public CI.
  • Separate model execution from restricted-data re-estimation.
  • Record provenance hashes and output-clearance dates.

UK-HANK implementation plan

Phase 0: access and licence gate, 2–4 weeks

  • Contact the paper authors/Bank of England.
  • Request source code, runtime environment, licence, calibration artefacts, estimation scripts, and benchmark outputs.
  • Decide official-code wrapper versus independent replication.
  • Define supported uses: monetary, housing, household-balance-sheet, and selected fiscal-monetary scenarios.

Phase 1: reproducibility shell

  • Add a capability-registry entry.
  • Add a versioned model manifest.
  • Add standard request/result metadata.
  • Implement local asynchronous jobs.
  • Create a published steady-state target fixture.
  • Define an explicit supported-scenario catalogue.

Phase 2: household block

  • Reproduce renters, owners, and mortgagors.
  • Implement liquid assets, housing, and secured borrowing.
  • Reproduce the earnings-risk process.
  • Implement tax/benefit progressivity.
  • Match published wealth, income, tenure, and MPC evidence.

Phase 3: aggregate blocks

  • Housing and rental sectors.
  • Firms, investment, and capital.
  • Small open economy.
  • Fiscal block and public debt.
  • Monetary policy.
  • Sticky expectations.
  • Sequence-space Jacobians.

Phase 4: validation

  • Reproduce published steady-state tables.
  • Reproduce distributional figures.
  • Reproduce monetary-policy IRFs.
  • Reproduce housing-price scenarios.
  • Reproduce fiscal-rule comparisons.
  • Reproduce transmission decompositions.
  • Compare distributions and mechanisms, not only headline GDP.

Phase 5: product integration

  • Start with precomputed scenarios.
  • Add asynchronous custom runs.
  • Return results by tenure, income, and wealth alongside GDP and inflation.
  • Add hard warnings that this is scenario analysis, not a statutory reform scorer or baseline forecast.
  • Map only selected, explicitly reviewed PolicyEngine reforms into HANK parameters.
  • Never infer reform mappings or units silently.

Delivery sequence

Sprint 1: trust and correctness

  • Fix the SVAR window.
  • Fix FRB-US packaging.
  • Correct OG-UK geography.
  • Rewrite README positioning.
  • Remove unsupported borrowing claims.
  • Rename and qualify the earnings overlay.
  • Fix mobile overflow and navigation.

Sprint 2: one platform contract

  • Build the capability registry.
  • Add shared provenance.
  • Add comparability states.
  • Generate website/CLI/MCP status from the registry.
  • Add deterministic question routing.

Sprint 3: real validation

  • Pin model versions.
  • Create immutable benchmark fixtures.
  • Test clean installation.
  • Run a real benchmark per model nightly.
  • Publish validation by output and intended use.

Sprint 4: useful expansion

  • Implement the ONS input-output model.
  • Prototype SC-MIDAS using public indicators.
  • Begin the UK-HANK code/licence gate.
  • Request the BoE DSGE replication materials.

Definition of done

A visitor or API client can determine, before running anything:

  • What question each model can answer.
  • What it cannot answer.
  • Whether the relevant capability is hosted, local, package-only, or experimental.
  • The data and estimation vintages.
  • The baseline and scenario definitions.
  • The model/package/source versions.
  • The runtime.
  • The uncertainty method.
  • The validation evidence for that specific use.
  • Whether results from two models are genuinely comparable.

A clean installation can execute the documented examples, and every deployed model has at least one real, version-pinned benchmark running automatically.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions