Add versioned UK staging telemetry and non-release smoke verification

[anth-volk]

Problem

Microcosm's existing staging records are specific to the current US producer and do not provide a versioned, country-neutral contract for UK spine and calibration runs. The UK workflow also needs an explicitly non-release smoke mode that exercises the real construction command without publishing population data.

Selecting a fixed number of source households before construction is not a valid way to request a final dataset size. Construction can add support records and donor records, so the input count and final household count differ.

Scope

• Preserve the current US version 1 staging paths and behavior.
• Add validated version 2 run-index, latest-run, manifest, progress, calibration-progress, and event records.
• Store UK staging records in the private policyengine/populace-uk-staging dataset repository.
• Permit only reviewed aggregate JSON artifacts; reject H5 files, source microdata, archives, credentials, and row-level output before upload.
• Run the UK spine command against the complete committed synthetic fixture in integration tests.
• Retain fractional input sampling for scale tests, while removing the source-household-count option.
• Mark smoke H5 files, build records, and staging records as non-release output that release assembly and publication reject.
• Record ordered spine-stage and calibration progress, including elapsed time and aggregate entity counts.
• Propagate validated delivery evidence through release assembly and publication checks.
• Add repository provisioning, GitHub environment setup, integration tests, and operator documentation.
• Maintain matching versioned fixtures for downstream staging-data consumers.

Exact final household counts belong to the separate post-construction calibration and release workflow tracked in #898.

Acceptance criteria

• Existing US staging regression tests continue to pass.
• Version 2 producer and consumer contract tests pass.
• The integration workflow runs on every pull request that changes code and invokes its shell logic from tools/run_integration_tests.sh.
• The complete committed UK synthetic fixture runs every current spine transformation through the production command.
• The integration output reports total elapsed time and elapsed time for every transformation.
• Smoke output is explicitly identified as non-release and cannot be assembled or published as a release.
• Local-only staging tests validate the current run-scoped layout, ordered event stream, progress records, run discovery records, and permitted-file policy without requiring external credentials.
• Authenticated delivery uploads only permitted aggregate JSON records and does not upload the H5 dataset or source microdata.
• No production artifact or production release reference is modified by verification.
• The Microcosm implementation is published through a feature-branch pull request; production publication remains a separate operator action.

Historical authenticated transport evidence

On 2026-09-09, an explicitly non-release licensed-input smoke run used the former source-household-count option and successfully uploaded and read back five approved version 2 JSON records from policyengine/populace-uk-staging. It did not upload its H5 dataset or source microdata.

That run verifies repository authentication and the permitted-file policy. It is not evidence for a 100-household final dataset, and it predates the final run-scoped-only storage layout. The current layout is covered by the local integration and contract tests in #896.

Related work

• Add versioned UK staging telemetry #896 implements this staging-data contract and smoke verification.
• Complete UK exact-K release parity and make the runtime country-neutral #898 tracks shared exact-household-count calibration and its release requirements.