Investigate UK Populace runtime Hugging Face download timeout in simulation API

[anth-volk]

Summary

A UK national simulation using the Populace default dataset timed out after 3600s in Modal while trying to materialize/download the UK Populace HDF5 at runtime.

This appears distinct from the US national Populace timeout pattern: the US failures reached reform simulation computation, while this UK failure was still in ensure_datasets -> create_datasets -> materialize_dataset_source -> hf_hub_download when Modal canceled the input.

Evidence

Logfire span:

• Project: anth-volk/policyengine-simulation
• run_id: e599f855-bd7c-43ec-b01f-d95367b5b9d9
• Trace ID: 019f1e5bdc5be2998a2dd2eef8a3ffb8
• Span: run_simulation
• Country/region: uk / uk
• Time period: 2026
• Start: 2026-07-01T15:46:04.507951Z
• End: 2026-07-01T16:46:04.593926Z
• Duration: 3600.085974101s
• Status: ERROR
• Exception: modal.exception.InputCancellation: Input was cancelled by user

Relevant stack excerpts from Logfire:

policyengine_api_simulation/simulation_runtime.py:_load_dataset
  country_module.ensure_datasets(...)
policyengine/tax_benefit_models/uk/datasets.py:244 ensure_datasets
  return create_datasets(...)
policyengine/tax_benefit_models/uk/datasets.py:114 create_datasets
  runtime_dataset = materialize_dataset_source(resolved_dataset)
policyengine/provenance/dataset_sources.py:112 materialize_dataset_source
  return hf_hub_download(... repo_type="dataset" ...)
huggingface_hub/file_download.py:1861
  xet_get(...)
/pkg/modal/_container_entrypoint.py:241
  raise InputCancellation("Input was cancelled by user")

The same trace also shows the first helper attempting the default/model-style HF URL and getting a 404:

https://huggingface.co/policyengine/populace-uk-private/resolve/populace-uk-2023-dd68c73-4aa4b14-20260619T023711Z/populace_uk_2023.h5
HTTPStatusError: 404 Not Found

It then falls back to hf_hub_download(... repo_type="dataset"), which is where the job remained until Modal's 3600s timeout.

Why this is concerning

The versioned Modal image is intended to install certified bundle datasets during image build:

uvx --from policyengine==<version> policyengine bundle install <version> --python /usr/local/bin/python --country us --country uk --data-dir /opt/policyengine/data --yes

At runtime, _load_dataset passes POLICYENGINE_DATA_FOLDER (/opt/policyengine/data) into country_module.ensure_datasets.

For UK, ensure_datasets checks for derived yearly files:

/opt/policyengine/data/<dataset_stem>_year_<year>.h5

If those are missing, it calls create_datasets, which materializes the source dataset URI from Hugging Face at request time.

The Logfire stack confirms that happened for the UK national run. That means either:

1. the Modal image did not contain the expected UK yearly file, or
2. the bundle installer wrote the UK Populace artifact under a different filename/shape than ensure_datasets checks, or
3. the image had only the raw populace_uk_2023.h5, while runtime expects populace_uk_2023_year_2026.h5 and therefore treats the dataset as absent.

Working diagnosis and confidence

• Very high confidence (~95%): the observed UK failure was a runtime Hugging Face/Xet download/materialization timeout, not simulation computation. The stack is inside materialize_dataset_source -> hf_hub_download -> xet_get at cancellation.
• High confidence (~85%): the runtime fell back to create_datasets because the expected local UK yearly HDF5 was missing or not recognized under /opt/policyengine/data.
• Moderate confidence (~65%): the root cause is a mismatch between what policyengine bundle install stores (populace_uk_2023.h5, derived from the artifact URI filename) and what uk.ensure_datasets checks (populace_uk_2023_year_2026.h5). This needs confirmation against the actual Modal image filesystem or the bundle receipt.

Suggested checks

1. Inspect the deployed policyengine-simulation-py4-18-5 image/container for:
   • /opt/policyengine/data/.policyengine-bundle-receipt.json
   • /opt/policyengine/data/populace_uk_2023.h5
   • /opt/policyengine/data/populace_uk_2023_year_2026.h5
2. Add a startup/deploy check that policyengine bundle status --data-dir /opt/policyengine/data --country uk passes for the exact runtime Python and expected country bundle.
3. Consider failing fast in the simulation API if a default certified bundle dataset is not locally available, rather than silently entering a runtime HF download path.
4. Align bundle installation and ensure_datasets expectations for Populace datasets:
   • either prebuild/install the yearly files that ensure_datasets expects, or
   • teach ensure_datasets to load/use the bundled raw Populace artifact directly when present.
5. If runtime fallback remains supported, make the first HF attempt use the dataset repo type directly for Populace releases to avoid the model-repo 404 before fallback.

Related broader observation

A broader Logfire scan over 2026-06-30 through 2026-07-01 found this as the only UK national timeout. Other UK national runs completed around 250-320s. This makes it look like a packaging/local-availability miss for this runtime/app/version rather than baseline UK simulation performance.