Where does TPMA data live?

What is the difference between the NHP model data and the TPMA explorer data?

Same source underlying both datasets. TPMA explorer is aggregated and standardised for rates accounting for population structures (age/sex standardised). This allows TPMAs to also be shared publicly in an app whilst NHP model data is more granular. Standardisation of TPMAs: https://connect.strategyunitwm.nhs.uk/nhp/project_information/modelling_methodology/activity_mitigators/direct_standardisation_of_tpmas.html

NHP model data has the TPMAs at individual patient level and is much more sensitive. For more information see the How are TPMAs generated page.

What NHP model data is available?

The NHP demand model is currently operating at trust level. Trusts using the model can only see the NHP model data for their organisation. Although we currently provide data at trust level, there are longer term plans to expand to ICBs or other types of provider areas.

Note: There are restrictions on what statistics are available, for example rates. Boundaries for areas do change in the NHS/Public health and these may affect underlying data that is available. Providers are also Trusts but there are also individual hospitals. Some areas have multiple hospital sites and specialities within a geography.

Where is the mitigator rates data for the TPMA explorer and how do I access it?

TPMA data is not downloadable or available in bulk to external users. It is only available internally in two places: UDAL Databricks and MLCSU Azure Blob Storage.

(Link to TPMA explorer app itself: Explore opportunities to reduce hospital care)

This data isn't different or separate to the NHP model inputs data. The data pipeline generates all the data in UDAL and saves it as parquet files on Azure Blob Storage.

Where is the TPMA explorer mitigator rates data and how do I access it?

Parquet files from MLCSU local machine

It's possible to use the R package azkit (maintained by SU) to access the data:

container <- azkit::get_container(
  # Ask for this value, put it in .Renviron
  Sys.getenv("AZ_INPUTS_CONTAINER")
) 

rates <- azkit::read_azure_parquet(
  container,
  "vX.Y/provider/rates.parquet"
)

TPMA explorer mitigator rates data on UDAL (dev version only)

spark.read.parquet("/Volumes/udal_lake_mart/newhospitalprogramme/files/inputs_data/dev/provider/rates.parquet")

Where is the NHP model data with TPMAs at patient level?

Tables on UDAL

https://github.com/The-Strategy-Unit/nhp_data/wiki/Data-Structure

In UDAL, udal_lake_mart.newhospitalprograme.default_apc_mitigators has data on TPMA, you can join to udal_lake_mart.newhospitalprograme.default_apc to get patient/spell info. From that, data is aggregated and standardised to create the data that goes into the tpma_explorer. That data is not stored in UDAL though.

Parquet files on local machine

container <- azkit::get_container(
  # Ask for this value, put it in .Renviron
  Sys.getenv("AZ_DATA_CONTAINER")
) 

mitigators <- azkit::list_files(data_container, "vX.Y.Z/ip_efficiencies_strategies/fyear=FYEAR/dataset=ZZZ", "parquet") |>
  purrr::map(\(x) azkit::read_azure_parquet(data_container, x))

You can replace ip_efficiencies_strategies with ip_activity_avoidance_strategies.

Is there a central repository for data about the TPMAs, e.g. codes and categories?

This TPMAs repository has the intention of being a central place to get the data, descriptions but possibly not code.

https://connect.strategyunitwm.nhs.uk/nhp/project_information/user_guide/mitigators_lookup.html is more technical around the codes used and lookups. This is linked from the tpma-explorer app.

Where are the layperson descriptions of the TPMAs, including references?

The NHP documentation has information on TPMAs with descriptions.

We must ensure that the English descriptions/definitions of the TPMAs get updated if and when the code definitions of the TPMAs change.

Where do the technical / code definitions of the TPMAs live?

All the coding for the TPMAs lives in the https://github.com/The-Strategy-Unit/nhp_data repository, particularly in the raw_data processing step

Why is some data in UDAL and some not? Longer term, is the aim to move all of the data to one place?

The data pipeline has to run in UDAL because that is where the HES data is supplied to us from NHS England. However, when we run the NHP model (currently using Docker containers running on Azure Container Instances), or create apps like the TPMA explorer (hosted on Connect), these cannot access the data on UDAL due to security restrictions. We therefore extract the data from UDAL into parquet files which live on Azure Blob Storage. This is the data that you access using {azkit}.

There is no long term plan to move all the data to one place. I don't think any of our external facing apps will ever be able to read directly from UDAL due to security restrictions, so there will always be some underlying need to export the data from UDAL. Additionally, the tables on UDAL cannot be easily versioned, whereas the parquet files are versioned.