Where does TPMA data live?

https://github.com/The-Strategy-Unit/nhp_data/wiki/Data-Structure is incomplete we also want to record inputs-data and model-data as well as the tables on UDAL, and the reference files on the TPMAs repo. Possibly the TPMA descriptions on project info as well.

What is the difference between the NHP model data and the TPMA explorer data?

Same source underlying, same manipulation but TPMA is aggregated and standardised for rates accounting for population structures (age/sex standardised). This allows TPMA to also be shared publicly in an app whilst NHP model data is more granular for specific Provider work and available only to the Provider that owns that data.

By Provider this can mean originally hospitals but this has been (perhaps will be) expanded to ICBs or other types of provider areas. There are restrictions on what statistics are available, for example rates. Boundaries for areas do change in the NHS/Public health and these may affect underlying data that is available. Providers are also Trusts but there are also individual hospitals. Some areas have multiple hospital sites and specialities within a geography.

TMPA data though is not downloadable or available in bulk.

TPMAs are in the NHP model data.

What do we mean by "rates" and how do these relate / how are these used in the Inputs app and/or the TPMA Explorer app?

Where is the data for the TPMA explorer and how do I access it?

(Link to TPMA explorer app itself: Explore opportunities to reduce hospital care)

This data isn't different or separate to NHP model for outputs. The inputs pipeline generates all the data (includes rates) goes to Azure (central repository for data) and then is presented in these apps.

Where is the NHP model data and how do I access it?

Parquet files

It's possible to use the R package azkit (maintained by SU) to access the data:

container <- azkit::get_container(
  # Ask for this value, put it in .Renviron
  Sys.getenv("AZ_INPUTS_CONTAINER")
) 

rates <- azkit::read_azure_parquet(
  container,
  "vX.Y/provider/rates.parquet"
)

Tables on UDAL

https://github.com/The-Strategy-Unit/nhp_data/wiki/Data-Structure

In UDAL, udal_lake_mart.newhospitalprograme.default_apc_mitigators has data on TPMA, you can join to udal_lake_mart.newhospitalprograme.default_apc to get patient/spell info. From that, data is aggregated and standardised to create the data that goes into the tpma_explorer. That data is not stored in UDAL though.

Is there a central repository for data about the TPMAs, e.g. codes and categories?

Might not be the definitive answer but the NHP documentation has information on TPMAs with descriptions.

Exploring this a bit more it's metadata for the TPMAs and explaining what each includes.

https://connect.strategyunitwm.nhs.uk/nhp/project_information/user_guide/mitigators_lookup.html is more technical around the codes used and lookups. This is linked from the tpma-explorer app.

This repository has the intention of being a central place to get the data, descriptions but possibly not code.

Where are the layperson descriptions of the TPMAs, including references?

How do we ensure that the English descriptions/definitions of the TPMAs get updated if and when the code definitions of the TPMAs change?

Wonder if this section should live under "What is a TPMA?" ?

Where do the technical / code definitions of the TPMAs live?

And how useful are these?

[TODO] link to other page about where the data originates before the models.

Difficulties with data access

Some data is on UDAL, some isn't but analysis requires combining of the two data locations. Ideal would be to be in both places but that could create syncing issues. Data needs to be on UDAL to be used and cannot be brought out to do that work.

TMPA analysis doesn't always need other data.

Difficulties can arise when slight changes to the TPMA definitions are needed to be analysed, for example the TMPA may say 28 days but the analysis needs the same logic but with 29 days. Further to that the existing agreed TPMA data needs to go back into UDAL to be compared as UDAL doesn't have a facility to extract the data for analysis.

Code used for reproducing TMPAs https://github.com/The-Strategy-Unit/nhp_data/tree/28e223627090c5ec49d54b2d4d0a8f34ea39be1a/src/nhp/data/raw_data/mitigators to produce new TPMA, aggregating at provider level, standardising functions from NHP_data python package to apply the logic to replicates.

Standardisation of TPMAs: https://connect.strategyunitwm.nhs.uk/nhp/project_information/modelling_methodology/activity_mitigators/direct_standardisation_of_tpmas.html

UDAL use of python/R bespoke packages can require more work than local use can be.

Why is some data in UDAL and some not? Longer term, is the aim to move all of the data to one place?

The data pipeline has to run in UDAL because that is where the HES data is supplied to us from NHS England. However, when we run the NHP model (currently using Docker containers running on Azure Container Instances), or create apps like the TPMA explorer (hosted on Connect), these cannot access the data on UDAL due to security restrictions. We therefore extract the data from UDAL into parquet files which live on Azure Blob Storage. This is the data that you access using {azkit}.

There is no long term plan to move all the data to one place. I don't think any of our external facing apps will ever be able to read directly from UDAL due to security restrictions, so there will always be some underlying need to export the data from UDAL. Additionally, the tables on UDAL cannot be easily versioned, whereas the parquet files are versioned.