Problem with the HES-APC-2021 data set

[GiovanadelaCruz]

Hello!

I am having trouble working with the HES-APC-2021 dataset.

I have noticed that the 2021 file includes a different set of columns compared to previous and subsequent years, which raises the question of whether this might be a different dataset rather than HES-APC (Hospital Episode Statistics - Admitted Patient Care).

I would appreciate your guidance.

Thanks,
Giovana

[klharron]

Hi Giovana

We will look into this and get back to you!

Thanks
Katie

[z2an]

Hi Giovana

Thank you for your query.
May I ask if you are using Python?

When reading a Microsoft SQL Server table into a Python DataFrame e.g., using pandas or sqlalchemy, you may occasionally encounter ‘Operational Metadata’ columns in a table which appear unexpectedly. Such columns are used for processing, ingestion, monitoring, or debugging purposes, e.g., run_id, ingestion_status, is_processed, created_at etc

I have investigated this issue and found table APC_2021 only contained 30 additional columns when using Python script. I did not come across this issue when using R or SQL scripts. Therefore, I would suggest modifying your scripts to ignore these additional columns. Let me know how you get on.

Best wishes

Zan

[GiovanadelaCruz]

Hi Zan,

I'm pulling the data using R and the RODBC package. The APC_2021 file only has 30 columns, and they seem to match the ones you mentioned (processing, ingestion, monitoring, etc. columns). I'm not sure why this only happens with the 2021 data and not with the other years. Hopefully, you can help me figure it out.

Thanks a lot for your time.

Best wishes
Giovana

[tony-stone]

I think this may be an issue with how the ONS team have provisioned the data for your project.

I'm sure @z2an will be in contact.

I'm going to re-label this question as "ONS SRS" related rather than a "data-quality" issue.

[z2an]

Hi Giovana

Thank you for clarifying, and thanks Tony for your support.
It appears that the data source you're using is mistakenly reading the 'Operational Metadata' columns rather than the intended APC_2021 dataset.

To comply with confidentiality policies, please email ich.echild@ucl.ac.uk directly from your university email address, including your project number and a brief description of the issue. We will liaise with the ONS to help resolve the matter and follow up with you directly.

Thanks

Zan