Skip to content

Post-processing: Refactor low-level data representation into a numpy array of size (n_ranks, n_threads)#234

Draft
EdHone wants to merge 10 commits intoMetOffice:mainfrom
EdHone:py-vernier-arrays
Draft

Post-processing: Refactor low-level data representation into a numpy array of size (n_ranks, n_threads)#234
EdHone wants to merge 10 commits intoMetOffice:mainfrom
EdHone:py-vernier-arrays

Conversation

@EdHone
Copy link
Collaborator

@EdHone EdHone commented Mar 16, 2026

Description

This PR changes the VernierCalliper class data structure from a set of Lists into numpy.arrays with a shape of (n_ranks, n_threads). This will enable data from multiple threads to be included into a single calliper. This fundamental change to the data structure has changed the required formatting of the unit tests but the numbers remain the same. Output from CLI tools is unaffected.

Additionally - as implementing the aggregate function with numpy arrays proved difficult, this function has been removed and the functionality baked into the the _load_from_directory function that called it, directly.
This change could be implemented in a separate PR if needed

Linked issues

Closes #232

Type of change

  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How has this been tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • New tests have been added
  • Tests have been modified to accommodate this change

Checklist:

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes, for both debug and optimised builds

@github-actions github-actions bot added the cla-required The CLA has not yet been signed by the author of this PR - added by GA label Mar 16, 2026
@github-actions github-actions bot added cla-signed The CLA has been signed as part of this PR - added by GA and removed cla-required The CLA has not yet been signed by the author of this PR - added by GA labels Mar 17, 2026
@EdHone
Copy link
Collaborator Author

EdHone commented Mar 17, 2026

Loading data is horribly slow at the moment - working out why

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed The CLA has been signed as part of this PR - added by GA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Custom Issue]: Post-processing - Refactor internal representation of vernier data for better thread/rank representation

1 participant