Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved performance of cf.read "select" parameter for PP/UM files #201

Closed
davidhassell opened this issue Apr 26, 2021 · 0 comments · Fixed by #210
Closed

Improved performance of cf.read "select" parameter for PP/UM files #201

davidhassell opened this issue Apr 26, 2021 · 0 comments · Fixed by #210
Assignees
Labels
enhancement New feature or request performance Relating to speed and memory performance um/pp Relating to UM or PP format files
Milestone

Comments

@davidhassell
Copy link
Collaborator

davidhassell commented Apr 26, 2021

When selecting fields with the "select" parameter of cf.read with PP/UM file inputs, the currently selection occurs after all of the input files have been processed. This can be made much more efficient by avoiding processing the non-selected fields.

E.g.

import re
import cf
f = cf.read('~/aaadpa#pg000003385c1+.pp', select=["stash_code=3237", "stash_code=1"])
g = cf.read('~/aaadpa#pg000003385c1+.pp', select=re.compile('^stash_code=\d\d\d\d$'))
@davidhassell davidhassell added the enhancement New feature or request label Apr 26, 2021
@davidhassell davidhassell added this to the 3.9.0 milestone Apr 26, 2021
@davidhassell davidhassell self-assigned this Apr 26, 2021
@davidhassell davidhassell added the performance Relating to speed and memory performance label Apr 26, 2021
@davidhassell davidhassell added the um/pp Relating to UM or PP format files label Jun 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request performance Relating to speed and memory performance um/pp Relating to UM or PP format files
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant