-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf: use lru_cache for polstr and variants #1250
Conversation
Ech. It's not working because the polarization numbers can be input as arrays, which I didn't realize. Currently this works fine for hera_cal because we never do this, but it won't work in general for pyuvdata. Unless you have a good idea about how to do this more generally, @bhazelton or @mkolopanis, I'll just close this PR and do it in hera_cal. |
can we ditch the deep copy operation? |
@bhazelton possibly, I'll have a check of the logic. |
@plaplant suggests creating a private function with the decorator to be called by the existing function which can convert from an array or list to a tuple as needed. |
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #1250 +/- ##
=======================================
Coverage 99.91% 99.91%
=======================================
Files 33 33
Lines 18655 18677 +22
=======================================
+ Hits 18639 18661 +22
Misses 16 16
Continue to review full report at Codecov.
|
a45c41d
to
0032ceb
Compare
@bhazelton I added @plaplant as a reviewer as well, since he had ideas about this :-) But in any case it should be ready to go. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks reasonable to me, not sure why pre-commit is so unhappy...
setup.cfg
Outdated
@@ -5,7 +5,7 @@ description_file = README.md | |||
# B905 is for using zip without the `strict` argument, which was introduced in | |||
# python 3.10. We should probably add this check (remove it from the ignore) when we | |||
# require 3.10. | |||
ignore = W503, E203, N806, B905, B907 | |||
ignore = W503, E203, N806, B905, B907, B028 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think B028
is the same as B907
. It started out as B028 but then was converted to B907 in flake8-bugbear version 23.1.17 . We now require that version or newer, so I don't think we need B028
in this list. See: https://github.com/PyCQA/flake8-bugbear#23117
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. Thanks!!
Description
Applies the
lru_cache
decorator to some utility functions, especially thepolnum2str
function and its variants. These are good functions for this decorator, as they have a very limited set of possible input parameters, so do not take much extra cache memory.Motivation and Context
It turns out that these functions can take a LOT of the total time for some
hera_cal
scripts, because they are called every time you request a baseline-pol key from aHERAData
object. Thus, if you loop over all baselines in a file to get their data, you end up callingpolstr2num
like 65k times. Sincepolstr2num
does a deepcopy of a dict, it actually takes a non-negligible amount of time (for example, the delay filter was taking ~20min for a 2-integration file, of which about 7min was taken by thepolstr2num
function).Types of changes
Checklist:
Bug fix checklist: