Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

speedup get_source with lookupdict #599

Merged
merged 9 commits into from Dec 9, 2021
Merged

Conversation

JoranAngevaare
Copy link
Member

@JoranAngevaare JoranAngevaare commented Nov 29, 2021

What is the problem / what does the code in this PR do

Speedup #590 with ~40x. We did a lot of checks twice since we went all the way to the lowest dependency for each of the dependencies. This means that e.g. if you have a plugin depending on records_a and records_b which both depend on raw_records you would check raw_records twice.

Rather than using a set, we can also use a lookup dict to keep track of what we have seen. This speeds up the function by ~40x.

Additional benefits

  • Allow multiple targets st.get_source('029000', ('cuts_basic', 'event_info'))
  • Fix getting dependencies from CutMerger plugins (depends_on as class property)
  • Informative function stored_dependencies

Example 1

What sources do we need to start for making 'cuts_basic' and 'event_info':

st = cutax.xenonnt_v5()
st.get_source('029000', ('cuts_basic', 'event_info'))
# gives
{'cut_daq_veto', 'event_basics', 'event_pattern_fit', 'event_positions'}

This means 'cut_daq_veto', 'event_basics', 'event_pattern_fit', 'event_positions' are stored and we can compute 'cuts_basic' and 'event_info' for run '029000'.

Example 2

or get a list of plugins that are needed and stored:

st.stored_dependencies('029000', 'cuts_basic')
# gives
{'cuts_basic': False,
 'cut_daq_veto': True,
 'event_info': False,
 'corrected_areas': False,
 ... }

Meaning we still need to make 'event_info' and 'corrected_areas' to compute cuts_basic whereas we already have cut_daq_veto computed.

@JoranAngevaare JoranAngevaare changed the title speedup with lookupdict speedup get_source with lookupdict Nov 29, 2021
@JoranAngevaare JoranAngevaare added the enhancement New feature or request label Nov 29, 2021
@JoranAngevaare JoranAngevaare marked this pull request as ready for review November 29, 2021 14:37
Copy link
Contributor

@ershockley ershockley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works very well, thanks @JoranAngevaare! (and sorry for the delay)

Copy link
Collaborator

@WenzDaniel WenzDaniel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a typo I will commit and merge.

strax/context.py Outdated Show resolved Hide resolved
@JoranAngevaare JoranAngevaare merged commit 28d1806 into master Dec 9, 2021
@JoranAngevaare JoranAngevaare deleted the speedup_get_source branch December 9, 2021 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants