Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Give RunDB an option to find files in storage #1244

Merged
merged 2 commits into from Aug 29, 2023
Merged

Conversation

dachengx
Copy link
Collaborator

@dachengx dachengx commented Aug 27, 2023

Give RunDB an option to find files in storage but not in database in find_several function.

What does the code in this PR do / what does it improve?

Historically, the RunDB knows where are all the available files located on servers and assumes they are all available. But if we have multiple partitions, then RunDB will provide runs in the database but not available.

This happens a lot when running strax.Context.select_runs, in the strax.StorageFrontend.find_several function. So this PR gives RunDB an option to run strax.StorageFrontend.find_several directly.

Can you briefly describe how it works?

Can you give a minimal working example (or illustrate with a figure)?

Please include the following if applicable:

  • Update the docstring(s)
  • Update the documentation
  • Tests to check the (new) code is working as desired.
  • Does it solve one of the open issues on github?

Notes on testing

  • Until the automated tests pass, please mark the PR as a draft.
  • On the XENONnT fork we test with database access, on private forks there is no database access for security considerations.

All italic comments can be removed from this template.

@coveralls
Copy link

coveralls commented Aug 27, 2023

Coverage Status

coverage: 93.548% (+0.05%) from 93.502% when pulling 69df71e on check_metadata_rundb into 3258746 on master.

@JYangQi00
Copy link
Contributor

Thanks Dacheng, I tried to use st.select_runs with _find_in_storage = False and with _find_in_storage = True, however, I'm having a bit of trouble understanding the output. I'd like to find some runs that cannot be loaded when found using st.select_runs with _find_in_storage=False. Then I would expect that when _find_in_storage=True, these runs do not show up when st.select_runs is called. Is my interpretation of this fix correct?

However, instead I find that when using _find_in_storage=False, I do not get any extra runs compared to when _find_in_storage=True. I also specified that peak basics should be available when using select_runs. Is there an example of a run+data_type that cannot be loaded despite the output of select_runs saying that it can? I knew there were some in the past but I haven't found any this time around.

@dachengx
Copy link
Collaborator Author

However, instead I find that when using _find_in_storage=False, I do not get any extra runs compared to when _find_in_storage=True. I also specified that peak basics should be available when using select_runs. Is there an example of a run+data_type that cannot be loaded despite the output of select_runs saying that it can? I knew there were some in the past but I haven't found any this time around.

Like the 050034-peaklets-ui5hguaz2k of offline context, but you need to be on a midway computing node like jupyter notebook.

@dachengx dachengx merged commit 185dad3 into master Aug 29, 2023
8 of 9 checks passed
dachengx added a commit that referenced this pull request Aug 29, 2023
…base in find_several function (#1244)"

This reverts commit 185dad3.
dachengx added a commit that referenced this pull request Aug 30, 2023
…base in find_several function (#1244)"

This reverts commit 185dad3.
@dachengx dachengx deleted the check_metadata_rundb branch August 30, 2023 16:13
dachengx added a commit that referenced this pull request Aug 30, 2023
…base in find_several function (#1244)" (#1248)

This reverts commit 185dad3.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants