Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up getFiles by start/stop time (Closes #23) #31

Merged
merged 6 commits into from Oct 14, 2020

Conversation

jtniehof
Copy link
Member

@jtniehof jtniehof commented Oct 5, 2020

This PR speeds up the calculation of possible output files and thus closes #23. This works by JOINing the file table with a table containing Unix times and doing the search using that table. If that table doesn't exist, the old (slower) code will be used. Included is a script to migrate databases by adding this table and populating it with existing file entries; eventually this script can be expanded to perform other migration tasks (e.g. the yesterday/tomorrow columns) as part of #7.

PR Checklist

  • Pull request has descriptive title
  • Pull request gives overview of changes
  • New code has inline comments where necessary
  • Any new modules, functions or classes have docstrings consistent with dbprocessing style
  • Major new functionality has appropriate Sphinx documentation
  • (N/A, see below) Added an entry to CHANGELOG if fixing a major bug or providing a major new feature
  • New features and bug fixes should have unit tests
  • Relevant issues are linked in the description (e.g. See issue # or Closes #)

No changelog entry since it doesn't exist, release notes pending (#28)

@jtniehof jtniehof requested a review from balarsen October 5, 2020 20:49
@jtniehof jtniehof requested review from dnadeau-lanl and removed request for balarsen October 13, 2020 21:39
@jtniehof
Copy link
Member Author

@dnadeau-lanl, @balarsen has requested that you do the review as part of his stepping back from the project, so I've changed the request.

@jtniehof
Copy link
Member Author

Plan is to roll this out to PSP/ISOIS product as part of the current downlink. I've tested it in isolation as much as possible and I feel pretty good about it, and this'll give it a good beating in production. I think it can still be merged to master; doesn't need to wait for further testing.

Science data for this downlink should be completed by Friday 16 October. I'll spend some time next Monday reviewing the logs and results just in case anything crops up, and assuming that's good it can deploy to ECT. If anything does come up, I'll get in a unit test to catch it and either add to this PR or open another one.

@dnadeau-lanl dnadeau-lanl merged commit b5f30ef into spacepy:master Oct 14, 2020
@jtniehof jtniehof deleted the speedup_work branch October 15, 2020 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ProcessQueue "Command Build Progress" is slow
2 participants