Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speedup 10: 25 % quicker _get_array_dicts and 10 % quicker _prep_data_for_correlation #536

Merged

Conversation

flixha
Copy link
Collaborator

@flixha flixha commented Jan 10, 2023

What does this PR do?

Implements:

  • 25 % speedup for utils.correlate._get_array_dicts (from 200 s to 154 seconds total time spent in my test)
  • 10 % speedup for utils.pre_processing._prep_data_for_correlation (from 702 s to 640 s total time spent)
  • (for comparison to the numbers above: 494 s were spent for correlating data in parallel on the GPU)

Main speedups with:

  • direct operations on UTCDateTime.__dict__['_UTCDateTime__ns'] (5x quicker for one +/- operation)
  • index-based access to trace in stream with stream.traces[j] instead of stream[j] (4.5x quicker)

Why was it initiated? Any relevant Issues?

For big datasets run on powerful hardware, the cumulative time spent in utils.correlate._get_array_dicts and utils.pre_processing._prep_data_for_correlation makes up a considerable part (>50 %) of a detection run. This PR tries to optimize the functions with some simple changes; I think further speed ups would need changes that are not so trivial...

This PR is part of the list of speedups in #522

PR Checklist

  • develop base branch selected?
  • This PR is not directly related to an existing issue (which has no PR yet).
  • [] All tests still pass.
    - [ ] Any new features or fixed regressions are be covered via new tests.
    - [ ] Any new or changed features have are fully documented.
  • Significant changes have been added to CHANGES.md.
    - [ ] First time contributors have added your name to CONTRIBUTORS.md.

@flixha
Copy link
Collaborator Author

flixha commented Jan 10, 2023

Sorry for the mess with 2 commits with the same message; I thought I had reset / removed the first one locally before pushing. As it's so small I think it's easier to leave it than trying to completely remove that commit from remote; anyways it's fixed by the last commit.

Copy link
Member

@calum-chamberlain calum-chamberlain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tiny change that I will push through and merge if this goes green.

eqcorrscan/utils/pre_processing.py Outdated Show resolved Hide resolved
very minor speed advantage
@calum-chamberlain calum-chamberlain merged commit cdde8c9 into eqcorrscan:develop Mar 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants