Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-33331: Monitor memory usage at select steps of BPS submission #112

Merged
merged 2 commits into from Feb 22, 2022

Conversation

mxk62
Copy link
Contributor

@mxk62 mxk62 commented Feb 9, 2022

Checklist

  • ran Jenkins
  • added a release note for user-visible changes to doc/changes

@mxk62 mxk62 force-pushed the tickets/DM-33331 branch 2 times, most recently from db4ce54 to 957e7ab Compare February 9, 2022 16:04
@codecov
Copy link

codecov bot commented Feb 9, 2022

Codecov Report

Merging #112 (e453de1) into main (63a1f4f) will increase coverage by 0.34%.
The diff coverage is 99.13%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #112      +/-   ##
==========================================
+ Coverage   91.51%   91.86%   +0.34%     
==========================================
  Files          38       40       +2     
  Lines        2417     2532     +115     
==========================================
+ Hits         2212     2326     +114     
- Misses        205      206       +1     
Impacted Files Coverage Δ
tests/test_usage.py 96.15% <96.15%> (ø)
python/lsst/utils/timer.py 96.96% <100.00%> (+0.70%) ⬆️
python/lsst/utils/usage.py 100.00% <100.00%> (ø)
tests/test_timer.py 99.48% <100.00%> (+0.16%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 63a1f4f...e453de1. Read the comment docs.

@ktlim
Copy link
Contributor

ktlim commented Feb 9, 2022

Please fix the ticket number in the PR title. (While the branch appears correct, I think some tools rely on the title.)

@mxk62 mxk62 changed the title DM-3333: Monitor memory usage at select steps of BPS submission DM-33331: Monitor memory usage at select steps of BPS submission Feb 10, 2022
@mxk62
Copy link
Contributor Author

mxk62 commented Feb 10, 2022

@ktlim Fixed.

Copy link
Member

@timj timj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many of my comments are about reorganizing things as mentioned on the BPS review. I also think that the log output for time_this can be simplified.

One other thing that we might want to do is to remove the getrusage call out of logInfo and move that into the new usage.py. If you make it return a DataClass with all the entries that logPairs uses and then the get_peak_usage could either disappear or else call that and unpack the information from the DataClass.

python/lsst/utils/timer.py Outdated Show resolved Hide resolved
python/lsst/utils/timer.py Outdated Show resolved Hide resolved
python/lsst/utils/timer.py Outdated Show resolved Hide resolved
python/lsst/utils/timer.py Outdated Show resolved Hide resolved
python/lsst/utils/timer.py Outdated Show resolved Hide resolved
python/lsst/utils/timer.py Outdated Show resolved Hide resolved
python/lsst/utils/timer.py Outdated Show resolved Hide resolved
python/lsst/utils/timer.py Outdated Show resolved Hide resolved
@mxk62 mxk62 force-pushed the tickets/DM-33331 branch 8 times, most recently from a8a791b to b0b8ffb Compare February 21, 2022 20:22
Copy link
Member

@timj timj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. Some minor comments.

python/lsst/utils/timer.py Show resolved Hide resolved
python/lsst/utils/timer.py Outdated Show resolved Hide resolved
python/lsst/utils/usage.py Outdated Show resolved Hide resolved
python/lsst/utils/timer.py Outdated Show resolved Hide resolved
python/lsst/utils/timer.py Outdated Show resolved Hide resolved
python/lsst/utils/timer.py Outdated Show resolved Hide resolved
python/lsst/utils/timer.py Outdated Show resolved Hide resolved
tests/test_timer.py Show resolved Hide resolved
python/lsst/utils/timer.py Outdated Show resolved Hide resolved
@mxk62 mxk62 force-pushed the tickets/DM-33331 branch 4 times, most recently from d9bc7da to cbabf2f Compare February 22, 2022 21:15
Moved functions related to measuring memory consumption in a separate
module lsst.utils.usage.
@mxk62 mxk62 merged commit 79a716f into main Feb 22, 2022
@mxk62 mxk62 deleted the tickets/DM-33331 branch February 22, 2022 23:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants