Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log maximum memory usage in main ERT thread #3357

Merged
merged 1 commit into from
Aug 3, 2022
Merged

Conversation

pinkwah
Copy link
Contributor

@pinkwah pinkwah commented May 5, 2022

Resolves: #2986

The terminology is a bit obtuse. RSS is "Resident Set Size", which is the amount of RAM that the program is using. This number does not include swap (when paging out) or on disk (because parts of the program haven't been loaded). I have chosen to use "Peak memory use" instead so that it's understandable without having to duckduckgo what RSS means.

Virtual memory usage is another metric that is possible to record, but is not useful for this purpose. Basically, the 64-bit address space doesn't actually need to be RAM, and the OS can map other stuff to that area. A simple example is mmap, which opens a file and lets the user access it as if it were in memory, but really it's still on disk. So instead of .read()/.write(), you can just read from a byte array directly. We don't care about this number though.

Compare results with /usr/bin/time -v ert test_run snake_oil.ert. The -v will record "Maximum RSS" which is the peak memory usage.

@pinkwah pinkwah self-assigned this May 5, 2022
@codecov-commenter
Copy link

codecov-commenter commented May 5, 2022

Codecov Report

Merging #3357 (8fe8215) into main (2a3dba0) will increase coverage by 0.00%.
The diff coverage is 92.85%.

@@           Coverage Diff           @@
##             main    #3357   +/-   ##
=======================================
  Coverage   63.52%   63.53%           
=======================================
  Files         603      603           
  Lines       46480    46494   +14     
  Branches     4140     4140           
=======================================
+ Hits        29526    29538   +12     
- Misses      15667    15669    +2     
  Partials     1287     1287           
Impacted Files Coverage Δ
src/ert_shared/main.py 87.08% <92.85%> (+0.35%) ⬆️
src/ert/data/record/_transformation.py 88.57% <0.00%> (-0.48%) ⬇️

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

@pinkwah pinkwah force-pushed the memuse branch 2 times, most recently from 7b85a91 to d81ec65 Compare May 6, 2022 09:06
@pinkwah pinkwah requested a review from oyvindeide May 9, 2022 08:52
ert_shared/main.py Outdated Show resolved Hide resolved
@sregales-TNO sregales-TNO added the telemetry Issues related to gathering telemetry-data from real usage label Jun 7, 2022
@pinkwah pinkwah force-pushed the memuse branch 4 times, most recently from 49b101a to 9fc896d Compare June 21, 2022 08:53
tests/ert_tests/shared/test_log_process_usage.py Outdated Show resolved Hide resolved
ert_shared/main.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@oyvindeide oyvindeide left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@oyvindeide
Copy link
Collaborator

If you rebase this again it should be good to go.

@pinkwah pinkwah merged commit 1037f24 into equinor:main Aug 3, 2022
@pinkwah pinkwah deleted the memuse branch August 3, 2022 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
telemetry Issues related to gathering telemetry-data from real usage
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Log maximum memory usage in ert
6 participants