Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Produce plots for memory usage #137

Closed
mdboom opened this issue Feb 26, 2024 · 2 comments
Closed

Produce plots for memory usage #137

mdboom opened this issue Feb 26, 2024 · 2 comments
Assignees

Comments

@mdboom
Copy link
Contributor

mdboom commented Feb 26, 2024

It would be interesting to see memory usage plots, just as we have time plots.

The existing data has max_rss for each of the benchmark runs -- as a start, we can use that (though it's pretty large-grained). If that doesn't seem helpful, maybe we investigate add a small amount of memory tracking to the pystats and using that.

Cc: @brandtbucher (for the idea)

@mdboom mdboom self-assigned this Feb 26, 2024
@mdboom
Copy link
Contributor Author

mdboom commented Feb 28, 2024

Added in #138.

This is based on the “max_rss” support that pyperf has had for a long time, for which we already have data. This is pretty “coarse grained” and effected by other things going on on the system. On the other hand, this is probably what really matters when most people consider “running out of memory”.

As shown by this A/A test, the variability in the results is only 0.16% overall, so is much better than the ~2% we see for timings.

If we need a more accurate number, we can add stats about the size of all objects allocated / deallocated (probably as part of the pystats system).

We also don't have any data on Windows, since pyperf doesn't support the equivalent system call there.

The summary tables now include a memory usage change number (the last number (Y.Zx m)):

image

Each benchmarking summary page includes the memory change number and a link to a plot of per-benchmark memory change against its immediate base.

image

image

Lastly, there is a memory longitudinal summary page showing the change of memory over time, and under different configurations:

image

image

@mdboom mdboom closed this as completed Feb 28, 2024
@brandtbucher
Copy link
Member

Awesome! See python/cpython#116017 for further discussion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants