-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pytest and TMPDIR housekeeping #6036
Comments
I would like to work on this |
@shawn-peng assigned |
* remove temporary isolated venv after isolated notebook/packaging test done * remove temporary venv after cloned env test done * remove temporary directories for figures after heatmap testings Fix pytest and TMPDIR housekeeping (quantumlib#6036)
This PR simply removes the temporary directories. Here are the options we have,
Here is some reasoning for the options we have for the isolated test cases. First, if I understand correctly, these isolated test cases are testing these notebooks or other test cases to be able to run from a base environment by automatically installing packages by themselves. So, the environment shouldn't have all packages set up. The second method takes about 10s to restore each environment. It's slower than just removing and copying a new one (about 5s). The pip-sync also takes about 8s still worse than simply copying a new one. |
@pavoljuhas to review the PR and @shawn-peng to verify that all files are cleaned up as per the last command in the original issue. |
Thanks @tanujkhattar, @pavoljuhas Here is what's left,
cirq-pytest takes most of the usage, here are what's left,
These will be reused later, so I think it would be better to keep them. Inside pytest-of-<username>
4 tmp* dir is empty, 4 browser temp dirs take little space
|
The top usage with 4 threads/workers is less than 5GB. Each isolated env is deleted immediately after the test is done. |
pytest leaves behind a LOT of temporary files
After executing all cirq unit tests with pytest the root temporary directory contains 31 leftover files and 38 leftover directories taking about 30G of disk space in total. The majority of that is taken by /tmp/cirq-pytest which has 59 virtual environments - 3 base environments and 56 per each isolated notebook test.
Some ideas how to reduce disk usage in isolated_notebook_test
pip install
commands run in the notebook should then keep their base environment unchanged.rsync -a --delete base-backup/ base/
before each test. This would probably need several base environments for each parallel pytest worker.Here is how I checked for the leftover files (docker tests are not set up on my workstation).
Note: my work tree had a change in the isolated_notebook_test.py which makes it execute all isolated notebook tests.
Cirq version
1.2.0.dev at 8b97aa5
The text was updated successfully, but these errors were encountered: