-
Notifications
You must be signed in to change notification settings - Fork 211
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory Leak Python 3.12.2 #628
Comments
There's been some known performance problems with coverage on Python 3.12, might be related: nedbat/coveragepy#1665 (comment) Perhaps try using |
Hi @nicoddemus! Thank you for referring to those references. I've played around with my configuration, and so far, I have the following to document. Attempt 1
This was the most straightforward thing to try. I changed my {
"name": "Test: Pytest",
"type": "python",
"request": "launch",
"module": "pytest",
"env": {
"PYDEVD_DISABLE_FILE_VALIDATION": "1",
"COVERAGE_CORE": "sysmon"
},
"justMyCode": true
} And made sure to disable "branch" in my pyproject.toml file. [tool.coverage.run]
# Whether to measure branch coverage in addition to statement coverage
branch = false The documentation states it doesn't work for branch coverage, therefor I turned it off. While keeping an eye on my memory usage, I'm writing this message with 18% in use while my tests are running on my Windows 11 system. After my tests hit 100%, the memory usage skyrockets like before. ConclusionI don't think Attempt 2
I'm aware that CPython underwent significant changes from version I removed I removed [tool.pytest.ini_options]
addopts = "--exitfirst -vs --junitxml htmlcov/pytest.xml" I also removed the django plugin plugins = [] Time to try again. ConclusionThat worked. Something is up with either Attempt 3I'll reinstall Installing collected packages: pytest-cov
Successfully installed pytest-cov-4.1.0 Lets fire it up! ConclusionThat also works. Let's add the first flag back but without adding the django plugin. Attampt 4I added ConclusionSo we are back at square one. Attempt 5Let's remove ConclusionNo luck. Attempt 6Perhaps the My [tool.pytest.ini_options]
addopts = "--exitfirst -vs --junitxml htmlcov/pytest.xml --cov-report xml" ConclusionI'm surprised... I was already writing my overall conclusion, but let's try to add the HTML report back! Attempt 7My [tool.pytest.ini_options]
addopts = "--exitfirst -vs --junitxml htmlcov/pytest.xml --cov-report html --cov-report xml" ConclusionHOLYMOLY THAT WORKED!. ... Is what I thought.. Only the XML file actually generated. Overall ConclusionAs mentioned in this ticket, I could run the Django test suite without coverage, but that doesn't work for my pipeline. I need the XML file for my pipeline! So just a few adjustments in the I'm not sure if I can be convinced that CPython is the issue here. My testing really points to the HTML generation of Edit: I just discovered that junitxml also aint working with how I thought my bandaid was put down... woops |
Hi @0x78f1935, Sorry it is late so I kinda skimmed through your post, but before going to bed I decided to leave another suggestion you can try: have you tried using |
Hi @nicoddemus, No worries! I really appreciate your time. Thank you for your response! I tried your suggestion! Firstly, I added the following section to my .vscode/launch.json: {
"name": "Test: Pytest Coverage directly",
"type": "python",
"request": "launch",
"module": "coverage",
"args": [
"run",
"-m",
"pytest",
],
"env": {
"PYDEVD_DISABLE_FILE_VALIDATION": "1",
"COVERAGE_CORE": "sysmon"
},
"justMyCode": true
}, My addopts currently looks like this: [tool.pytest.ini_options]
addopts = "--exitfirst -vs --junitxml htmlcov/pytest.xml --cov-report xml" We know that Running it once more results into: Unfortunately, the same issue. To make double sure, I'll remove Visual Studio Code out of the equation in the hope I can ctrl + c out of the memory leak. Running And it keeps rising. I straightup forgot to remove [tool.pytest.ini_options]
addopts = "--exitfirst -vs --junitxml htmlcov/pytest.xml Lets run it again. Test session starts (platform: win32, Python 3.12.2, pytest 8.0.0, pytest-sugar 1.0.0)
cachedir: .pytest_cache
django: version: 5.0.2, settings: backend.application.settings (from ini)
rootdir: xxx
configfile: pyproject.toml
plugins: anyio-4.2.0, django-4.8.0, sugar-1.0.0 That worked. Lets add back [tool.pytest.ini_options]
addopts = "--exitfirst -vs --junitxml htmlcov/pytest.xml --cov --cov-report xml And try again! ERROR: usage: __main__.py [options] [file_or_dir] [file_or_dir] [...]
__main__.py: error: unrecognized arguments: --cov --cov-report
inifile: xxx\pyproject.toml
rootdir: xxx So lets remove Unfortunately no luck |
Indeed seems related to |
Hey, I'm trying to make sense of this and I have some questions:
|
I'm happy to answer!
This issue occurs on both Windows and Unix systems, specifically Debian and Alpine. It doesn't seem to be dependent on the execution environment, as it happens consistently across architectures (64-bit). Where Alpine is my production environment, Debian is my github actions environment and Windows is my local development environment.
I have to acknowledge that the files are private, and the Python environment gets overwritten in the pipeline, although it's my image. You could use the same environment. Create a new Dockerfile, touch a new file, and tail it in the entrypoint.
When starting a container like this, you can use exec to run and check the Python environment. docker exec <container_id> python3.12 --version
docker exec <container_id> python3.12 -m pip freeze But then again, the environment gets overwritten, so I don't really think the image is very helpful in this case.
Yes, Ill put me addopts = "--exitfirst -vs --junitxml htmlcov/pytest.xml --cov --cov-report html --cov-report xml --cov-report term" Results: Test session starts (platform: win32, Python 3.12.2, pytest 8.0.0, pytest-sugar 1.0.0)
cachedir: .pytest_cache
django: version: 5.0.2, settings: backend.application.settings (from ini)
rootdir: xxx
configfile: pyproject.toml
plugins: anyio-4.2.0, cov-4.1.0, django-4.8.0, sugar-1.0.0
collected 42 items I guarantee that this issue also arises outside Visual Studio Code. When using the top command in Unix, I also observe my memory being consumed. I have 64GB of DDR4 RAM. The screenshots I shared of my Windows environment are merely indicative of the issue occurring again while I guide you through the steps I took to exclude other libraries. I hope this helps!
If that issue still persists after this main issue has been resolved (the memory leak), I'll create another ticket. However, I'm very confident that those XML files will generate after this has been resolved, just like my environment in Python 3.11, which functions normally. Edit: I'll look into a minimalistic environment, where I perhaps can reproduce the issue. |
I can't put my finger on it. While attempting to reproduce the issue with a fresh Django application and some tests, I'm unable to replicate the exact problem. When removing my tests from the stack, pytest-cov actually finishes with all the files. At this point, I'm wondering if there's something within my own tests causing the issue. It's strange that I don't have this problem with Python 3.11. I think the best thing I can do is start from scratch and slowly port all my components one by one until I have more details to share. |
I can confirm that issue. I could reproduce it with python 3.12.0 and 3.12.3. When I run pytest on my django application with It happens on this particular test when
Happens within my pipeline on Ubuntu24.04 with python3.12 as well as on my mac with python3.12.0 and python3.12.3. When I downgrade python to python3.11.9 pytest runs normaly again. All tested with:
@0x78f1935 Maybe that helps to narrow the issue down. |
@sebastian-muthwill , Thank you for your response. See my previous response:
I haven't found a fix yet. I just don't generate reports anymore with coverage. For now I run > coverage run I encounter the same issue with vanilla pytest and coverage running: > coverage report I kinda moved on @sebastian-muthwill . |
If this happens with coverage.py without pytest-cov, please open an issue in https://github.com/nedbat/coveragepy and include reproduction instructions. Many people here have showed fragments of their |
@sebastian-muthwill "I can confirm that issue. I could reproduce it with python 3.12.0 and 3.12.3." If you can provide us a way to reproduce the problem, that would be a great help. |
Summary
I conducted several tests with Python 3.11 with success and recently migrated to Python 3.12. I observed that on my Debian system, the SSH connection is lost after running my GitHub Actions pipeline. This issue consistently occurs after all tests have completed, and the results are about to be fetched.
Upon further investigation, I was able to reproduce this problem on a Windows system running Visual Studio Code.
As the repository is private, I will provide as much information as possible in this ticket to aid in troubleshooting.
Note: This issue persists until all available memory has been used.
The terminal is stuck at:
When I develop on my remote server and this happens, SSH simply times out. When I run this on my desktop, I can barely use my computer due to the memory leak, which prevents me from stopping the generation of the coverage.
During my investigation, I toggled various settings within my test suite and did the same with libraries. I removed xdist because I thought it might be related to multiprocessing, but this was not the case. What remained was:
I'm utilizing
pyproject.toml
and use various flags to start my tests.which should results into:
python -m pystest --exitfirst -vs --junitxml htmlcov/pytest.xml --cov --cov-report html --cov-report xml --cov-report term
Note: When running in Python3.11 this works fine.
When running on Python3.12 with only the flags
![image](https://private-user-images.githubusercontent.com/14298262/304760686-c162a588-b8d7-480c-8447-682820c5509c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0NTY2NzYsIm5iZiI6MTcyNDQ1NjM3NiwicGF0aCI6Ii8xNDI5ODI2Mi8zMDQ3NjA2ODYtYzE2MmE1ODgtYjhkNy00ODBjLTg0NDctNjgyODIwYzU1MDljLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIzVDIzMzkzNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTEyYzAwNTA0ODYzYzQyMTBjOTI4NTlkMDlhOTY1MzVhYTc1OWY3NDYzOTEwZmNkYTFkNmI4MWI3YjVhNWQ3ZDAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.2nYQL_C_ofSMD-_mIQ0GY0rCzAvOPKBA6bL0fswLHgg)
python -m pystest --exitfirst -vs
everything runs fine.You might have noticed by now this is a django application. When running the same tests but with the django suite, everything works fine, even in Pyhton3.12. The command would look like
![image](https://private-user-images.githubusercontent.com/14298262/304761207-d2344a0f-c710-4406-b192-63504d7a9408.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0NTY2NzYsIm5iZiI6MTcyNDQ1NjM3NiwicGF0aCI6Ii8xNDI5ODI2Mi8zMDQ3NjEyMDctZDIzNDRhMGYtYzcxMC00NDA2LWIxOTItNjM1MDRkN2E5NDA4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIzVDIzMzkzNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTU3YjUxM2FlMjBmY2NkNzAwZGVhMzZiZDNiY2Q4NzU2OTkwNGUzOGI3YzAzYjgwNDEwZmVmNzQ4ZWFmNWM3MDgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.jPdO0rSKGIJfL6Rcdlll6mYZLY9IXbt1ZnDKAEdprGM)
python manage.py test --debug-mode --noinput --pythonpath backend
Note: The last two images do not generate coverage data. (Django test suite and Pytest without coverage)
Expected vs actual result
I expect my tests to run just like Python3.11 without memory issues.
Reproducer
I don't necesarely have a reproducable environment, but I do have a base image which I use for my Django application. All requirements are in there eventho I listed them already. This image doesn't include any tests.
Versions
Python 3.12.2
Config
Do note the usage of
-n 8
, which was originally employed for xdist and has already been removed from my configuration to exclude xdist from the problem.Original config
Code
An example how my tests might look like:
The text was updated successfully, but these errors were encountered: