New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

setuptools 6.0 causes giant memory leaks in python often crashing the interpretter #259

Closed
bb-migration opened this Issue Sep 27, 2014 · 8 comments

Comments

Projects
None yet
1 participant
@bb-migration

bb-migration commented Sep 27, 2014

Originally reported by: sdague (Bitbucket: sdague, GitHub: sdague)


About 20% of the build environments in OpenStack are seeing MemoryError trying to pip install requirements / test-requirements. An example fail is here - http://logs.openstack.org/90/124590/2/check/gate-elastic-recheck-python27/b8c7724/console.html#_2014-09-27_12_19_36_039

When running locally I found that python process was rapidly growing to north of 5GB of used memory. Putting a setuptools<6 pin in requirements.txt made the issue go away.

The issue only cropped up after the setuptools 6.0 release, so I think there is a high correlation that it is related to that version.


@bb-migration

This comment has been minimized.

bb-migration commented Sep 27, 2014

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):


Your traceback implicates d2100af86513 or maybe one of the other refactorings:

  • c7eb24b975bd
  • 1aef200a4f1e
  • 9374ba53ff04
  • 7e6413a32780
  • 7b0a3ec2693c

I've pulled setuptools 6 from PyPI for now.

@bb-migration

This comment has been minimized.

bb-migration commented Sep 27, 2014

Original comment by sdague (Bitbucket: sdague, GitHub: sdague):


Thanks for the quick response. From the pattern of failing jobs, I expect that you need a certain complexity of requirements.txt. Probably requiring libraries that require other libraries in their own requirements.txt, as few projects with super simple requirements.txt seem to still be passing.

@bb-migration

This comment has been minimized.

bb-migration commented Sep 27, 2014

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):


I think I see the error in 9374ba53ff04. I think a single letter is missing. I tested the behavior in the command-line, but the behavior I committed is subtly different.

More importantly, there appears to be no test coverage of this behavior.

@bb-migration

This comment has been minimized.

bb-migration commented Sep 27, 2014

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):


Confirmed. The issue was in the use of partition and not rpartition. Ironically, that was the only function where I did add a test, but because it was a doctest, it wasn't ever invoked. That function goes into an infinite loop when called.

@bb-migration

This comment has been minimized.

bb-migration commented Sep 27, 2014

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):


Use rpartition here, essential to the algorithm. Fixes #259.

@bb-migration

This comment has been minimized.

bb-migration commented Sep 27, 2014

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):


Add doctesting of modules. Ref #259.

@bb-migration

This comment has been minimized.

bb-migration commented Sep 27, 2014

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):


Released as 6.0.1.

@bb-migration

This comment has been minimized.

bb-migration commented Sep 27, 2014

Original comment by sdague (Bitbucket: sdague, GitHub: sdague):


Thanks for the fast resolution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment