setuptools 6.0 causes giant memory leaks in python often crashing the interpretter

[bb-migration]

Originally reported by: sdague (Bitbucket: sdague, GitHub: sdague)

---

About 20% of the build environments in OpenStack are seeing MemoryError trying to pip install requirements / test-requirements. An example fail is here - http://logs.openstack.org/90/124590/2/check/gate-elastic-recheck-python27/b8c7724/console.html#_2014-09-27_12_19_36_039

When running locally I found that python process was rapidly growing to north of 5GB of used memory. Putting a setuptools<6 pin in requirements.txt made the issue go away.

The issue only cropped up after the setuptools 6.0 release, so I think there is a high correlation that it is related to that version.

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/259

[bb-migration]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Your traceback implicates d2100af86513 or maybe one of the other refactorings:

• c7eb24b975bd
• 1aef200a4f1e
• 9374ba53ff04
• 7e6413a32780
• 7b0a3ec2693c

I've pulled setuptools 6 from PyPI for now.

[bb-migration]

Original comment by sdague (Bitbucket: sdague, GitHub: sdague):

---

Thanks for the quick response. From the pattern of failing jobs, I expect that you need a certain complexity of requirements.txt. Probably requiring libraries that require other libraries in their own requirements.txt, as few projects with super simple requirements.txt seem to still be passing.

[bb-migration]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I think I see the error in 9374ba53ff04. I think a single letter is missing. I tested the behavior in the command-line, but the behavior I committed is subtly different.

More importantly, there appears to be no test coverage of this behavior.

[bb-migration]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Confirmed. The issue was in the use of partition and not rpartition. Ironically, that was the only function where I did add a test, but because it was a doctest, it wasn't ever invoked. That function goes into an infinite loop when called.

[bb-migration]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Use rpartition here, essential to the algorithm. Fixes #259.

[bb-migration]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Add doctesting of modules. Ref #259.

[bb-migration]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Released as 6.0.1.

[bb-migration]

Original comment by sdague (Bitbucket: sdague, GitHub: sdague):

---

Thanks for the fast resolution!