RuntimeError: can't start new thread #11665
Comments
dcraigmv
added
S: needs triage
|
Can you run pip with /cc @willmcgugan since this is a threading-related exception coming out of rich. I don't know if this has been fixed / wrapped in a nicer exception in a newer version of rich; but it'd be nice for pip if it could catch a specific error raised by rich for this; instead of a generic RuntimeError coming out of |
|
Output of |
|
That's useful but not what I was looking for. I guess I should have been explicit. Can you run |
|
I'm sorry, i already had a debug command in my history. https://gist.github.com/dcraigmv/0ffe10635f154c92094a8b15d79ee6cd |
That's a new one on me. Google tells me it is due to having too many threads open in a process. It probably would be possible to catch that and disable auto refresh, but it does suggest that there are a lot of threads running in the process. Could there be some kind of thread leak in pip? Or maybe threads not being joined early enough? |
|
This is the first instance of pip trying to start a thread and failing, IIUC, so it's unlikely to be that. I don't think anything other that rich uses threads within pip, but that might not be 100% accurate. It's more likely to be that the system is hitting the OS thread limit already. |
|
Injecting |
|
Ah, so its a system limit. @dcraigmv You said that this effects larger packages only? Can you see from htop if something is using a lot of threads prior to running pip? |
|
Only in that i first saw this when installing from a requirements.txt, it installed a couple of packages and then failed at torch. I know torch is big and 2+2 lead me to think rich has enough time/payload size to auto refresh and start a thread.. then fail.
I will get back onto the server in a few hours to run it again and check process list. |
|
IIRC, pip only shows the progress bar above a certain size. I'm AFK at the moment, but that'd explain the "only big packages" issue here. |
pradyunsg
added
project: vendored dependency
Huh, that doesn't look like the debug flag was passed. |
|
Any updates on this? I have a similar issue (I think) when trying to use pip to install some packages during a docker build within CircleCI. CircleCI uses cimg/python:3.10 image to create the remote docker environment and then begins the docker build from the base image jupyter/tensorflow-notebook:hub-3.1.0. The next step in the docker file is to install blosc and poetry using pip. It downloads the blosc wheel and then immediately fails: |
|
In case this can help: I stumbled upon this issue as I had the same issue with the same stack. And I solved it by upgrading my docker version. I had this version when I had the issue. and I have now: |
|
I'm gonna close this out now -- there's nothing actionable for pip here and it's a system-limitation of whatever system you're running on. Consider looking into why a new thread can't be created on your system, by Python itself. |
pradyunsg
added
resolution: not a bug
This comment was marked as resolved.
This comment was marked as resolved.
Description
Installing larger packages such as torch cause pip to fail while it renders a progress bar.
Expected behavior
Collecting [package-name]
Downloading [package-name]
|| progress bar ||
Done
pip version
22.0.2
Python version
3.10
OS
Ubuntu Jammy
How to Reproduce
docker run --rm ubuntu:jammy sh -c "apt update && apt install -y python3-pip && pip install torch"Output
Code of Conduct
The text was updated successfully, but these errors were encountered: