Handle multiple concurrent pip processes on the same environment #2361
Comments
|
Have you seen incorrect behaviour? I'd tend to assume it's probably OK unless there's evidence otherwise (disclaimer: I haven't looked at the code myself). |
|
My assumption is that it's going to do something wrong, it might end up OK in most scenarios, but in some scenarios it's going to break or do something nonsensical. We should probably have a lock on whatever environment we're currently working on. |
|
OK, cool. Concurrency is hard :-) |
|
Mostly pip seems to behave fine (it's not crashing) however I think that if the repository of installed packages is not stable it will be a real problem when the second process attempts to resolve versions, unless at least that part is done in a serial manner. For example, I ran a test with like 5 concurrent |
xavfernandez
changed the title
|
As rbtcollins pointed out, it "applies not just to virtualenvs as the other bug does, but to any target environment, including directories, alternate roots, and so on". |
ghost
mentioned this issue
|
Just to mention one simple scenario in that use case, that I previously encountered : Installing multiple packages concurrently in and as a result, some package are missing : But let me know if this is worth making a special issue for it... |
|
Just a side note on this, in case someone is stuck on the same problem, using setlock seems to fix the issue : The mutex / queueing behaviour works, however the lockfile remains there after execution. Not sure why, but this doesn't seem to cause any problem so far... |
|
Would it be enough if there's a single file in a global appdir ( |
pradyunsg
added
type: enhancement
|
This would break usecases where people invoke pip multiple times in parallel, which can make sense when using different virtualenvs, from some kind of script or build framework. It can also happen in the same virtualenv (my usecase) because pip is hooked up with a build system doing things in parallel (make). Also when someone invoke two pip instances at the same time, I would think the calls should be linearized (one wait for the completion of the other) instead of aborting, to keep whats currently working, working. |
pradyunsg
added
type: feature request
|
I think I faced a real issue with multiple Collecting attrs==19.1.0 (from -r requirements.txt (line 3))
1 location(s) to search for versions of attrs:
* https://pypi.org/simple/attrs/
Getting page https://pypi.org/simple/attrs/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/attrs/" in the cache
Request header has "max_age" as 0, cache bypassed
https://pypi.org:443 "GET /simple/attrs/ HTTP/1.1" 304 0
Skipping page https://pypi.org/simple/attrs/ because the GET request got Content-Type:
Given no hashes to check 0 links for project 'attrs': discarding no candidates
ERROR: Could not find a version that satisfies the requirement attrs==19.1.0 (from -r requirements.txt (line 3))It fails on different packages in different processes, and it works well if the only process started. $ pip --version
pip 19.2.3 |
|
That is the same case as #6970. |
|
FYI, if people are interested, we've implemented concurrent locking in Spack at the package level. You can run multiple instances of Spack in parallel, on the same node or on multiple nodes (with most shared filesystems -- provided locks are enabled). The processes will coordinate through See spack/spack#13100. Feel free to steal this stuff -- Spack's Apache-2.0 OR MIT. |
This adds PKG_BUILD_PARALLEL:=0 to packages that depend on host Python packages (HOST_PYTHON3_PACKAGE_BUILD_DEPENDS), because installing packages with multiple concurrent pip processes can lead to errors or unexpected results[1]. This also: * Move HOST_PYTHON3_PACKAGE_BUILD_DEPENDS definitions to before python3-package.mk is included * Update Python folder readme to include PKG_BUILD_PARALLEL:=0 [1]: pypa/pip#2361 Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This adds PKG_BUILD_PARALLEL:=0 to packages that depend on host Python packages (HOST_PYTHON3_PACKAGE_BUILD_DEPENDS), because installing packages with multiple concurrent pip processes can lead to errors or unexpected results[1]. This also: * Move HOST_PYTHON3_PACKAGE_BUILD_DEPENDS definitions to before python3-package.mk is included * Update Python folder readme to include PKG_BUILD_PARALLEL:=0 [1]: pypa/pip#2361 Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This adds PKG_BUILD_PARALLEL:=0 to packages that depend on host Python packages (HOST_PYTHON3_PACKAGE_BUILD_DEPENDS), because installing packages with multiple concurrent pip processes can lead to errors or unexpected results[1]. This also: * Move HOST_PYTHON3_PACKAGE_BUILD_DEPENDS definitions to before python3-package.mk is included * Update Python folder readme to include PKG_BUILD_PARALLEL:=0 [1]: pypa/pip#2361 Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This adds PKG_BUILD_PARALLEL:=0 to packages that depend on host Python packages (HOST_PYTHON3_PACKAGE_BUILD_DEPENDS), because installing packages with multiple concurrent pip processes can lead to errors or unexpected results[1]. This also: * Move HOST_PYTHON3_PACKAGE_BUILD_DEPENDS definitions to before python3-package.mk is included * Update Python folder readme to include PKG_BUILD_PARALLEL:=0 [1]: pypa/pip#2361 Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This adds PKG_BUILD_PARALLEL:=0 to packages that depend on host Python packages (HOST_PYTHON3_PACKAGE_BUILD_DEPENDS), because installing packages with multiple concurrent pip processes can lead to errors or unexpected results[1]. This also: * Move HOST_PYTHON3_PACKAGE_BUILD_DEPENDS definitions to before python3-package.mk is included * Update Python folder readme to include PKG_BUILD_PARALLEL:=0 [1]: pypa/pip#2361 Signed-off-by: Jeffery To <jeffery.to@gmail.com>
|
In mypy we run our test suite in parallel. Some of our tests for PEP 561 require pip to install packages, so we're recently run into some instability in our test suite. Note that each copy of pip is installed in its own venv. Sadly it seems that even with For now we will likely put a lock around package install, but it would be nice if @pradyunsg's suggestion of throwing an error if pip is already running would be implemented, it would certainly have made it easier to discover what was going wrong with our tests :) What would the next steps be to get to concurrent pip? I imagine a list of all the places pip relies on global state would be a good start. |
This should still work. Could you file an issue with the failure logs and a reproducer for the behaviour you're seeing? Multiple invocations of pip in separate environments should work already. That's literally what pip's own test suite does. |
|
We are facing this (or a similar) issue in our CI environment: This happens intermittently. This particular run was with We have only single pip execution per virtual environment, we do the same for 2 different virtual environments. My understanding is that this should work fine. Any ideas? How do I turn on some debug logging? Could the |
The tests run multiple instances, each instance using a different virtual environment. The issue looks similar to this pypa/pip#2361 It seems there are still issues with concurrent pip runs. Added flock to guard all pip executions. Fixes hazelcast#22599
The tests run multiple instances, each instance using a different virtual environment. The issue looks similar to this pypa/pip#2361 It seems there are still issues with concurrent pip runs. Added flock to guard all pip executions. Fixes hazelcast#22599 Backport of hazelcast#22829
The tests run multiple instances, each instance using a different virtual environment. The issue looks similar to this pypa/pip#2361 It seems there are still issues with concurrent pip runs. Added flock to guard all pip executions. Fixes #22599
The tests run multiple instances, each instance using a different virtual environment. The issue looks similar to this pypa/pip#2361 It seems there are still issues with concurrent pip runs. Added flock to guard all pip executions. Fixes #22599 Backport of #22829
I saw that pip 6 added file-based locking to the http cache to avoid multi-process contention. Something should be done to handle multiple packages being installed into the same virtualenv at the same time. For example if I had a concurrent makefile that installed independent
foo_webapp1andfoo_webapp2packages with a common dependency onfoo_core, what would happen right now? What should happen? I guess the easiest is just to abort the second process.The text was updated successfully, but these errors were encountered: