Poetry now does not add the package.dependencies

[martingaldeca]

• Poetry version:1.5.0
• Python version: >3.10
• OS version and name: Ubuntu 22.04
• pyproject.toml: private

Issue

I was using the poetry version 1.4.2, when adding a new package from a private pypi it adds also the package dependencies well as you can see here: (I remove the private values, but I promise that are the same for the other example)

When using poetry version 1.5.0, for the same package, poetry did not added the dependencies, also the description was not added and the python version not specified:

So in that case for us this is not a minor update, this is a major update as it is breaking some of our dependencies.

[dimbleby]

I use private dependencies and repositories, and poetry 1.5.0 works fine

without a way to reproduce this there's not going to be very much that anyone can do.

[martingaldeca]

We use poetry inside docker. The image we use is public.ecr.aws/docker/library/python:3.10, then inside the container, we use poetry add.

[cquick01]

Also seeing the same behavior on Poetry 1.5.0, Python 3.9+. We have several Python packages hosted in a private repo, and when trying to add them to a new project with poetry add ..., the package itself is installed but no dependencies are resolved. Note that the dependencies that are missing are just normal public PyPi packages.

Also noticed as well that the descriptions of our private packages are now empty in our 'new project' lock files.

I have my sources defined as follows, if it helps.

[[tool.poetry.source]]
name = "private_pypi"
url = "https://pypi.domain.tld/simple/"
priority = "supplemental"

[[tool.poetry.source]]
name = "PyPI"
priority = "primary"

---

Edit: I created a simple project to reproduce the issue. See https://github.com/cquick01/cq-test-poetry
The package can be built with poetry build, then the wheel copied to the private pypi server at /simple/cq-test-poetry/cq_test_poetry-0.1.0-py3-none-any.whl.

When adding to a separate project with poetry add cq-test-poetry, it only finds cq-test-poetry, not the included requests package.

$ poetry add cq-test-poetry
Using version ^0.1.0 for cq-test-poetry

Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 1 install, 0 updates, 0 removals

  • Installing cq-test-poetry (0.1.0)

Writing lock file

Pip freeze shows only one package installed.

$ poetry run pip freeze
cq-test-poetry==0.1.0

However, if installing with pip, the dependencies are installed.

$ poetry run pip install --index-url https://pypi.domain.tld/simple --no-cache --progress-bar off cq-test-poetry
Looking in indexes: https://pypi.domain.tld/simple
Collecting cq-test-poetry
  Downloading https://pypi.domain.tld/simple/cq-test-poetry/cq_test_poetry-0.1.0-py3-none-any.whl (1.1 kB)
Collecting requests<3.0.0,>=2.30.0 (from cq-test-poetry)
  Downloading requests-2.30.0-py3-none-any.whl (62 kB)
Collecting charset-normalizer<4,>=2 (from requests<3.0.0,>=2.30.0->cq-test-poetry)
  Downloading charset_normalizer-3.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)
Collecting idna<4,>=2.5 (from requests<3.0.0,>=2.30.0->cq-test-poetry)
  Downloading idna-3.4-py3-none-any.whl (61 kB)
Collecting urllib3<3,>=1.21.1 (from requests<3.0.0,>=2.30.0->cq-test-poetry)
  Downloading urllib3-2.0.2-py3-none-any.whl (123 kB)
Collecting certifi>=2017.4.17 (from requests<3.0.0,>=2.30.0->cq-test-poetry)
  Downloading certifi-2023.5.7-py3-none-any.whl (156 kB)
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests, cq-test-poetry
Successfully installed certifi-2023.5.7 charset-normalizer-3.1.0 cq-test-poetry-0.1.0 idna-3.4 requests-2.30.0 urllib3-2.0.2

$ poetry run pip freeze
certifi==2023.5.7
charset-normalizer==3.1.0
cq-test-poetry==0.1.0
idna==3.4
requests==2.30.0
urllib3==2.0.2

[dimbleby]

Works fine for me using pypi-server as my private repository:

$ tree ~/packages/
/home/dch/packages/
└── simple
    └── cq-test-poetry
        └── cq_test_poetry-0.1.0-py3-none-any.whl

2 directories, 1 file

$ pypi-server run --hash-algo sha256 -p 8080 ~/packages/

[tool.poetry.dependencies]
python = "^3.10"

[[tool.poetry.source]]
name = "private_pypi"
url = "http://localhost:8080/simple/"
priority = "supplemental"

[[tool.poetry.source]]
name = "PyPI"
priority = "primary"

$ poetry --version
Poetry (version 1.5.0)

$ poetry add cq-test-poetry
Using version ^0.1.0 for cq-test-poetry

Updating dependencies
Resolving dependencies... (0.3s)

Package operations: 6 installs, 0 updates, 0 removals

  • Installing certifi (2023.5.7)
  • Installing charset-normalizer (3.1.0)
  • Installing idna (3.4)
  • Installing urllib3 (2.0.2)
  • Installing requests (2.30.0)
  • Installing cq-test-poetry (0.1.0)

[cquick01]

Try with python -m http.server 8080 instead to test with a basic web server instance. We are serving our real one simply with nginx.

$ tree simple
simple
└── cq-test-poetry
    └── cq_test_poetry-0.1.0-py3-none-any.whl

1 directory, 1 file

$ python -m http.server 8080
Serving HTTP on 0.0.0.0 port 8080 (http://0.0.0.0:8080/) ...
127.0.0.1 - - [19/May/2023 14:58:44] "GET /simple/cq-test-poetry/ HTTP/1.1" 200 -
127.0.0.1 - - [19/May/2023 14:58:45] "GET /simple/cq-test-poetry/cq_test_poetry-0.1.0-py3-none-any.whl HTTP/1.1" 200 -

pyproject.toml (from new project we're trying to install cq-test-poetry into)

$ cat pyproject.toml
[tool.poetry]
name = "install-test"
version = "0.1.0"
description = ""
authors = ["Author"]
readme = "README.md"
packages = [{include = "install_test"}]

[tool.poetry.dependencies]
python = "^3.9"

[[tool.poetry.source]]
name = "private_pypi"
url = "http://localhost:8080/simple/"
priority = "supplemental"

[[tool.poetry.source]]
name = "PyPI"
priority = "primary"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

And trying to add the package fails to install dependencies with poetry, but pip works fine.

$ poetry add cq-test-poetry
Using version ^0.1.0 for cq-test-poetry

Updating dependencies
Resolving dependencies... (0.2s)

Package operations: 1 install, 0 updates, 0 removals

  • Installing cq-test-poetry (0.1.0)

Writing lock file

$ poetry run pip freeze
cq-test-poetry==0.1.0

$ poetry run pip install cq-test-poetry
Requirement already satisfied: cq-test-poetry in /home/cquick/.cache/pypoetry/virtualenvs/install-test-v0Ud3NDT-py3.10/lib/python3.10/site-packages (0.1.0)
Collecting requests<3.0.0,>=2.30.0 (from cq-test-poetry)
  Using cached requests-2.30.0-py3-none-any.whl (62 kB)
Collecting charset-normalizer<4,>=2 (from requests<3.0.0,>=2.30.0->cq-test-poetry)
  Using cached charset_normalizer-3.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)
Collecting idna<4,>=2.5 (from requests<3.0.0,>=2.30.0->cq-test-poetry)
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting urllib3<3,>=1.21.1 (from requests<3.0.0,>=2.30.0->cq-test-poetry)
  Using cached urllib3-2.0.2-py3-none-any.whl (123 kB)
Collecting certifi>=2017.4.17 (from requests<3.0.0,>=2.30.0->cq-test-poetry)
  Using cached certifi-2023.5.7-py3-none-any.whl (156 kB)
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
Successfully installed certifi-2023.5.7 charset-normalizer-3.1.0 idna-3.4 requests-2.30.0 urllib3-2.0.2

$ poetry run pip freeze
certifi==2023.5.7
charset-normalizer==3.1.0
cq-test-poetry==0.1.0
idna==3.4
requests==2.30.0
urllib3==2.0.2

[dimbleby]

That also works just fine for me

[cquick01]

So I tried to repro in docker, but was unable to. After messing with it a bit more, it appears that deleting ~/.cache/pypoetry on my local machine gets it working again. Makes sense that it worked when in a 'clean' environment.

I have that cache directory backed up just in case. If there is any specific info that might be useful, I can look for it. I'd rather not just post the whole dir up somewhere because I don't know if any private info, like my company packages, are within.

It'd be good to avoid putting users into a weird state like this if poetry could detect and clean up the problematic cache. But for now this gets things working again.

Thanks!

[chrisgilmerproj]

I was struggling with not getting the dependencies installed today when upgrading to Poetry 1.5.0 and python 3.10.11. The original pyprojects.toml file had only one source and it was my local devpi server that I had previously been using as a pull-through cache and private repository.

[tool.poetry.dependencies]
python = "^3.10"

[[tool.poetry.source]]
name = "private_pypi"
url = "http://localhost:8080/simple/"
priority = "primary"

However, when I modified it to include PyPI like this then the dependencies arrived:

[tool.poetry.dependencies]
python = "^3.10"

[[tool.poetry.source]]
name = "private_pypi"
url = "http://localhost:8080/simple/"
priority = "supplemental"

[[tool.poetry.source]]
name = "PyPI"
priority = "primary"

This means that I'm no longer using my pull-through cache to get the set of packages. However, it works for now so I can move forward. Was there some change that causes this behavior? I'm thinking its related to this ticket too since the behavior seems similar.

[dimbleby]

And is your case also resolved by clearing your poetry cache?

[dimbleby]

We're back to: please provide a way to reproduce.

(So far the only reproducer that we've seen is resolved by clearing the cache)

[cquick01]

I'm sort of able to reproduce this again, this time in docker. The entirety of ~/.cache is completely empty, however the dependencies are only found with --no-cache or when pointing to a local .whl file.

Fails to find deps:

➜  ~/test git:(main) ✗ docker run -it poetry-docker
root@4e0af102c3fb:/opt# ls ~/.cache/
pip
root@4e0af102c3fb:/opt# rm -rf ~/.cache/pip
root@4e0af102c3fb:/opt# ls -la ~/.cache/
total 16
drwxr-xr-x 1 root root 4096 May 22 18:19 .
drwx------ 1 root root 4096 May 22 17:38 ..
root@4e0af102c3fb:/opt# poetry shell
Creating virtualenv opt-y8366zdl-py3.11 in /root/.cache/pypoetry/virtualenvs
Spawning shell within /root/.cache/pypoetry/virtualenvs/opt-y8366zdl-py3.11
root@4e0af102c3fb:/opt# . /root/.cache/pypoetry/virtualenvs/opt-y8366zdl-py3.11/bin/activate
(opt-py3.11) root@4e0af102c3fb:/opt# poetry add cq-test-poetry
Using version ^0.1.0 for cq-test-poetry

Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 1 install, 0 updates, 0 removals

  • Installing cq-test-poetry (0.1.0)

Writing lock file

Finds deps with --no-cache

➜  ~/test git:(main) ✗ docker run -it poetry-docker
root@05a04d106afa:/opt# ls ~/.cache/
pip
root@05a04d106afa:/opt# rm -rf ~/.cache/pip
root@05a04d106afa:/opt# ls -la ~/.cache/
total 16
drwxr-xr-x 1 root root 4096 May 22 18:20 .
drwx------ 1 root root 4096 May 22 17:38 ..
root@05a04d106afa:/opt# poetry shell
Creating virtualenv opt-y8366zdl-py3.11 in /root/.cache/pypoetry/virtualenvs
Spawning shell within /root/.cache/pypoetry/virtualenvs/opt-y8366zdl-py3.11
root@05a04d106afa:/opt# . /root/.cache/pypoetry/virtualenvs/opt-y8366zdl-py3.11/bin/activate
(opt-py3.11) root@05a04d106afa:/opt# poetry add --no-cache cq-test-poetry
Using version ^0.1.0 for cq-test-poetry

Updating dependencies
Resolving dependencies... Downloading https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-pyResolving dependencies... (0.7s)

Package operations: 6 installs, 0 updates, 0 removals

  • Installing certifi (2023.5.7)
  • Installing charset-normalizer (3.1.0)
  • Installing idna (3.4)
  • Installing urllib3 (2.0.2)
  • Installing requests (2.31.0)
  • Installing cq-test-poetry (0.1.0)

Writing lock file

Installing from .whl finds deps:

➜  ~/test git:(main) ✗ docker run -it poetry-docker
root@53a33d15cfc7:/opt# ls ~/.cache
pip
root@53a33d15cfc7:/opt# rm -rf ~/.cache/pip/
root@53a33d15cfc7:/opt# ls -la ~/.cache
total 16
drwxr-xr-x 1 root root 4096 May 22 18:21 .
drwx------ 1 root root 4096 May 22 17:38 ..
root@53a33d15cfc7:/opt# poetry shell
Creating virtualenv opt-y8366zdl-py3.11 in /root/.cache/pypoetry/virtualenvs
Spawning shell within /root/.cache/pypoetry/virtualenvs/opt-y8366zdl-py3.11
root@53a33d15cfc7:/opt# . /root/.cache/pypoetry/virtualenvs/opt-y8366zdl-py3.11/bin/activate
(opt-py3.11) root@53a33d15cfc7:/opt# poetry add ./cq_test_poetry-0.1.0-py3-none-any.whl

Updating dependencies
Resolving dependencies... (0.7s)

Package operations: 6 installs, 0 updates, 0 removals

  • Installing certifi (2023.5.7)
  • Installing charset-normalizer (3.1.0)
  • Installing idna (3.4)
  • Installing urllib3 (2.0.2)
  • Installing requests (2.31.0)
  • Installing cq-test-poetry (0.1.0 /opt/cq_test_poetry-0.1.0-py3-none-any.whl)

Writing lock file

I've updated https://github.com/cquick01/cq-test-poetry with a Dockerfile, hopefully others can reproduce too now...

[dimbleby]

@cquick01 once again this works just fine for me.

you keep using https://pypi.domain.tld/simple as if that should work for the rest of us but so far as I can see it is just something you have made up? So I again set up python -m http.server 8080 on the host, invoked docker with --add-host=host.docker.internal:host-gateway, and switched in url = http://host.docker.internal:8080/simple/

Then, as before, poetry add cq-test-poetry installs both that package and all dependencies.

[cquick01]

Our real pypi server is not accessible from the outside world.

Here, I've setup a VPS with an nginx server hosting the package at http://ec2-18-219-190-206.us-east-2.compute.amazonaws.com/simple/ (probably offline now)

And trying to install cq-test-poetry from within that instance, with source set to localhost, still fails the same way.

ubuntu@ip-172-31-20-126:~/test$ cat pyproject.toml
[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"

[[tool.poetry.source]]
name = "private_pypi"
url = "http://localhost/simple/"
priority = "supplemental"

[[tool.poetry.source]]
name = "PyPI"
priority = "primary"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
ubuntu@ip-172-31-20-126:~/test$ poetry add cq-test-poetry
Creating virtualenv test-QNrpxocX-py3.10 in /home/ubuntu/.cache/pypoetry/virtualenvs
Using version ^0.1.0 for cq-test-poetry

Updating dependencies
Resolving dependencies... (0.2s)

Package operations: 1 install, 0 updates, 0 removals

  • Installing cq-test-poetry (0.1.0)

Writing lock file

And --no-cache works again

ubuntu@ip-172-31-20-126:~/test$ poetry remove cq-test-poetry
Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 0 installs, 0 updates, 1 removal

  • Removing cq-test-poetry (0.1.0)

Writing lock file
ubuntu@ip-172-31-20-126:~/test$ poetry add --no-cache cq-test-poetry
Using version ^0.1.0 for cq-test-poetry

Updating dependencies
Resolving dependencies... (0.5s)

Package operations: 6 installs, 0 updates, 0 removals

  • Installing certifi (2023.5.7)
  • Installing charset-normalizer (3.1.0)
  • Installing idna (3.4)
  • Installing urllib3 (2.0.2)
  • Installing requests (2.31.0)
  • Installing cq-test-poetry (0.1.0)

Writing lock file

Can you send me a public key and I'll just give you access to the instance?

[chrisgilmerproj]

And is your case also resolved by clearing your poetry cache?

I apologize that this took me a while to get back to and reproduce related to my bug report above. I did in fact find that clearing my entire poetry cache did the trick. There did not appear to be another way for it to work with my devpi service as the primary other than that. I used the poetry command itself to clear for me (where private_repo is a made up cache):

poetry clear cache private_repo --all

Just to be sure this worked I did this for several repos I'm in the process of upgrading and was able to reproduce that clearing the cache works for them as well. I was upgrading from 1.3.2 to 1.5.0 in case other folks are curious..

Thanks again for the hard work you put into this repository and issues like this. I know its not always great dealing with user reports.

[chriskuehl]

We are also running an issue that looks like this with our internal registry and Poetry 1.5. I think I've narrowed it down to some kind of issue with response caching, because if we disable Etags on our PyPI registry, the issues go away.

It's been tricky to reproduce this but I've managed to create two small publicly accessible reproduction indexes which have boto3==1.26.137 and its dependencies. These are identical except that one serves Etags and one does not (one has etag off; in the nginx config):

• https://ckuehl.me/tmp/poetry-7961/simple/
• https://ckuehl.me/tmp/poetry-7961/simple-no-etags/

Here is a test pyproject.toml I am using:

[tool.poetry]
name = "test-project"
version = "0.0.0"
authors = []
description = ""

[tool.poetry.dependencies]
python = "^3.11"
boto3 = "1.26.137"

[[tool.poetry.source]]
name = "my-pypi"
url = "https://ckuehl.me/tmp/poetry-7961/simple"
# Switch to this URL to test without etags:
# url = "https://ckuehl.me/tmp/poetry-7961/simple-no-etags"
priority = "default"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

When installing from the first repo, only boto3 is locked (none of its dependencies are). With the second repo, dependencies are also locked as expected.

Output with the first registry (Etags enabled)

$ venv/bin/poetry --version                                       
Poetry (version 1.5.0)

$ rm -rf poetry.lock ~/.cache/pypoetry && venv/bin/poetry lock -vv
Creating virtualenv test-project-ovAjWgsE-py3.11 in /home/ckuehl/.cache/pypoetry/virtualenvs
Using virtualenv: /home/ckuehl/.cache/pypoetry/virtualenvs/test-project-ovAjWgsE-py3.11
Updating dependencies
Resolving dependencies...
   1: fact: test-project is 0.0.0
   1: derived: test-project
   1: fact: test-project depends on boto3 (1.26.137)
   1: selecting test-project (0.0.0)
   1: derived: boto3 (==1.26.137)
   1: selecting boto3 (1.26.137)
   1: Version solving took 0.486 seconds.
   1: Tried 1 solutions.

Writing lock file

$ venv/bin/poetry show                                            
boto3 1.26.137 

Output with the second registry (Etags disabled)

$ venv/bin/poetry --version                              
Poetry (version 1.5.0)

$ rm -rf poetry.lock ~/.cache/pypoetry && venv/bin/poetry lock -vv
Creating virtualenv test-project-ovAjWgsE-py3.11 in /home/ckuehl/.cache/pypoetry/virtualenvs
Using virtualenv: /home/ckuehl/.cache/pypoetry/virtualenvs/test-project-ovAjWgsE-py3.11
Updating dependencies
Resolving dependencies...
   1: fact: test-project is 0.0.0
   1: derived: test-project
   1: fact: test-project depends on boto3 (1.26.137)
   1: selecting test-project (0.0.0)
   1: derived: boto3 (==1.26.137)
   1: fact: boto3 (1.26.137) depends on botocore (>=1.29.137,<1.30.0)
   1: fact: boto3 (1.26.137) depends on jmespath (>=0.7.1,<2.0.0)
   1: fact: boto3 (1.26.137) depends on s3transfer (>=0.6.0,<0.7.0)
   1: selecting boto3 (1.26.137)
   1: derived: s3transfer (>=0.6.0,<0.7.0)
   1: derived: jmespath (>=0.7.1,<2.0.0)
   1: derived: botocore (>=1.29.137,<1.30.0)
   1: fact: s3transfer (0.6.1) depends on botocore (>=1.12.36,<2.0a.0)
   1: selecting s3transfer (0.6.1)
   1: selecting jmespath (1.0.1)
   1: fact: botocore (1.29.137) depends on jmespath (>=0.7.1,<2.0.0)
   1: fact: botocore (1.29.137) depends on python-dateutil (>=2.1,<3.0.0)
   1: fact: botocore (1.29.137) depends on urllib3 (>=1.25.4,<1.27)
   1: selecting botocore (1.29.137)
   1: derived: urllib3 (>=1.25.4,<1.27)
   1: derived: python-dateutil (>=2.1,<3.0.0)
   1: selecting urllib3 (1.26.15)
   1: fact: python-dateutil (2.8.2) depends on six (>=1.5)
   1: selecting python-dateutil (2.8.2)
   1: derived: six (>=1.5)
   1: selecting six (1.16.0)
   1: Version solving took 6.119 seconds.
   1: Tried 1 solutions.

Writing lock file

$ venv/bin/poetry show
boto3           1.26.137 The AWS SDK for Python
botocore        1.29.137 Low-level, data-driven core of boto 3.
jmespath        1.0.1    JSON Matching Expressions
python-dateutil 2.8.2    Extensions to the standard Python datetime module
s3transfer      0.6.1    An Amazon S3 Transfer Manager
six             1.16.0   Python 2 and 3 compatibility utilities
urllib3         1.26.15  HTTP library with thread-safe connection pooling, file post, and more.

I'm still trying to figure out exactly what happens; the problem may not be Etags themselves but just that they enable the caching logic which causes the issue. I'm continuing to look into this but since there has been a lot of searching for reproductions I figured I would post this sooner rather than later.

[dimbleby]

@ralbertazzi looks like this was introduced at #7916

I think the problem is that the the cached file returned by get_cached_file_for_url is not the cached wheel, it's the cached http response (with the wheel still somewhere inside it)

[chriskuehl]

@dimbleby I just came to the same conclusion. Here's a (very rough/awkward, probably not what we want to merge) patch which fixes it for me:

diff --git a/src/poetry/repositories/http_repository.py b/src/poetry/repositories/http_repository.py
index ae96481a..88977d69 100644
--- a/src/poetry/repositories/http_repository.py
+++ b/src/poetry/repositories/http_repository.py
@@ -78,7 +78,14 @@ class HTTPRepository(CachedRepository):
     def _cached_or_downloaded_file(self, link: Link) -> Iterator[Path]:
         filepath = self._authenticator.get_cached_file_for_url(link.url)
         if filepath:
-            yield filepath
+            import os
+            from cachecontrol.serialize import Serializer
+            serializer = Serializer()
+            with temporary_directory() as temp_dir:
+                path = os.path.join(temp_dir, "wheel.whl")
+                with open(path, "wb") as f:
+                    f.write(serializer.loads(None, open(filepath, "rb").read()).data)
+                yield path
         else:
             self._log(f"Downloading: {link.url}", level="debug")
             with temporary_directory() as temp_dir:

Even though this is not a good patch, maybe other people experiencing this could also test it to verify if it's the same root cause for all these reports?

[cquick01]

Can confirm disabling etags in nginx.conf fixes it for me. I'll try testing with that patch. Thanks guys!

Edit: yes, that patch works for me with etags enabled!

ubuntu@ip-172-31-20-126:~/test$ pip install poetry==1.5.0 --quiet
ubuntu@ip-172-31-20-126:~/test$ pip freeze | grep poetry
poetry==1.5.0
poetry-core==1.6.0
poetry-plugin-export==1.3.1
ubuntu@ip-172-31-20-126:~/test$ rm -rf ~/.cache/pypoetry/*
ubuntu@ip-172-31-20-126:~/test$ poetry add cq-test-poetry
Creating virtualenv test-QNrpxocX-py3.10 in /home/ubuntu/.cache/pypoetry/virtualenvs
Using version ^0.1.0 for cq-test-poetry

Updating dependencies
Resolving dependencies... (0.2s)

Package operations: 1 install, 0 updates, 0 removals

  • Installing cq-test-poetry (0.1.0)

Writing lock file
ubuntu@ip-172-31-20-126:~/test$ poetry remove cq-test-poetry
Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file
ubuntu@ip-172-31-20-126:~/test$ pip install ~/poetry/dist/poetry-1.6.0.dev0-py3-none-any.whl --quiet
ubuntu@ip-172-31-20-126:~/test$ pip freeze | grep poetry
poetry @ file:///home/ubuntu/poetry/dist/poetry-1.6.0.dev0-py3-none-any.whl#sha256=91d0ee790d5f72fc8bbf37467a44de77134e44963a5726f21864c09891a28ba1
poetry-core==1.6.0
poetry-plugin-export==1.3.1
ubuntu@ip-172-31-20-126:~/test$ rm -rf ~/.cache/pypoetry/*
ubuntu@ip-172-31-20-126:~/test$ poetry add cq-test-poetry
Creating virtualenv test-QNrpxocX-py3.10 in /home/ubuntu/.cache/pypoetry/virtualenvs
Using version ^0.1.0 for cq-test-poetry

Updating dependencies
Resolving dependencies... Downloading https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbbResolving dependencies... (0.6s)

Package operations: 6 installs, 0 updates, 0 removals

  • Installing certifi (2023.5.7)
  • Installing charset-normalizer (3.1.0)
  • Installing idna (3.4)
  • Installing urllib3 (2.0.2)
  • Installing requests (2.31.0)
  • Installing cq-test-poetry (0.1.0)

Writing lock file

[kis87988]

Not sure if this is related to a bug under psf/cachecontrol#292 or not, but I set urllib3==1.26.15 to my environment. The poetry working fine.

[dimbleby]

I don't think #7916 made a lot of sense. As this MR shows, the file that is being inspected is an internal implementation detail for cachecontrol.

The suggested patch above, deserializing the file, is presumably doing only what cachecontrol would have done all along. If poetry repeats a GET that cachecontrol has already seen, cachecontrol can take care of that.

@ralbertazzi

[ralbertazzi]

I'll check myself but it seems like the best solution is to revert the change. @dimbleby do you have suggestions on how to best deal with the buffer size when the file is already cached locally? We use a chunk_size: int = 1024 that works well when the package needs to be fetched from the internet, but it's terribly inefficient when the file is already cached locally (a good chunk size in that case would be at least 10x)

[dimbleby]

in the long run I think the right fix is probably not to use cachecontrol at all for package distributions - we have an ArtifactCache as used by the Chef, we should find a way to share that for all artifact downloads. Then there's no need for the additional cache in cachecontrol.

I don't know any particularly good reason for the chunk size to be so small even when downloading from the internet, though I don't know whether this is a change that would be accepted in - I assume - an upcoming patch release.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.