ValueError, whl in cache does not exist when running poetry install again

[Transmitt0r]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• Windows 10 1909

• poetry version: 1.2.0a1

• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/Transmitt0r/ed8414abd90efaef9d4453ec05c48e1c

Issue

Hey everyone, I'm having issues when trying to run poetry install via jenkins. poetry config virtualenvs.in-project is true.
The install fails on the second installation attempt (always). It seems that there is an issue with the way caching works. Here is the trace:
Thanks in advance!

de\user@PC C:\workspace>poetry install -vvv 
Loading configuration file C:\Users\user\AppData\Roaming\pypoetry\config.toml
Adding repository private (https://artifactory-url.de)
Creating virtualenv envname in C:\workspace\.venv
Using virtualenv: C:\workspace\.venv
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 20 installs, 0 updates, 0 removals, 3 skipped

  • Removing importlib-metadata (4.5.0): Skipped for the following reason: Not currently installed
  • Removing typing-extensions (3.10.0.0): Skipped for the following reason: Not currently installed
  • Removing zipp (3.4.1): Skipped for the following reason: Not currently installed
  • Installing pyparsing (2.4.7)

  Stack trace:

  6  c:\users\user\appdata\roaming\pypoetry\venv\lib\site-packages\poetry\installation\executor.py:244 in _execute_operation
      242| 
      243|             try:
    > 244|                 result = self._do_execute_operation(operation)
      245|             except EnvCommandError as e:
      246|                 if e.e.returncode == -2:

  5  c:\users\user\appdata\roaming\pypoetry\venv\lib\site-packages\poetry\installation\executor.py:321 in _do_execute_operation
      319|             return 0
      320| 
    > 321|         result = getattr(self, f"_execute_{method}")(operation)
      322| 
      323|         if result != 0:

  4  c:\users\user\appdata\roaming\pypoetry\venv\lib\site-packages\poetry\installation\executor.py:462 in _execute_install
      460| 
      461|     def _execute_install(self, operation: Union[Install, Update]) -> int:
    > 462|         status_code = self._install(operation)
      463| 
      464|         self._save_url_reference(operation)

  3  c:\users\user\appdata\roaming\pypoetry\venv\lib\site-packages\poetry\installation\executor.py:498 in _install
      496|             archive = self._download_link(operation, Link(package.source_url))
      497|         else:
    > 498|             archive = self._download(operation)
      499| 
      500|         operation_message = self.get_operation_message(operation)

  2  c:\users\user\appdata\roaming\pypoetry\venv\lib\site-packages\poetry\installation\executor.py:648 in _download
      646|         link = self._chooser.choose_for(operation.package)
      647| 
    > 648|         return self._download_link(operation, link)
      649| 
      650|     def _download_link(self, operation: Union[Install, Update], link: Link) -> Link:

  1  c:\users\user\appdata\roaming\pypoetry\venv\lib\site-packages\poetry\installation\executor.py:676 in _download_link
      674|             archive_hash = (
      675|                 "sha256:"
    > 676|                 + FileDependency(
      677|                     package.name,
      678|                     Path(archive.path) if isinstance(archive, Link) else archive,

  ValueError

  File \C:\Users\user\AppData\Local\pypoetry\Cache\artifacts\92\0f\cf\effdcd5d76a6186df0969f85b3b030284ff8058936d5016540b5258ea3\pyparsing-2.4.7-py2.py3-none-any.whl does not exist

  at c:\users\user\appdata\roaming\pypoetry\venv\lib\site-packages\poetry\core\packages\file_dependency.py:41 in __init__
       37|             except FileNotFoundError:
       38|                 raise ValueError("Directory {} does not exist".format(self._path))
       39| 
       40|         if not self._full_path.exists():
    >  41|             raise ValueError("File {} does not exist".format(self._path))
       42| 
       43|         if self._full_path.is_dir():
       44|             raise ValueError("{} is a directory, expected a file".format(self._path))
       45| 

[Knaifhogg]

Same issue for me using Windows and Poetry 1.2.0a1 and trying to install pywin32. The file does exist in cache, but I noticed the file path written in the ValueError is "\C:\Users". Is that the issue?

[gygabyte017]

Same here, I as well noticed the \ before the absolute path, may that be the reason why it is not finding the whl even though it exist?

[vsutardja]

I ran into this same issue today.

Calling urllib.parse.urlsplit on a Windows file scheme URI (file:///C:/Users/...) leaves the extra leading / in the result's path attribute.

We can strip this character by passing the malformed path into urllib.request.url2pathname() to get a valid Windows-style path string.

>>> url = 'file:///C:/Users/foo/bar'
>>> split = urllib.parse.urlsplit(url)
>>> split.path
'/C:/Users/foo/bar'
>>> urllib.request.url2pathname(split.path)
'C:\\Users\\foo\\bar'

Note that url2pathname is only recommended for use on file scheme URIs, and it basically just wraps urllib.parse.unquote on non-Windows systems.[0] On Windows, it ends up using nturl2path.url2pathname.[1]

In fact, pip also uses url2pathname, along with some extra logic to handle the special case of a Windows UNC path (e.g. for shared network folders).[2]

[0] https://github.com/python/cpython/blob/3.9/Lib/urllib/request.py#L1675
[1] https://github.com/python/cpython/blob/v3.6.1/Lib/nturl2path.py#L3
[2] https://github.com/pypa/pip/blob/9.0.1/pip/download.py#L442

[MrHappy]

I ran into this yesterday also... Is there a temporary fix (workaround) until this is fixed permanently?

[Transmitt0r]

I ran into this yesterday also... Is there a temporary fix (workaround) until this is fixed permanently?

My workaround is a short batch script that deletes the poetry cache as part of my ci pipeline before poetry install is executed.

[jgentil]

This bug actually gets stranger. The error message is also incorrect. If you look here:

https://github.com/python-poetry/poetry-core/blob/ad33bc2f92be03dc5b31a666664903c439fb1173/poetry/core/packages/file_dependency.py#L40-L41

We see that it tests self._full_path.exists() but when it raises ValueError it puts the raw self._path in instead.

I'm not sure why self._path is being given a bad URL but I'm working through tracing that still.

[jgentil]

I found the culprit!

poetry/poetry/installation/executor.py

Lines 673 to 680 in e180aab

	if package.files:
	archive_hash = (
	"sha256:"
	+ FileDependency(
	package.name,
	Path(archive.path) if isinstance(archive, Link) else archive,
	).hash()
	)

At this point, archive is always going to be a file URL because it's been downloaded and cached. But Link.path is broken for file URLs on Windows, so the patch is as follows:

https://github.com/python-poetry/poetry-core/blob/ad33bc2f92be03dc5b31a666664903c439fb1173/poetry/core/packages/utils/link.py#L9
becomes:

from .utils import path_to_url, url_to_path

and

https://github.com/python-poetry/poetry-core/blob/ad33bc2f92be03dc5b31a666664903c439fb1173/poetry/core/packages/utils/link.py#L103-L105
becomes

    @property
    def path(self) -> str:
        if self.url.startswith("file:"):
            p = str(url_to_path(self.url))
        else:
            p = urlparse.unquote(urlparse.urlsplit(self.url)[2])
        return p

Unfortunately this impacts a number of places where improper path handling happens, mainly in the executor module. It's a rabbit hole for sure and I need to look at my local edits and compare with a fresh clone to get ALL of the spots I changed code.

In hindsight, it may be better to leave Link.path alone and instead fix all of the misuses of it.

[gwdekker]

Is it known in which version this was introduced? I am seeing something very similar in poetry 1.1.9.

[m-haisham]

As a workaround you could add the dependencies to pyproject.toml manually, then install them by generating the requirements.txt file.

poetry lock --no-update
poetry poetry export -f requirements.txt > requirements.txt
pip install -r requirements.txt

The downside is that this wont install the dev dependencies.

[PablocFonseca]

using version 1.1.10 here, same issue
downgraded to 1.0.8 works fine

[simontuohy]

Had this as well downgrading to 1.1.8 fixed for me on Windows 10

python get-poetry.py --version 1.1.8

[Jackiexiao]

I downgrade to 1.1.8 then encounter this issue: #4409

[MadJlzz]

Also have this issue with 1.1.8 on Windows 10.
My workaround is to go to the path specified, delete the cache and run poetry install again.

e.g.

PS C:\GithubTech\ngx\ngx-cli> poetry install
Installing dependencies from lock file

Package operations: 32 installs, 0 updates, 0 removals

  • Installing pyparsing (2.4.7)

  ValueError

  File \C:\Users\julie\AppData\Local\pypoetry\Cache\artifacts\92\0f\cf\effdcd5d76a6186df0969f85b3b030284ff8058936d5016540b5258ea3\pyparsing-2.4.7-py2.py3-none-any.whl does not exist

  at c:\users\julie\.pyenv\pyenv-win\versions\3.9.6\lib\site-packages\poetry\core\packages\file_dependency.py:40 in __init__
       36│             except FileNotFoundError:
       37│                 raise ValueError("Directory {} does not exist".format(self._path))
       38│
       39│         if not self._full_path.exists():
    →  40│             raise ValueError("File {} does not exist".format(self._path))
       41│
       42│         if self._full_path.is_dir():
       43│             raise ValueError("{} is a directory, expected a file".format(self._path))
       44│

Go to C:\Users\julie\AppData\Local\pypoetry\Cache, delete everything.

Run poetry install again and:

PS C:\GithubTech\ngx\ngx-cli> poetry install
Installing dependencies from lock file

Package operations: 32 installs, 0 updates, 0 removals

  • Installing pyparsing (2.4.7)
  • Installing atomicwrites (1.4.0)
  • Installing attrs (21.2.0)
  • Installing backports.entry-points-selectable (1.1.0)
  • Installing filelock (3.0.12)
  • Installing iniconfig (1.1.1)
  • Installing lazy-object-proxy (1.6.0)
  • Installing packaging (21.0)
  • Installing platformdirs (2.3.0)
  • Installing pluggy (0.13.1)
  • Installing py (1.10.0)
  • Installing six (1.16.0)
  • Installing toml (0.10.2)
  • Installing wrapt (1.12.1)
  • Installing astroid (2.7.3)
  • Installing identify (2.2.14)
  • Installing mccabe (0.6.1)
  • Installing mypy-extensions (0.4.3)
  • Installing pytest (6.2.4)
  • Installing typing-extensions (3.10.0.2)
  • Installing cfgv (3.3.1)
  • Installing nodeenv (1.6.0)
  • Installing isort (5.9.3)
  • Installing pyyaml (5.4.1)
  • Installing virtualenv (20.7.2)
  • Installing mypy (0.910)
  • Installing pylint (2.10.2)
  • Installing pre-commit (2.14.0)
  • Installing yapf (0.31.0)
  • Installing pytest-mock (3.6.1)

Installing the current project: ngx-cli (0.1.0)

Hope it helps other people in the meantime!

[vsutardja]

After patching Link.path to properly handle Windows file URIs, poetry still fails to install the cached package:

  Stack trace:

  2  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\env.py:1296 in _run
      1294│                 return subprocess.call(cmd, stderr=subprocess.STDOUT, env=env, **kwargs)
      1295│             else:
    → 1296│                 output = subprocess.check_output(
      1297│                     cmd, stderr=subprocess.STDOUT, env=env, **kwargs
      1298│                 )

  1  ~\AppData\Local\Programs\Python\Python39\lib\subprocess.py:424 in check_output
       422│         kwargs['input'] = empty
       423│
    →  424│     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
       425│                **kwargs).stdout
       426│

  CalledProcessError

  Command 'C:\Users\user\AppData\Local\pypoetry\Cache\virtualenvs\poetry-cache-test-keI1PhHb-py3.9\Scripts\python.exe C:\Users\user\AppData\Roaming\pypoetry\venv\lib\site-packages\virtualenv\seed\wheels\embed\pip-21.0.1-py3-none-any.whl\pip install --disable-pip-version-check --prefix C:\Users\user\AppData\Local\pypoetry\Cache\virtualenvs\poetry-cache-test-keI1PhHb-py3.9 --no-deps file:\C:\Users\user\AppData\Local\pypoetry\Cache\artifacts\92\0f\cf\effdcd5d76a6186df0969f85b3b030284ff8058936d5016540b5258ea3\pyparsing-2.4.7-py2.py3-none-any.whl' returned non-zero exit status 1.

  at ~\AppData\Local\Programs\Python\Python39\lib\subprocess.py:528 in run
       524│             # We don't call process.wait() as .__exit__ does that for us.
       525│             raise
       526│         retcode = process.poll()
       527│         if check and retcode:
    →  528│             raise CalledProcessError(retcode, process.args,
       529│                                      output=stdout, stderr=stderr)
       530│     return CompletedProcess(process.args, retcode, stdout, stderr)
       531│
       532│

The following error occurred when trying to handle this error:


  Stack trace:

  3  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\pip.py:47 in pip_install
       45│
       46│     try:
    →  47│         return environment.run_pip(*args)
       48│     except EnvCommandError as e:
       49│         if sys.version_info < (3, 7) and not is_wheel:
                                                                                                           
  2  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\env.py:1264 in run_pip
      1262│         pip = self.get_pip_command(embedded=True)
      1263│         cmd = pip + list(args)
    → 1264│         return self._run(cmd, **kwargs)
      1265│
      1266│     def run_python_script(self, content: str, **kwargs: Any) -> str:
                                                                                                           
  1  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\env.py:1547 in _run
      1545│     def _run(self, cmd: List[str], **kwargs: Any) -> Optional[int]:
      1546│         kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
    → 1547│         return super()._run(cmd, **kwargs)
      1548│
      1549│     def get_temp_environ(

  EnvCommandError

  Command C:\Users\user\AppData\Local\pypoetry\Cache\virtualenvs\poetry-cache-test-keI1PhHb-py3.9\Scripts\python.exe C:\Users\user\AppData\Roaming\pypoetry\venv\lib\site-packages\virtualenv\seed\wheels\embed\pip-21.0.1-py3-none-any.whl\pip install --disable-pip-version-check --prefix C:\Users\user\AppData\Local\pypoetry\Cache\virtualenvs\poetry-cache-test-keI1PhHb-py3.9 --no-deps file:\C:\Users\user\AppData\Local\pypoetry\Cache\artifacts\92\0f\cf\effdcd5d76a6186df0969f85b3b030284ff8058936d5016540b5258ea3\pyparsing-2.4.7-py2.py3-none-any.whl errored with the following return code 1, and output:
  ERROR: Invalid requirement: '\\C:\\Users\\user\\AppData\\Local\\pypoetry\\Cache\\artifacts\\92\\0f\\cf\\effdcd5d76a6186df0969f85b3b030284ff8058936d5016540b5258ea3\\pyparsing==2.4.7'
  Hint: It looks like a path. File '\C:\Users\user\AppData\Local\pypoetry\Cache\artifacts\92\0f\cf\effdcd5d76a6186df0969f85b3b030284ff8058936d5016540b5258ea3\pyparsing==2.4.7' does not exist.


  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\env.py:1300 in _run
      1296│                 output = subprocess.check_output(
      1297│                     cmd, stderr=subprocess.STDOUT, env=env, **kwargs
      1298│                 )
      1299│         except CalledProcessError as e:
    → 1300│             raise EnvCommandError(e, input=input_)
      1301│
      1302│         return decode(output)
      1303│
      1304│     def execute(self, bin: str, *args: str, **kwargs: Any) -> Optional[int]:

The following error occurred when trying to handle this error:


  Stack trace:

  5  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\installation\executor.py:246 in _execute_operation
      244│
      245│             try:
    → 246│                 result = self._do_execute_operation(operation)
      247│             except EnvCommandError as e:
      248│                 if e.e.returncode == -2:

  4  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\installation\executor.py:323 in _do_execute_operation
      321│             return 0
      322│
    → 323│         result = getattr(self, f"_execute_{method}")(operation)
      324│
      325│         if result != 0:

  3  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\installation\executor.py:464 in _execute_install
      462│
      463│     def _execute_install(self, operation: Union[Install, Update]) -> int:
    → 464│         status_code = self._install(operation)
      465│
      466│         self._save_url_reference(operation)

  2  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\installation\executor.py:509 in _install
      507│         )
      508│         self._write(operation, message)
    → 509│         return self.pip_install(str(archive), upgrade=operation.job_type == "update")
      510│
      511│     def _update(self, operation: Union[Install, Update]) -> int:

  1  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\installation\executor.py:131 in pip_install
      129│         # req = Link(req).path if isinstance(req, str) else req
      130│         try:
    → 131│             func(req, self._env, upgrade=upgrade)
      132│         except EnvCommandError as e:
      133│             output = decode(e.e.output)

  PoetryException

  Failed to install file:/C:/Users/user/AppData/Local/pypoetry/Cache/artifacts/92/0f/cf/effdcd5d76a6186df0969f85b3b030284ff8058936d5016540b5258ea3/pyparsing-2.4.7-py2.py3-none-any.whl

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\pip.py:60 in pip_install
       56│                     *env.get_pip_command(),
       57│                     *args,
       58│                     env={**os.environ, "PYTHONPATH": str(env.purelib)},
       59│                 )
    →  60│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       61│
       62│
       63│ def pip_editable_install(directory: Path, environment: Env) -> Union[int, str]:
       64│     return pip_install(

This is because Executor.pip_install is being called with a string req containing - yet again - a Windows file URI. When this URI string gets passed to pip_install, it's used to construct a Path, which, like the original Link.path, also doesn't quite understand URIs:

>>> path = 'file:///C:/Users/foo/bar'
>>> Path(path)
WindowsPath('file:/C:/Users/foo/bar')

We can get around this by using the fixed Link.path to "sanitize" string inputs to Executor.pip_install, stripping away the URI prefix before passing it to the utility pip_install function.

# Sanitize req of type str
req = Link(req).path if isinstance(req, str) else req
try:
    func(req, self._env, upgrade=upgrade)

With these two changes, I can successfully install from the cache.

[petermbauer]

...

With these two changes, I can successfully install from the cache.

Can this be made into a PR or is there something else to consider?

[f0ff886f]

I made two PRs:

#4651 and python-poetry/poetry-core#222

I made them with the suggestions from this issue. I can not repro @vsutardja's issue that required a fixed executor (when I applied the fixes to Link.path my repro went away). I don't know if the fixes @jgentil is mentioning in executor.py still need to be put into the PRs.

[f0ff886f]

I found some much better PRs that already had some eyes, not sure what the path to get the PRs merged is.

1.1: #4549
1.2+: #4531

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.