Encoding failure is not caught and propagated

[kristang]

• Poetry version: 1.2.1 and 1.3.0.dev01
• Python version: 3.9.13
• OS version and name: Windows 11 Enterprise (build 22000.856)
• pyproject.toml:

[tool.poetry]
name = "test-poetry-master"
version = "0.1.0"
description = ""
authors = ["kristang"]
readme = "README.md"

[[tool.poetry.source]]
name = "private-pypi"
url = "https://private/pypi/private-pypi/simple/"
secondary = true

[tool.poetry.dependencies]
python = "^3.9"

[tool.poetry.dev-dependencies]
pytest = "^7.1.1"
mypy = "^0.982"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Additionally

• Discussed the issue with @neersighted as well as @finswimmer and @radoering on the Poetry Discord.

Issue

When adding a secondary, private source and attempting to authenticate via basic-http, poetry silently ignores a UnicodeEncodingError originating from a failed latin1 encoding of a str.

Instead, poetry moves on to the public pypi.org feed and informs the user that the searched package is not available - even if it is available on pypi.org

The error stems from loading in a user/password from Windows Certificate Manager which appears to be corrupted and cannot be encoded with 'latin1'

The source of the encoding error is poetry invoking _basic_auth_str from auth.py in the requests package:
https://github.com/psf/requests/blob/7104ad4b135daab0ed19d8e41bd469874702342b/requests/auth.py#L56-L60

The originating call in poetry is request in utils.authenticator.

poetry/src/poetry/utils/authenticator.py

Lines 213 to 216 in 0ffe91c

	if credential.username is not None or credential.password is not None:
	request = requests.auth.HTTPBasicAuth(
	credential.username or "", credential.password or ""
	)(request)

solve() has a fairly wide try-catch with a base Exception

poetry/src/poetry/mixology/version_solver.py

Lines 110 to 123 in 0ffe91c

	try:
	next: str | None = self._root.name
	while next is not None:
	self._propagate(next)
	next = self._choose_package_version()
	return self._result()
	except Exception:
	raise
	finally:
	self._log(
	f"Version solving took {time.time() - start:.3f} seconds.\n"
	f"Tried {self._solution.attempted_solutions} solutions."
	)

Reproducing

As this involves a broken str that fails to encode reproducibility is a bit hard. I will try and create tests and create a PR if my skills are sufficient enough to create something useful.

Running pipenv lock -vvv yields:

  Stack trace:

  4  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:157 in _solve
      155│ 
      156│         try:
    → 157│             result = resolve_version(self._package, self._provider)
      158│ 
      159│             packages = result.packages

  3  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\__init__.py:18 in resolve_version
       16│     solver = VersionSolver(root, provider)
       17│ 
    →  18│     return solver.solve()
       19│ 

  2  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:113 in solve
      111│             next: str | None = self._root.name
      112│             while next is not None:
    → 113│                 self._propagate(next)
      114│                 next = self._choose_package_version()
      115│ 

  1  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:152 in _propagate
      150│                     # where that incompatibility will allow us to derive new assignments
      151│                     # that avoid the conflict.
    → 152│                     root_cause = self._resolve_conflict(incompatibility)
      153│ 
      154│                     # Back jumping erases all the assignments we did at the previous

  SolveFailure

  Because test-poetry-master depends on mypy (^0.982) which doesn't exist, version solving failed.

  at ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:351 in _resolve_conflict
      347│             )
      348│             self._log(f'! which is caused by "{most_recent_satisfier.cause}"')
      349│             self._log(f"! thus: {incompatibility}")
      350│ 
    → 351│         raise SolveFailure(incompatibility)
      352│ 
      353│     def _choose_package_version(self) -> str | None:
      354│         """
      355│         Tries to select a version of a required package.

The following error occurred when trying to handle this error:


  Stack trace:

  11  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:329 in run
       327│ 
       328│             try:
     → 329│                 exit_code = self._run(io)
       330│             except Exception as e:
       331│                 if not self._catch_exceptions:

  10  ~\Documents\Projects\GITHUB\poetry\src\poetry\console\application.py:185 in _run
       183│         self._load_plugins(io)
       184│ 
     → 185│         exit_code: int = super()._run(io)
       186│         return exit_code
       187│ 

   9  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│ 
     → 423│         exit_code = self._run_command(command, io)
       424│         self._running_command = None
       425│ 

   8  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:465 in _run_command
       463│ 
       464│         if error is not None:
     → 465│             raise error
       466│ 
       467│         return event.exit_code

   7  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:449 in _run_command
       447│ 
       448│             if event.command_should_run():
     → 449│                 exit_code = command.run(io)
       450│             else:
       451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

   6  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\commands\base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

   5  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\commands\command.py:83 in execute
        81│ 
        82│         try:
     →  83│             return self.handle()
        84│         except KeyboardInterrupt:
        85│             return 1

   4  ~\Documents\Projects\GITHUB\poetry\src\poetry\console\commands\lock.py:54 in handle
        52│         self.installer.lock(update=not self.option("no-update"))
        53│ 
     →  54│         return self.installer.run()
        55│ 

   3  ~\Documents\Projects\GITHUB\poetry\src\poetry\installation\installer.py:114 in run
       112│             self._execute_operations = False
       113│ 
     → 114│         return self._do_install()
       115│ 
       116│     def dry_run(self, dry_run: bool = True) -> Installer:

   2  ~\Documents\Projects\GITHUB\poetry\src\poetry\installation\installer.py:247 in _do_install
       245│                 source_root=self._env.path.joinpath("src")
       246│             ):
     → 247│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       248│         else:
       249│             self._io.write_line("Installing dependencies from lock file")

   1  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:74 in solve
        72│         with self._progress(), self._provider.use_latest_for(use_latest or []):
        73│             start = time.time()
     →  74│             packages, depths = self._solve()
        75│             end = time.time()
        76│ 

  SolverProblemError

  Because test-poetry-master depends on mypy (^0.982) which doesn't exist, version solving failed.

  at ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:163 in _solve
      159│             packages = result.packages
      160│         except OverrideNeeded as e:
      161│             return self._solve_in_compatibility_mode(e.overrides)
      162│         except SolveFailure as e:
    → 163│             raise SolverProblemError(e)
      164│ 
      165│         combined_nodes = depth_first_search(PackageNode(self._package, packages))
      166│         results = dict(aggregate_package_nodes(nodes) for nodes in combined_nodes)
      167│

No-where in the stacktrace is a user informed that the underlying issue is an encoding failure of credentials.

Pseduo fix

Wrapping requests.auth.HTTPBasicAuth in a try-except can catch the UnicodeEncodeError and propagate it to the user:

if credential.username is not None or credential.password is not None:
    try:
        request = requests.auth.HTTPBasicAuth(
            credential.username or "", credential.password or ""
        )(request)
    except UnicodeEncodeError as e:
        raise PoetryException(e)

The new stacktrace now contains the correct UnicodeEncodeError:

  Stack trace:

  2  ~\Documents\Projects\GITHUB\poetry\src\poetry\utils\authenticator.py:216 in request
      214│         if credential.username is not None or credential.password is not None:
      215│             try:
    → 216│                 request = requests.auth.HTTPBasicAuth(
      217│                     credential.username or "", credential.password or ""
      218│                 )(request)

  1  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\requests\auth.py:95 in __call__
       93│ 
       94│     def __call__(self, r):
    →  95│         r.headers["Authorization"] = _basic_auth_str(self.username, self.password)
       96│         return r
       97│ 

  UnicodeEncodeError

  'latin-1' codec can't encode characters in position 0-5: ordinal not in range(256)

  at ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\requests\auth.py:60 in _basic_auth_str
       56│     if isinstance(username, str):
       57│         username = username.encode("latin1")
       58│ 
       59│     if isinstance(password, str):
    →  60│         password = password.encode("latin1")
       61│ 
       62│     authstr = "Basic " + to_native_string(
       63│         b64encode(b":".join((username, password))).strip()
       64│     )

The following error occurred when trying to handle this error:


  Stack trace:

  24  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:329 in run
       327│ 
       328│             try:
     → 329│                 exit_code = self._run(io)
       330│             except Exception as e:
       331│                 if not self._catch_exceptions:

  23  ~\Documents\Projects\GITHUB\poetry\src\poetry\console\application.py:185 in _run
       183│         self._load_plugins(io)
       184│ 
     → 185│         exit_code: int = super()._run(io)
       186│         return exit_code
       187│ 

  22  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│ 
     → 423│         exit_code = self._run_command(command, io)
       424│         self._running_command = None
       425│ 

  21  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:465 in _run_command
       463│ 
       464│         if error is not None:
     → 465│             raise error
       466│ 
       467│         return event.exit_code

  20  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\application.py:449 in _run_command
       447│ 
       448│             if event.command_should_run():
     → 449│                 exit_code = command.run(io)
       450│             else:
       451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  19  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\commands\base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

  18  ~\AppData\Local\pypoetry\Cache\virtualenvs\poetry-4e7WOrbt-py3.9\lib\site-packages\cleo\commands\command.py:83 in execute
        81│ 
        82│         try:
     →  83│             return self.handle()
        84│         except KeyboardInterrupt:
        85│             return 1

  17  ~\Documents\Projects\GITHUB\poetry\src\poetry\console\commands\lock.py:54 in handle
        52│         self.installer.lock(update=not self.option("no-update"))
        53│ 
     →  54│         return self.installer.run()
        55│ 

  16  ~\Documents\Projects\GITHUB\poetry\src\poetry\installation\installer.py:114 in run
       112│             self._execute_operations = False
       113│ 
     → 114│         return self._do_install()
       115│ 
       116│     def dry_run(self, dry_run: bool = True) -> Installer:

  15  ~\Documents\Projects\GITHUB\poetry\src\poetry\installation\installer.py:247 in _do_install
       245│                 source_root=self._env.path.joinpath("src")
       246│             ):
     → 247│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       248│         else:
       249│             self._io.write_line("Installing dependencies from lock file")

  14  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:74 in solve
        72│         with self._progress(), self._provider.use_latest_for(use_latest or []):
        73│             start = time.time()
     →  74│             packages, depths = self._solve()
        75│             end = time.time()
        76│ 

  13  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\solver.py:157 in _solve
       155│ 
       156│         try:
     → 157│             result = resolve_version(self._package, self._provider)
       158│ 
       159│             packages = result.packages

  12  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\__init__.py:18 in resolve_version
        16│     solver = VersionSolver(root, provider)
        17│ 
     →  18│     return solver.solve()
        19│ 

  11  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:114 in solve
       112│             while next is not None:
       113│                 self._propagate(next)
     → 114│                 next = self._choose_package_version()
       115│ 
       116│             return self._result()

  10  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:413 in _choose_package_version
       411│             dependency = unsatisfied[0]
       412│         else:
     → 413│             dependency = min(*unsatisfied, key=_get_min)
       414│ 
       415│         locked = self._provider.get_locked(dependency)

   9  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:399 in _get_min
       397│ 
       398│             try:
     → 399│                 num_packages = len(self._dependency_cache.search_for(dependency))
       400│             except ValueError:
       401│                 num_packages = 0

   8  ~\Documents\Projects\GITHUB\poetry\src\poetry\mixology\version_solver.py:62 in _search_for
        60│         packages = self.cache.get(key)
        61│         if packages is None:
     →  62│             packages = self.provider.search_for(dependency)
        63│         else:
        64│             packages = [

   7  ~\Documents\Projects\GITHUB\poetry\src\poetry\puzzle\provider.py:316 in search_for
        314│             return PackageCollection(dependency, packages)
        315│ 
     →  316│         packages = self._pool.find_packages(dependency)
        317│ 
        318│         packages.sort(

   6  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\pool.py:136 in find_packages
       134│         packages: list[Package] = []
       135│         for repo in self.repositories:
     → 136│             packages += repo.find_packages(dependency)
       137│         return packages
       138│ 

   5  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\repository.py:42 in find_packages
        40│         ignored_pre_release_packages = []
        41│ 
     →  42│         for package in self._find_packages(dependency.name, constraint):
        43│             if package.yanked and not isinstance(constraint, Version):
        44│                 # PEP 592: yanked files are always ignored, unless they are the only

   4  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\legacy_repository.py:96 in _find_packages
        94│             versions = self._cache.store("matches").get(key)
        95│         else:
     →  96│             page = self._get_page(f"/{name}/")
        97│             if page is None:
        98│                 self._log(

   3  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\legacy_repository.py:150 in _get_page
       148│     def _get_page(self, endpoint: str) -> SimpleRepositoryPage | None:
       149│         try:
     → 150│             response = self._get_response(endpoint)
       151│         except UnicodeEncodeError as e:
       152│             raise e

   2  ~\Documents\Projects\GITHUB\poetry\src\poetry\repositories\http.py:272 in _get_response
       270│         url = self._url + endpoint
       271│         try:
     → 272│             response: requests.Response = self.session.get(
       273│                 url, raise_for_status=False, timeout=REQUESTS_TIMEOUT
       274│             )

   1  ~\Documents\Projects\GITHUB\poetry\src\poetry\utils\authenticator.py:274 in get
       272│ 
       273│     def get(self, url: str, **kwargs: Any) -> requests.Response:
     → 274│         return self.request("get", url, **kwargs)
       275│ 
       276│     def post(self, url: str, **kwargs: Any) -> requests.Response:

  PoetryException

  'latin-1' codec can't encode characters in position 0-5: ordinal not in range(256)

  at ~\Documents\Projects\GITHUB\poetry\src\poetry\utils\authenticator.py:220 in request
      216│                 request = requests.auth.HTTPBasicAuth(
      217│                     credential.username or "", credential.password or ""
      218│                 )(request)
      219│             except UnicodeEncodeError as e:
    → 220│                 raise PoetryException(e)
      221│ 
      222│         session = self.get_session(url=url)
      223│         prepared_request = session.prepare_request(request)
      224│ 

[dimbleby]

solve() has a fairly wide try-catch with a base Exception

but since it re-raises that exception that's not masking anything, surely? Seems plausible that there's a too-broad-except somewhere, but not that one?

[kristang]

@dimbleby: Yes, it might be somewhere else. That was just some extra info.

[dimbleby]

extra, but I think not relevant!

poetry/src/poetry/mixology/version_solver.py

Line 400 in 0ffe91c

except ValueError:

looks like a more plausible candidate to me, or possibly one of the other places where poetry catches ValueError - that would capture a unicode error.

[dimbleby]

so far as I can see that try-except can simply be removed, which should allow your error to propagate. If you fancy submitting an MR, I reckon that's the way to go

[kristang]

I'm not that experienced in best-practice in terms of changes like this, but it seems a bit dangerous to just remove a try-except like that - but if you feel confident in doing so I have created PR #6786.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.