Cannot resolve python=^3.4 with httpx

[pjw91]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

Poetry
Version: 1.2.0a0 (latest b753aaf4)
Python:  3.8.5

Virtualenv
Python:         3.8.5
Implementation: CPython
Path:           /dev/shm/newp/.venv
Valid:          True

System
Platform: linux (Ubuntu 20.04.2 aarch64)
OS:       posix
Python:   /usr

• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/pjw91/ab99d529e93f2f4c65dc395ac62f06dc

Issue

It's known that the package httpx requires Python>=3.6.

I want my package supports Python>=3.4, and, if there's Python>=3.6, depends on httpx, too.

However, it (poetry lock) only works when I specify tool.poetry.dependencies.python="^3.X" where X>=5.

When I specify python="^3.4", poetry lock reports a SolverProblemError.

The detail debug info:

Loading configuration file /home/pjw91/.config/pypoetry/config.toml
Loading configuration file /dev/shm/newp/poetry.toml
Using virtualenv: /dev/shm/newp/.venv
Updating dependencies
Resolving dependencies...
   1: fact: newp is 0.1.0
   1: derived: newp
   1: fact: newp depends on httpx (^0.17.1)
   1: selecting newp (0.1.0)
   1: derived: httpx[http2] (>=0.17.1,<0.18.0)
PyPI: No release information found for httpx-0.0.1, skipping
PyPI: 1 packages found for httpx >=0.17.1,<0.18.0
   1: fact: httpx (0.17.1) depends on httpx (0.17.1)
   1: fact: httpx (0.17.1) depends on certifi (*)
   1: fact: httpx (0.17.1) depends on sniffio (*)
   1: fact: httpx (0.17.1) depends on rfc3986 (>=1.3,<2)
   1: fact: httpx (0.17.1) depends on httpcore (>=0.12.1,<0.13)
   1: fact: httpx (0.17.1) depends on h2 (>=3.0.0,<4.0.0)
   1: selecting httpx[http2] (0.17.1)
   1: derived: h2 (>=3.0.0,<4.0.0)
   1: derived: httpcore (>=0.12.1,<0.13)
   1: derived: rfc3986[idna2008] (>=1.3,<2)
   1: derived: sniffio
   1: derived: certifi
   1: derived: httpx (==0.17.1)
PyPI: No release information found for h2-0.1.0, skipping
PyPI: 5 packages found for h2 >=3.0.0,<4.0.0
PyPI: 3 packages found for httpcore >=0.12.1,<0.13
PyPI: No release information found for rfc3986-0.0.0, skipping
PyPI: 4 packages found for rfc3986 >=1.3,<2
PyPI: 4 packages found for sniffio *
PyPI: No release information found for certifi-0, skipping
PyPI: 43 packages found for certifi *
PyPI: No release information found for httpx-0.0.1, skipping
PyPI: 1 packages found for httpx 0.17.1
   1: fact: httpx (0.17.1) depends on certifi (*)
   1: fact: httpx (0.17.1) depends on sniffio (*)
   1: fact: httpx (0.17.1) depends on rfc3986 (>=1.3,<2)
   1: fact: httpx (0.17.1) depends on httpcore (>=0.12.1,<0.13)
   1: selecting httpx (0.17.1)
   1: fact: httpcore (0.12.3) depends on h11 (<1.0.0)
   1: fact: httpcore (0.12.3) depends on sniffio (>=1.0.0,<2.0.0)
   1: selecting httpcore (0.12.3)
   1: derived: sniffio (>=1.0.0,<2.0.0)
   1: derived: h11 (<1.0.0)
PyPI: No release information found for h11-0.0.1, skipping
PyPI: 9 packages found for h11 <1.0.0
   1: fact: sniffio (1.2.0) requires Python >=3.5
   1: derived: not sniffio (==1.2.0)
   1: fact: sniffio (1.1.0) requires Python >=3.5
   1: derived: not sniffio (==1.1.0)
   1: fact: sniffio (1.0.0) requires Python >=3.5
   1: derived: not sniffio (==1.0.0)
   1: fact: no versions of sniffio match >1.0.0,<1.1.0 || >1.1.0,<1.2.0 || >1.2.0,<2.0.0
   1: conflict: no versions of sniffio match >1.0.0,<1.1.0 || >1.1.0,<1.2.0 || >1.2.0,<2.0.0
   1: ! sniffio (>1.0.0,!=1.1.0,!=1.2.0,<2.0.0) is partially satisfied by not sniffio (==1.0.0)
   1: ! which is caused by "sniffio (1.0.0) requires Python >=3.5"
   1: ! thus: sniffio is forbidden
   1: ! sniffio (>=1.0.0,!=1.1.0,!=1.2.0,<2.0.0) is partially satisfied by not sniffio (==1.1.0)
   1: ! which is caused by "sniffio (1.1.0) requires Python >=3.5"
   1: ! thus: sniffio is forbidden
   1: ! sniffio (>=1.0.0,!=1.2.0,<2.0.0) is partially satisfied by not sniffio (==1.2.0)
   1: ! which is caused by "sniffio (1.2.0) requires Python >=3.5"
   1: ! thus: sniffio is forbidden
   1: fact: sniffio is forbidden
   1: derived: not sniffio (>=1.0.0,<2.0.0)
   1: derived: not httpcore (==0.12.3)
   1: fact: httpx (0.17.1) depends on httpx (0.17.1)
   1: fact: httpx (0.17.1) depends on certifi (*)
   1: fact: httpx (0.17.1) depends on sniffio (*)
   1: fact: httpx (0.17.1) depends on rfc3986 (>=1.3,<2)
   1: fact: httpx (0.17.1) depends on httpcore (>=0.12.1,<0.13)
   1: fact: httpx (0.17.1) depends on h2 (>=3.0.0,<4.0.0)
   2: selecting httpx[http2] (0.17.1)
   2: derived: h2 (>=3.0.0,<4.0.0)
   2: derived: httpcore (>=0.12.1,<0.13)
   2: derived: rfc3986[idna2008] (>=1.3,<2)
   2: derived: sniffio
   2: derived: certifi
   2: derived: httpx (==0.17.1)
   2: selecting sniffio (0.0.0)
   2: fact: httpx (0.17.1) depends on certifi (*)
   2: fact: httpx (0.17.1) depends on sniffio (*)
   2: fact: httpx (0.17.1) depends on rfc3986 (>=1.3,<2)
   2: fact: httpx (0.17.1) depends on httpcore (>=0.12.1,<0.13)
   2: selecting httpx (0.17.1)
   2: fact: httpcore (0.12.2) requires Python >=3.6
   2: derived: not httpcore (==0.12.2)
   2: fact: httpcore (0.12.1) requires Python >=3.6
   2: derived: not httpcore (==0.12.1)
   2: fact: no versions of httpcore match >0.12.1,<0.12.2 || >0.12.2,<0.12.3 || >0.12.3,<0.13
   2: conflict: no versions of httpcore match >0.12.1,<0.12.2 || >0.12.2,<0.12.3 || >0.12.3,<0.13
   2: ! httpcore (>0.12.1,!=0.12.2,!=0.12.3,<0.13) is partially satisfied by not httpcore (==0.12.1)
   2: ! which is caused by "httpcore (0.12.1) requires Python >=3.6"
   2: ! thus: httpcore is forbidden
   2: fact: httpcore is forbidden
   2: derived: not httpcore (>=0.12.1,!=0.12.2,!=0.12.3,<0.13)
   2: conflict: httpcore (0.12.2) requires Python >=3.6
   2: ! httpcore (==0.12.2) is partially satisfied by not httpcore (>=0.12.1,!=0.12.2,!=0.12.3,<0.13)
   2: ! which is caused by "httpcore is forbidden"
   2: ! thus: httpcore is forbidden
   2: fact: httpcore is forbidden
   2: derived: not httpcore (>=0.12.1,!=0.12.3,<0.13)
   2: derived: not httpx (==0.17.1)
   2: derived: not httpx[http2] (==0.17.1)
   2: fact: no versions of httpx match >0.17.1,<0.18.0
   2: conflict: no versions of httpx match >0.17.1,<0.18.0
   2: ! httpx[http2] (>0.17.1,<0.18.0) is partially satisfied by not httpx[http2] (==0.17.1)
   2: ! which is caused by "httpx (0.17.1) depends on httpcore (>=0.12.1,<0.13)"
   2: ! thus: httpx (>=0.17.1,<0.18.0) requires httpcore (>=0.12.1,<0.13)
   2: ! not httpcore (>=0.12.1,<0.13) is partially satisfied by not httpcore (>=0.12.1,!=0.12.3,<0.13)
   2: ! which is caused by "httpcore is forbidden"
   2: ! thus: httpx (>=0.17.1,<0.18.0) requires httpcore (0.12.3)
   2: ! not httpcore (==0.12.3) is satisfied by not httpcore (==0.12.3)
   2: ! which is caused by "httpcore (0.12.3) depends on sniffio (>=1.0.0,<2.0.0)"
   2: ! thus: httpx (>=0.17.1,<0.18.0) requires sniffio (>=1.0.0,<2.0.0)
   2: ! not sniffio (>=1.0.0,<2.0.0) is satisfied by not sniffio (>=1.0.0,<2.0.0)
   2: ! which is caused by "sniffio is forbidden"
   2: ! thus: httpx is forbidden
   2: ! httpx[http2] (>=0.17.1,<0.18.0) is satisfied by httpx[http2] (>=0.17.1,<0.18.0)
   2: ! which is caused by "newp depends on httpx (^0.17.1)"
   2: ! thus: version solving failed
   2: Version solving took 0.562 seconds.
   2: Tried 2 solutions.

  Stack trace:

  4  .venv/lib/python3.8/site-packages/poetry/puzzle/solver.py:251 in _solve
      249│ 
      250│         try:
    → 251│             result = resolve_version(
      252│                 self._package, self._provider, locked=locked, use_latest=use_latest
      253│             )

  3  .venv/lib/python3.8/site-packages/poetry/mixology/__init__.py:24 in resolve_version
       22│     solver = VersionSolver(root, provider, locked=locked, use_latest=use_latest)
       23│ 
    →  24│     return solver.solve()
       25│ 

  2  .venv/lib/python3.8/site-packages/poetry/mixology/version_solver.py:80 in solve
       78│             next = self._root.name
       79│             while next is not None:
    →  80│                 self._propagate(next)
       81│                 next = self._choose_package_version()
       82│ 

  1  .venv/lib/python3.8/site-packages/poetry/mixology/version_solver.py:120 in _propagate
      118│                     # where that incompatibility will allow us to derive new assignments
      119│                     # that avoid the conflict.
    → 120│                     root_cause = self._resolve_conflict(incompatibility)
      121│ 
      122│                     # Back jumping erases all the assignments we did at the previous

  SolveFailure

  The current project's Python requirement (>=3.4,<4.0) is not compatible with some of the required packages Python requirement:
    - httpcore requires Python >=3.6, so it will not be satisfied for Python >=3.4,<3.6
    - httpcore requires Python >=3.6, so it will not be satisfied for Python >=3.4,<3.6
    - sniffio requires Python >=3.5, so it will not be satisfied for Python >=3.4,<3.5
    - sniffio requires Python >=3.5, so it will not be satisfied for Python >=3.4,<3.5
    - sniffio requires Python >=3.5, so it will not be satisfied for Python >=3.4,<3.5
  
      Because no versions of httpcore match >0.12.1,<0.12.2 || >0.12.2,<0.12.3 || >0.12.3,<0.13
   and httpcore (0.12.1) requires Python >=3.6, httpcore is forbidden.
      And because httpcore (0.12.2) requires Python >=3.6, httpcore is forbidden.
      Because no versions of httpx match >0.17.1,<0.18.0
   and httpx (0.17.1) depends on httpcore (>=0.12.1,<0.13), httpx (>=0.17.1,<0.18.0) requires httpcore (>=0.12.1,<0.13).
      Thus, httpx (>=0.17.1,<0.18.0) requires httpcore (0.12.3).
  (1) So, because httpcore (0.12.3) depends on sniffio (>=1.0.0,<2.0.0), httpx (>=0.17.1,<0.18.0) requires sniffio (>=1.0.0,<2.0.0).
  
      Because no versions of sniffio match >1.0.0,<1.1.0 || >1.1.0,<1.2.0 || >1.2.0,<2.0.0
   and sniffio (1.0.0) requires Python >=3.5, sniffio is forbidden.
      And because sniffio (1.1.0) requires Python >=3.5
   and sniffio (1.2.0) requires Python >=3.5, sniffio is forbidden.
      And because httpx (>=0.17.1,<0.18.0) requires sniffio (>=1.0.0,<2.0.0) (1), httpx is forbidden
      So, because newp depends on httpx (^0.17.1), version solving failed.

  at .venv/lib/python3.8/site-packages/poetry/mixology/version_solver.py:314 in _resolve_conflict
      310│             )
      311│             self._log(f'{bang} which is caused by "{most_recent_satisfier.cause}"')
      312│             self._log(f"{bang} thus: {incompatibility}")
      313│ 
    → 314│         raise SolveFailure(incompatibility)
      315│ 
      316│     def _choose_package_version(self) -> Optional[str]:
      317│         """
      318│         Tries to select a version of a required package.

The following error occurred when trying to handle this error:


  Stack trace:

  11  .venv/lib/python3.8/site-packages/cleo/application.py:314 in run
       312│ 
       313│             try:
     → 314│                 exit_code = self._run(io)
       315│             except Exception as e:
       316│                 if not self._catch_exceptions:

  10  .venv/lib/python3.8/site-packages/poetry/console/application.py:164 in _run
       162│         self._load_plugins(io)
       163│ 
     → 164│         return super()._run(io)
       165│ 
       166│     def _configure_io(self, io: IO) -> None:

   9  .venv/lib/python3.8/site-packages/cleo/application.py:409 in _run
       407│                 io.set_input(ArgvInput(argv))
       408│ 
     → 409│         exit_code = self._run_command(command, io)
       410│         self._running_command = None
       411│ 

   8  .venv/lib/python3.8/site-packages/cleo/application.py:451 in _run_command
       449│ 
       450│         if error is not None:
     → 451│             raise error
       452│ 
       453│         return event.exit_code

   7  .venv/lib/python3.8/site-packages/cleo/application.py:435 in _run_command
       433│ 
       434│             if event.command_should_run():
     → 435│                 exit_code = command.run(io)
       436│             else:
       437│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

   6  .venv/lib/python3.8/site-packages/cleo/commands/base_command.py:118 in run
       116│         io.input.validate()
       117│ 
     → 118│         status_code = self.execute(io)
       119│ 
       120│         if status_code is None:

   5  .venv/lib/python3.8/site-packages/cleo/commands/command.py:85 in execute
        83│ 
        84│         try:
     →  85│             return self.handle()
        86│         except KeyboardInterrupt:
        87│             return 1

   4  .venv/lib/python3.8/site-packages/poetry/console/commands/lock.py:47 in handle
        45│         self._installer.lock(update=not self.option("no-update"))
        46│ 
     →  47│         return self._installer.run()
        48│ 

   3  .venv/lib/python3.8/site-packages/poetry/installation/installer.py:112 in run
       110│         local_repo = Repository()
       111│ 
     → 112│         return self._do_install(local_repo)
       113│ 
       114│     def dry_run(self, dry_run: bool = True) -> "Installer":

   2  .venv/lib/python3.8/site-packages/poetry/installation/installer.py:252 in _do_install
       250│             )
       251│ 
     → 252│             ops = solver.solve(use_latest=self._whitelist)
       253│         else:
       254│             self._io.write_line("Installing dependencies from lock file")

   1  .venv/lib/python3.8/site-packages/poetry/puzzle/solver.py:78 in solve
        76│         with self._provider.progress():
        77│             start = time.time()
     →  78│             packages, depths = self._solve(use_latest=use_latest)
        79│             end = time.time()
        80│ 

  SolverProblemError

  The current project's Python requirement (>=3.4,<4.0) is not compatible with some of the required packages Python requirement:
    - httpcore requires Python >=3.6, so it will not be satisfied for Python >=3.4,<3.6
    - httpcore requires Python >=3.6, so it will not be satisfied for Python >=3.4,<3.6
    - sniffio requires Python >=3.5, so it will not be satisfied for Python >=3.4,<3.5
    - sniffio requires Python >=3.5, so it will not be satisfied for Python >=3.4,<3.5
    - sniffio requires Python >=3.5, so it will not be satisfied for Python >=3.4,<3.5
  
      Because no versions of httpcore match >0.12.1,<0.12.2 || >0.12.2,<0.12.3 || >0.12.3,<0.13
   and httpcore (0.12.1) requires Python >=3.6, httpcore is forbidden.
      And because httpcore (0.12.2) requires Python >=3.6, httpcore is forbidden.
      Because no versions of httpx match >0.17.1,<0.18.0
   and httpx (0.17.1) depends on httpcore (>=0.12.1,<0.13), httpx (>=0.17.1,<0.18.0) requires httpcore (>=0.12.1,<0.13).
      Thus, httpx (>=0.17.1,<0.18.0) requires httpcore (0.12.3).
  (1) So, because httpcore (0.12.3) depends on sniffio (>=1.0.0,<2.0.0), httpx (>=0.17.1,<0.18.0) requires sniffio (>=1.0.0,<2.0.0).
  
      Because no versions of sniffio match >1.0.0,<1.1.0 || >1.1.0,<1.2.0 || >1.2.0,<2.0.0
   and sniffio (1.0.0) requires Python >=3.5, sniffio is forbidden.
      And because sniffio (1.1.0) requires Python >=3.5
   and sniffio (1.2.0) requires Python >=3.5, sniffio is forbidden.
      And because httpx (>=0.17.1,<0.18.0) requires sniffio (>=1.0.0,<2.0.0) (1), httpx is forbidden
      So, because newp depends on httpx (^0.17.1), version solving failed.

  at .venv/lib/python3.8/site-packages/poetry/puzzle/solver.py:259 in _solve
      255│             packages = result.packages
      256│         except OverrideNeeded as e:
      257│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      258│         except SolveFailure as e:
    → 259│             raise SolverProblemError(e)
      260│ 
      261│         results = dict(
      262│             depth_first_search(
      263│                 PackageNode(self._package, packages), aggregate_package_nodes

[abn]

This is quite odd. The following test in test_solver.py succeeds.

def test_solver_sub_dependencies_with_not_supported_python_version_x(
    solver, repo, package
):
    # uncomment for live test
    # from poetry.repositories.pypi_repository import PyPiRepository
    # solver._pool = Pool([PyPiRepository()])
    # solver.provider._pool = solver._pool

    solver.provider.set_package_python_versions("^3.4")

    package.add_dependency(
        Factory.create_dependency("httpx", {"version": "^0.17.1", "python": "^3.6"})
    )

    httpx = get_package("httpx", "0.17.1")
    httpx.python_versions = ">=3.6"

    httpcore = get_package("httpcore", "0.12.3")
    httpcore.python_versions = ">=3.6"

    sniffio = get_package("sniffio", "1.2.0")
    sniffio.python_versions = ">=3.5"

    httpx.add_dependency(Factory.create_dependency("httpcore", {"version": ">=0.12.1,<0.13"}))
    httpx.add_dependency(Factory.create_dependency("sniffio", {"version": "*"}))
    httpcore.add_dependency(Factory.create_dependency("sniffio", {"version": "==1.*"}))

    repo.add_package(httpx)
    repo.add_package(httpcore)
    repo.add_package(sniffio)

    ops = solver.solve()

    check_solver_result(ops, [
        {'job': 'install', 'package': sniffio, 'skipped': False},
        {'job': 'install', 'package': httpcore, 'skipped': False},
        {'job': 'install', 'package': httpx, 'skipped': False}
    ])

However, uncommenting the top bit to enable a live test fails. Seems to be due to how the package gets instantianted for a PyPiRepository. Either the mock test case is invalid, or somehow in the PyPI case the python constraint is not corectly propagated from the top level dependency when solving.

@sdispater any thoughts on this one?

PS: if solving is fixed for the live case, the check_solver_result will fail since the live case installs more packages.

[sdispater]

The following test (not live) reproduces the issue:

def test_solver_sub_dependencies_with_not_supported_python_version_x(
    solver, repo, package
):
    solver.provider.set_package_python_versions("^3.4")

    package.add_dependency(
        Factory.create_dependency("httpx", {"version": "^0.17.1", "python": "^3.6"})
    )

    httpx = get_package("httpx", "0.17.1")
    httpx.python_versions = ">=3.6"

    httpcore = get_package("httpcore", "0.12.3")
    httpcore.python_versions = ">=3.6"

    sniffio_1_1_0 = get_package("sniffio", "1.1.0")
    sniffio_1_1_0.python_versions = ">=3.5"

    sniffio = get_package("sniffio", "1.2.0")
    sniffio.python_versions = ">=3.5"

    httpx.add_dependency(
        Factory.create_dependency("httpcore", {"version": ">=0.12.1,<0.13"})
    )
    httpx.add_dependency(Factory.create_dependency("sniffio", {"version": "*"}))
    httpcore.add_dependency(Factory.create_dependency("sniffio", {"version": "==1.*"}))

    repo.add_package(httpx)
    repo.add_package(httpcore)
    repo.add_package(sniffio)
    repo.add_package(sniffio_1_1_0)

    ops = solver.solve()

    check_solver_result(
        ops,
        [
            {"job": "install", "package": sniffio, "skipped": False},
            {"job": "install", "package": httpcore, "skipped": False},
            {"job": "install", "package": httpx, "skipped": False},
        ],
    )

The issue comes from the fact that the httpcore dependency is treated first which trips up the resolver because httpcore also depends on sniffio. I am not quite sure why yet but it seems to be an issue with the propagation of the Python constraint.

[sdispater]

I tracked it down to the Dependency.with_constraint() method in poetry-core not copying the markers and transitive markers.

I'll make a PR there.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.