Bug: cyclic dependency with `tomli==1.2.1` and `flit_core==3.2.0` #451

lummax · 2021-10-11T08:53:57Z

I can not download tomli because there is a cyclic dependency with flit_core.

Reproducer:

python3.9 -m venv test
./test/bin/pip download --no-binary=":all:"  tomli  # fails
./test/bin/pip download --no-binary=":all:" flit_core # fails

Example output:

Collecting tomli
  File was already downloaded XXX/tomli-1.2.1.tar.gz
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 2:
   command: XXX/test/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-91g152mw/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core>=3.2.0,<4'
       cwd: None
  Complete output (31 lines):
  Collecting flit_core<4,>=3.2.0
    Downloading flit_core-3.4.0.tar.gz (27 kB)
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
      Preparing wheel metadata: started
      Preparing wheel metadata: finished with status 'done'
  Collecting tomli
    Downloading tomli-1.2.1.tar.gz (14 kB)
  ERROR: Exception:
  Traceback (most recent call last):
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
      status = self.run(options, args)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 357, in run
      resolver.resolve(requirement_set)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
      discovered_reqs.extend(self._resolve_one(requirement_set, req))
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
      abstract_dist = self._get_abstract_dist_for(req_to_install)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
      abstract_dist = self.preparer.prepare_linked_requirement(req)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 515, in prepare_linked_requirement
      abstract_dist = _get_prepared_distribution(
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 94, in _get_prepared_distribution
      with req_tracker.track(req):
    File "/usr/lib/python3.9/contextlib.py", line 117, in __enter__
      return next(self.gen)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/req/req_tracker.py", line 148, in track
      self.add(req)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/req/req_tracker.py", line 115, in add
      raise LookupError(message)
  LookupError: https://files.pythonhosted.org/packages/75/50/973397c5ba854445bcc396b593b5db1958da6ab8d665b27397daa1497018/tomli-1.2.1.tar.gz#sha256=a5b75cb6f3968abb47af1b40c1819dc519ea82bcc065776a866e8d74c5ca9442 (from https://pypi.org/simple/tomli/) (requires-python:>=3.6) is already being built: tomli from https://files.pythonhosted.org/packages/75/50/973397c5ba854445bcc396b593b5db1958da6ab8d665b27397daa1497018/tomli-1.2.1.tar.gz#sha256=a5b75cb6f3968abb47af1b40c1819dc519ea82bcc065776a866e8d74c5ca9442
  ----------------------------------------
ERROR: Command errored out with exit status 2: XXX/test/bin/python3.9 XXX/test/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-91g152mw/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core>=3.2.0,<4' Check the logs for full command output.

The text was updated successfully, but these errors were encountered:

lummax · 2021-10-11T08:54:39Z

Companion ticket: hukkin/tomli#130

takluyver · 2021-10-11T09:44:49Z

Have a look at the bootstrapping page in the docs.

My assumption is that most 'normal' users will use pre-built wheels - at least of some of these initial packages - and thus avoid the situation. If you need to build everything from tarballs - e.g. to populate another packaging system, then you need a little bit of manual intervention, and that's what the bootstrapping document is about.

lummax · 2021-10-11T09:57:42Z

I agree, my use case is not that of a normal user. But even bootstrapping does not help: hukkin/tomli#130 (comment)

This actually broke the release-pipeline for a product in maintenance. Manual intervention was difficult but possible. Lockfiles don't help here, as pip download does not respect them during the "build-dependencies" setup.

My only wish is for a little stricter dependency constraints so this does not happen again.

Feel free to close.

takluyver · 2021-10-11T10:42:03Z

I think ultimately the fix for this will be to have a TOML parser in the Python standard library. As things stand, we've standardised the TOML format for packaging, but we need a third party package to parse it. Tomli's packaging information is in a TOML file, so it needs a TOML parser to build it. But adding to the standard library is slow and contentious, so we'll have to live with that for a while.

dyeager-recurve · 2021-10-11T14:05:59Z

Wanted to just add here that we also have a release pipeline that was broken with this release - pathway forward to "fix" the issue is clear but wanted to flag it, exact same issue as described by @lummax

pradyunsg · 2021-10-11T20:07:52Z

FWIW, this has broken pip's own vendoring processes. It's probably fine, since those processes need some TLC anyway -- it's worth flagging, but I don't think that changes anything. :)

ap604 · 2021-10-11T21:28:38Z

Hello - we are using GCP Dataflow using python and we are seeing an error that is precisely related to this cyclic dependency. This breaks us unfortunately - and it is not easy for us to change the way it wants to run (because this is GCP's code and not our code).

takluyver · 2021-10-12T07:48:57Z

Can I ask you (or anyone) to investigate in a bit more detail what GCP dataflow is actually trying to do? I've never used it.

ap604 · 2021-10-12T11:59:59Z

We are using Dataflow's Flex Templates (https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates)
As a part of this, we build our own container image based off a GCP base image (of which we do not have any direct access to).

When the Dataflow job kicks off, the Apache beam program runs and it starts off by installing some python packages. This is where we are seeing the error - and it exits. This started failing yesterday, which coincided with the new flit_core release. Also, the log message below indicates the problem. A lot of these packages are required - and not specified by code in my control.

Anything you can do would be much appreciated - thanks!

Output from execution of subprocess: b'Collecting google-cloud-secret-manager==2.3.0
  Downloading google-cloud-secret-manager-2.3.0.tar.gz (90 kB)
  Saved /tmp/dataflow-requirements-cache/google-cloud-secret-manager-2.3.0.tar.gz
Collecting google-api-core[grpc]<2.0.0dev,>=1.22.2
  Downloading google-api-core-1.31.3.tar.gz (97 kB)
  Saved /tmp/dataflow-requirements-cache/google-api-core-1.31.3.tar.gz
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
  Downloading grpc-google-iam-v1-0.12.3.tar.gz (13 kB)
  Saved /tmp/dataflow-requirements-cache/grpc-google-iam-v1-0.12.3.tar.gz
Collecting proto-plus>=1.4.0
  Downloading proto-plus-1.19.4.tar.gz (48 kB)
  Saved /tmp/dataflow-requirements-cache/proto-plus-1.19.4.tar.gz
Collecting libcst>=0.2.5
  Downloading libcst-0.3.21.tar.gz (576 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-qhpytueu/normal --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- setuptools_scm
       cwd: None
  Complete output (69 lines):
  Collecting setuptools_scm
    Downloading setuptools_scm-6.3.2.tar.gz (57 kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-9hty5unr/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=45' wheel 'tomli>=1.0' 'packaging>=20.0'
         cwd: None
    Complete output (57 lines):
    Collecting setuptools>=45
      Using cached setuptools-58.2.0.tar.gz (2.3 MB)
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Getting requirements to build wheel: started
      Getting requirements to build wheel: finished with status 'done'
        Preparing wheel metadata: started
        Preparing wheel metadata: finished with status 'done'
    Collecting wheel
      Using cached wheel-0.37.0.tar.gz (65 kB)
    Collecting tomli>=1.0
      Downloading tomli-1.2.1.tar.gz (14 kB)
      Installing build dependencies: started
      Installing build dependencies: finished with status 'error'
      ERROR: Command errored out with exit status 2:
       command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-zp5sz97w/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core>=3.2.0,<4'
           cwd: None
      Complete output (35 lines):
      Collecting flit_core<4,>=3.2.0
        Downloading flit_core-3.4.0.tar.gz (27 kB)
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
          Preparing wheel metadata: started
          Preparing wheel metadata: finished with status 'done'
      Collecting tomli
        Using cached tomli-1.2.1.tar.gz (14 kB)
      ERROR: Exception:
      Traceback (most recent call last):
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
          status = self.run(options, args)
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
          return func(self, options, args)
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 324, in run
          reqs, check_supported_wheels=not options.target_dir
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
          discovered_reqs.extend(self._resolve_one(requirement_set, req))
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
          abstract_dist = self._get_abstract_dist_for(req_to_install)
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 340, in _get_abstract_dist_for
          abstract_dist = self.preparer.prepare_linked_requirement(req)
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 483, in prepare_linked_requirement
          req, self.req_tracker, self.finder, self.build_isolation,
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 90, in _get_prepared_distribution
          with req_tracker.track(req):
        File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
          return next(self.gen)
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 148, in track
          self.add(req)
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 114, in add
          raise LookupError(message)
      LookupError: https://files.pythonhosted.org/packages/75/50/973397c5ba854445bcc396b593b5db1958da6ab8d665b27397daa1497018/tomli-1.2.1.tar.gz#sha256=a5b75cb6f3968abb47af1b40c1819dc519ea82bcc065776a866e8d74c5ca9442 (from https://pypi.org/simple/tomli/) (requires-python:>=3.6) is already being built: tomli>=1.0 from https://files.pythonhosted.org/packages/75/50/973397c5ba854445bcc396b593b5db1958da6ab8d665b27397daa1497018/tomli-1.2.1.tar.gz#sha256=a5b75cb6f3968abb47af1b40c1819dc519ea82bcc065776a866e8d74c5ca9442
      WARNING: You are using pip version 20.2.4; however, version 21.2.4 is available.
      You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
      ----------------------------------------
    ERROR: Command errored out with exit status 2: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-zp5sz97w/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core>=3.2.0,<4' Check the logs for full command output.
    WARNING: You are using pip version 20.2.4; however, version 21.2.4 is available.
    You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
    ----------------------------------------
  ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-9hty5unr/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=45' wheel 'tomli>=1.0' 'packaging>=20.0' Check the logs for full command output.
  WARNING: You are using pip version 20.2.4; however, version 21.2.4 is available.
  You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-qhpytueu/normal --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- setuptools_scm Check the logs for full command output.
WARNING: You are using pip version 20.2.4; however, version 21.2.4 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.

ap604 · 2021-10-12T12:23:43Z

The pip command that it used to kick off the installation:

{"container_id":"453c923f035cb8125c1b4de5833928741d96f0b8d70e64a8242eb4a89f41896a","severity":"INFO","time":"2021/10/11 19:10:30.741774","line":"exec.go:64","message":"subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/dataflow/template/requirements.txt', '--exists-action', 'i', '--no-binary', ':all:']' returned non-zero exit status 1."}

juannyG · 2021-10-12T14:38:51Z

Looks like if you use a version of pip < 21.x , installs get hosed.

pip 19 yells at you about not supporting pep 517:

  pip._vendor.pep517.wrappers.BackendUnavailable
  You are using pip version 19.0.3, however version 21.3 is available.
  You should consider upgrading via the 'pip install --upgrade pip' command.

pip 20 - not so much (as seen in the above comment ) - it tries and fails.

pip>=21 has no issue.

Upgrading the pip version resolves the dataflow flex-template issue.

takluyver · 2021-10-12T15:03:48Z

Thanks! Quite a bit has changed in pip & packaging tools in the last couple of years - pip uses calendar versioning, so 19.x means released in 2019.

I see that people are 👍-ing @juannyG's comment, which looks like consensus that the issue specifically on GCP Dataflow is solved (or worked around :-) by upgrading pip. If that's not the case - if this is still a problem for some of you using GCP Dataflow even after upgrading to the newest pip, please do say.

I don't really have a good way to fix this, so I'm not promising anything. But having a sense of how many people are affected and how big an inconvenience it is might help to figure out whether a workaround of some kind is needed.

Per <pypa/flit#451 (comment)>.

darmadoo · 2021-10-19T23:48:51Z

I am running into a similar issue here. I am trying to install salt on MacOs through brew. This is where it is failing at the moment. It does look like I am using pip 21.0+

Test-MacBook-Pro:bin tester$   /usr/local/Cellar/salt/3004/libexec/bin/python3.7 /usr/local/Cellar/salt/3004/libexec/lib/python3.7/site-packages/pip -v --log /tmp/pip.log download --no-binary=":all:" tomli
Collecting tomli
  Using cached tomli-1.2.1.tar.gz (14 kB)
  Running command /usr/local/Cellar/salt/3004/libexec/bin/python3.7 /private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-build-env-b979ls4e/overlay --no-warn-script-location -v --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core>=3.2.0,<4'
  Using pip 21.3 from /private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip (python 3.7)
  Collecting flit_core<4,>=3.2.0
    Using cached flit_core-3.4.0.tar.gz (27 kB)
    Getting requirements to build wheel: started
    Running command /usr/local/Cellar/salt/3004/libexec/bin/python3.7 /var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/tmpz5ipgr3k get_requires_for_build_wheel /var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/tmp9qcknloo
    Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata (pyproject.toml): started
    Running command /usr/local/Cellar/salt/3004/libexec/bin/python3.7 /var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/tmpslh2846g prepare_metadata_for_build_wheel /var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/tmpr1iz10lc
    Preparing wheel metadata (pyproject.toml): finished with status 'done'
  Collecting tomli
    Using cached tomli-1.2.1.tar.gz (14 kB)
  ERROR: Exception:
  Traceback (most recent call last):
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
      status = run_func(*args)
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_internal/cli/req_command.py", line 204, in wrapper
      return func(self, options, args)
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_internal/commands/install.py", line 339, in run
      reqs, check_supported_wheels=not options.target_dir
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_internal/resolution/resolvelib/resolver.py", line 93, in resolve
      collected.requirements, max_rounds=try_to_avoid_resolution_too_deep
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 482, in resolve
      state = resolution.resolve(requirements, max_rounds=max_rounds)
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 374, in resolve
      failure_causes = self._attempt_to_pin_criterion(name)
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 214, in _attempt_to_pin_criterion
      criteria = self._get_updated_criteria(candidate)
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 205, in _get_updated_criteria
      self._add_to_criteria(criteria, requirement, parent=candidate)
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
      if not criterion.candidates:
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
      return bool(self._sequence)
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
      return any(self)
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
      return (c for c in iterator if id(c) not in self._incompatible_ids)
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
      candidate = func()
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
      version=version,
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_internal/resolution/resolvelib/candidates.py", line 287, in __init__
      version=version,
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
      self.dist = self._prepare()
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
      dist = self._prepare_distribution()
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_internal/resolution/resolvelib/candidates.py", line 292, in _prepare_distribution
      return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_internal/operations/prepare.py", line 481, in prepare_linked_requirement
      return self._prepare_linked_requirement(req, parallel_builds)
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_internal/operations/prepare.py", line 549, in _prepare_linked_requirement
      self.build_isolation,
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_internal/operations/prepare.py", line 57, in _get_prepared_distribution
      with req_tracker.track(req):
    File "/usr/local/Cellar/python@3.7/3.7.12_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
      return next(self.gen)
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_internal/req/req_tracker.py", line 122, in track
      self.add(req)
    File "/private/var/folders/hv/9nyfk38n5wn44qvy0tqwv7n80000gn/T/pip-standalone-pip-288jzhrz/__env_pip__.zip/pip/_internal/req/req_tracker.py", line 92, in add
      raise LookupError(message)
  LookupError: https://files.pythonhosted.org/packages/75/50/973397c5ba854445bcc396b593b5db1958da6ab8d665b27397daa1497018/tomli-1.2.1.tar.gz#sha256=a5b75cb6f3968abb47af1b40c1819dc519ea82bcc065776a866e8d74c5ca9442 (from https://pypi.org/simple/tomli/) (requires-python:>=3.6) is already being built: tomli from https://files.pythonhosted.org/packages/75/50/973397c5ba854445bcc396b593b5db1958da6ab8d665b27397daa1497018/tomli-1.2.1.tar.gz#sha256=a5b75cb6f3968abb47af1b40c1819dc519ea82bcc065776a866e8d74c5ca9442

takluyver · 2021-10-22T09:54:59Z

For most purposes - if you're just trying to get stuff installed and working - using --no-binary :all: is overkill. This prevents pip from downloading and installing a wheel, which is its default installation path. For pure Python packages (like flit & tomli), installing from wheels is pure upside: you get the same files installed but quicker.

If you want to build specific packages from source (normally those with compiled extensions), specify those like pip install --no-binary=h5py h5py, rather than using :all:.

The one case I'm aware of where people really want to avoid downloading wheels is when making packages for other packaging systems - Linux distributions, Spack, etc. If there's another scenario where this is important, let me know. These people will need a bit of manual fiddling as described in the bootstrapping page to get Tomli set up. There is no really good way to avoid this until & unless a TOML parser is added to the Python standard library.

hukkin · 2021-10-22T10:38:58Z

Any thoughts on changing Tomli's build backend from flit_core to poetry_core?

Poetry_core already bundles a TOML parser so I assume packagers who can't accept that already have a process in place for unbundling. I couldn't find any issues regarding this in Poetry's or Poetry-core's issue tracker so I get the feeling that unbundling may be less of a nuisance for packagers than bootstrapping using a source copy of a TOML parser. The latter has resulted in issues created on flit's and tomli's issue trackers, and some packagers (at least gentoo, arch, macports) setting up custom setup.py files and refusing to build Tomli using PEP517.

The lack of a standard library TOML parser is admittedly annoying, and means packagers need to either accept a bundled parser or set up a temporary one for the bootstrapping process. Maybe it's best if we make it easy on them by ensuring they only need to do this with one build backend (i.e. poetry_core) instead of all of them?

There's also the chance that I'm missing something critical or that this will only lead to these same issues being opened on poetry_core's issue tracker instead 😄

EDIT: Regarding my the last paragraph: if these packagers need e.g. the pypa/build tool (also depends on Tomli), then what I suggested probably won't help at all. But yeah I'm no packager so have limited knowledge of their required tools and processes.

takluyver · 2021-10-22T11:11:35Z

I believe that the packaging ecosystems that are strict about building everything from source also tend to strongly object to bundling, so I don't think a solution that relies on bundling a TOML parser in the build backend will make them happy. Even if they have streamlined unbundling in general, unbundling in this case just puts you back in the same dependency cycle (whether that's flit_core <-> tomli, or poetry_core <-> tomlkit). So it's going to need some kind of special intervention.

Maybe I'm wrong about this, though. 🙂 If there are significant use cases where people need to install from source but are fine with bundled dependencies, we can do that (either rely on Poetry, or bundle tomli in flit_core sdists).

My impression is that a lot of the people having issues don't actually need to build everything from source, but are doing that as a convenient shorthand for building something from source. It would be nice if pip had better ways to control this (e.g. 'build from source anything that involves native code' - but I don't think that's knowable ahead of time.)

hukkin · 2021-10-22T11:21:44Z

Even if they have streamlined unbundling in general, unbundling in this case just puts you back in the same dependency cycle (whether that's flit_core <-> tomli, or poetry_core <-> tomlkit). So it's going to need some kind of special intervention.

Yeah agreed. However, my theory was that maybe we can reduce the number of build backends where this special intervention is required to exactly one. And if yes, maybe it's the pragmatic thing to do?

takluyver · 2021-10-22T12:19:47Z

We could do, but poetry-core bundles quite a few packages, so it doesn't seem like the best starting point for bootstrapping. And if you unbundle those, some of them are built with setuptools, so we'd be chaining back to setuptools again, which I'd like to demonstrate isn't necessary. And setuptools has its own cycle if you use the PEP 517 APIs (setuptools <-> wheel).

Another option is that we could make a special backend to build tomli (either in flit_core.build_tomli or in tomli's own source tree) something like this:

import sys
sys.path.insert(0, '.')
from flit_core.buildapi import *

That would have the same effect as the manual workaround I've suggested for downstream packagers. But it wouldn't solve the issues people are having using pip, because flit_core would still have a dependency on tomli.

layday · 2021-10-30T09:35:05Z

Should build move to flit (see pypa/build#394), then to bootstrap Python you would only need five packages: flit_core, tomli, pep517, build and installer. If tomli switches to poetry-core, that's going to balloon to at least fourteen, not counting their build dependencies, assuming unbundling.

When building python dependencies with `pip install --no-binary :all:...`, we run into a bug caused by a cyclic dependency with `tomli==1.2.1` and `flit_core==3.2.0` while building dependencies of setuptools_scm. Apparently, it works with `pip >= 0.21` according to pypa/flit#451 (comment)

thinkingerrol · 2021-11-19T19:36:39Z

I am running into a similar issue here. I am trying to install salt on MacOs through brew. This is where it is failing at the moment. It does look like I am using pip 21.0+
Test-MacBook-Pro:bin tester$   /usr/local/Cellar/salt/3004/libexec/bin/python3.7 /usr/local/Cellar/salt/3004/libexec/lib/python3.7/site-packages/pip -v --log /tmp/pip.log download --no-binary=":all:" tomli

After pip install --upgrade pip I thought it didn't do anything although the installation claimed to be successful. Then I compared the output of:

pip --version
pip3 --version
pip3.8 --version

And changed my pip commands to use pip3.8 as only that one resulted in pip 21.x being used. And only then the cyclic dependency problem was fixed for me.

takluyver · 2021-11-19T19:41:17Z

If you've got multiple pip* commands which behave differently, it's likely that they point to different Python versions or environments. If there's any ambiguity, you can run python -m pip ... or path/to/python -m pip ... to control which Python you're running pip on (replacing ... with whatever subcommand or options you want).

takluyver · 2024-01-28T10:00:16Z

I'm going to close this now. Two things have happened to mitigate the cyclic dependency:

Python 3.11 introduced tomllib in the standard library, which flit_core will use if available. So on a new enough Python, flit_core can now build itself with no dependencies.
We bundled tomli into flit_core for older versions of Python, so in most cases there's no longer a dependency cycle.

That still leaves a problem to be solved for downstreams using Python 3.10 or older with a rigid rule of removing bundled dependencies - sorry about that. But I assume they have by now figured out some kind of workaround, however ugly, and it can be cleaned up as they update their Python version. So I don't think there's anything more to do here.

lummax mentioned this issue Oct 11, 2021

Bug: cyclic dependency with tomli==1.2.1 and flit_core==3.2.0 hukkin/tomli#130

Closed

pradyunsg mentioned this issue Oct 11, 2021

vendoring is broken, due to a cyclic dependency during license fetching pypa/pip#10570

Closed

cfm mentioned this issue Oct 12, 2021

Make targets build-debs[-notest] fails on new circular dependency between PyPI packages flit_core and tomli freedomofpress/securedrop#6137

Closed

cfm added a commit to freedomofpress/securedrop that referenced this issue Oct 13, 2021

[WIP] upgrades to pip ≥ 21 before installing translation requirements

3645ac4

Per <pypa/flit#451 (comment)>.

adamtheturtle mentioned this issue Oct 13, 2021

tests/test_admin/test_brew.py::test_brew failing on GitHub Actions VWS-Python/vws-cli#466

Closed

PiRK mentioned this issue Nov 18, 2021

fix AppImage deterministic build Bitcoin-ABC/ElectrumABC#159

Merged

Bo98 mentioned this issue Dec 2, 2021

rust 1.57.0 + apply patch to fix duplicate object file bug Homebrew/homebrew-core#90103

Closed

layday mentioned this issue Dec 29, 2021

Wheel violates PEP517 Build requirements pypa/wheel#429

Closed

takluyver closed this as completed Jan 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug: cyclic dependency with `tomli==1.2.1` and `flit_core==3.2.0` #451

Bug: cyclic dependency with `tomli==1.2.1` and `flit_core==3.2.0` #451

lummax commented Oct 11, 2021

lummax commented Oct 11, 2021

takluyver commented Oct 11, 2021

lummax commented Oct 11, 2021

takluyver commented Oct 11, 2021

dyeager-recurve commented Oct 11, 2021

pradyunsg commented Oct 11, 2021

ap604 commented Oct 11, 2021

takluyver commented Oct 12, 2021

ap604 commented Oct 12, 2021

ap604 commented Oct 12, 2021

juannyG commented Oct 12, 2021

takluyver commented Oct 12, 2021

darmadoo commented Oct 19, 2021

takluyver commented Oct 22, 2021

hukkin commented Oct 22, 2021 •

edited

takluyver commented Oct 22, 2021

hukkin commented Oct 22, 2021

takluyver commented Oct 22, 2021 •

edited

layday commented Oct 30, 2021 •

edited

thinkingerrol commented Nov 19, 2021 •

edited

takluyver commented Nov 19, 2021

takluyver commented Jan 28, 2024

Bug: cyclic dependency with tomli==1.2.1 and flit_core==3.2.0 #451

Bug: cyclic dependency with tomli==1.2.1 and flit_core==3.2.0 #451

Comments

lummax commented Oct 11, 2021

lummax commented Oct 11, 2021

takluyver commented Oct 11, 2021

lummax commented Oct 11, 2021

takluyver commented Oct 11, 2021

dyeager-recurve commented Oct 11, 2021

pradyunsg commented Oct 11, 2021

ap604 commented Oct 11, 2021

takluyver commented Oct 12, 2021

ap604 commented Oct 12, 2021

ap604 commented Oct 12, 2021

juannyG commented Oct 12, 2021

takluyver commented Oct 12, 2021

darmadoo commented Oct 19, 2021

takluyver commented Oct 22, 2021

hukkin commented Oct 22, 2021 • edited

takluyver commented Oct 22, 2021

hukkin commented Oct 22, 2021

takluyver commented Oct 22, 2021 • edited

layday commented Oct 30, 2021 • edited

thinkingerrol commented Nov 19, 2021 • edited

takluyver commented Nov 19, 2021

takluyver commented Jan 28, 2024

Bug: cyclic dependency with `tomli==1.2.1` and `flit_core==3.2.0` #451

Bug: cyclic dependency with `tomli==1.2.1` and `flit_core==3.2.0` #451

hukkin commented Oct 22, 2021 •

edited

takluyver commented Oct 22, 2021 •

edited

layday commented Oct 30, 2021 •

edited

thinkingerrol commented Nov 19, 2021 •

edited