[python][pip-compile] Dependabot cannot resolve dependency version if the lockfile is used as constraints in other manifest files. #6550

nwalsh1995 · 2023-01-30T23:31:42Z

Is there an existing issue for this?

I have searched the existing issues

Package ecosystem

pip/pip-compile

Package manager version

pip 22.3.1 pip-compile 6.12.1

Language version

3.7.5

Manifest location and content before the Dependabot update

No response

dependabot.yml content

No response

Updated dependency

No response

What you expected to see, versus what you actually saw

I expected dependabot to strip the extras out of this file since it is being used as a constraint to another file which dependabot will attempt to generate.

Instead dependabot does not strip the extras out of the produced file which means that the process fails with

pip._internal.exceptions.InstallationError: Constraints cannot have extras

Here is the code where dependabot runs pip-compile to resolve the version.

dependabot-core/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb

Lines 76 to 101 in 0a94b90

    
           filenames_to_compile.each do |filename| 
        
             # Shell out to pip-compile. 
        
             # This is slow, as pip-compile needs to do installs. 
        
             options = pip_compile_options(filename) 
        
             options_fingerprint = pip_compile_options_fingerprint(options) 
        
             run_pip_compile_command( 
        
               "pyenv exec pip-compile -v #{options} -P #{dependency.name} #{filename}", 
        
               fingerprint: "pyenv exec pip-compile -v #{options_fingerprint} -P <dependency_name> <filename>" 
        
             ) 
        
             next if dependency.top_level? 
        
             # Run pip-compile a second time for transient dependencies 
        
             # to make sure we do not update dependencies that are 
        
             # superfluous. pip-compile does not detect these when 
        
             # updating a specific dependency with the -P option. 
        
             # Running pip-compile a second time will automatically remove 
        
             # superfluous dependencies. Dependabot then marks those with 
        
             # update_not_possible. 
        
             write_original_manifest_files 
        
             run_pip_compile_command( 
        
               "pyenv exec pip-compile #{options} #{filename}", 
        
               fingerprint: "pyenv exec pip-compile #{options_fingerprint} <filename>" 
        
             ) 
        
           end

Given the manifest files below, notice how if we update a .in file that is being used as a -c to another file, then pip-compile will fail to run on that file.

Suggestion

There is a pip-tools discussion around making --strip-extras the default behavior: jazzband/pip-tools#1613. Therefore, pass --strip-extras by default in the two pip-compile calls here:

dependabot-core/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb

Line 83 in 0a94b90

"pyenv exec pip-compile -v #{options} -P #{dependency.name} #{filename}",

dependabot-core/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb

Line 98 in 0a94b90

"pyenv exec pip-compile #{options} #{filename}",

PR here #6551

Alternative to suggestion

dependabot supports parsing --strip-extras in the file_updater, it should support it in the version resolver as well:

dependabot-core/python/lib/dependabot/python/file_updater/pip_compile_file_updater.rb

Line 442 in 0a94b90

    
           options << "--strip-extras" if requirements_file.content.include?("--strip-extras")

Native package manager behavior

No response

Images of the diff or a link to the PR, issue, or logs

updater | ERROR <job_592338820> Error processing click (Dependabot::SharedHelpers::HelperSubprocessFailed)
updater | ERROR <job_592338820> Using indexes:
updater | <job_592338820>   https://pypi.org/simple
updater | <job_592338820> 
updater | <job_592338820>                           ROUND 1                           
updater | <job_592338820> /usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/pip/_internal/req/req_install.py:877: PipDeprecationWarning: DEPRECATION: Constraints are only allowed to take the form of a package name and a version specifier. Other forms were originally permitted as an accident of the implementation, but were undocumented. The new implementation of the resolver no longer supports these forms. A possible replacement is replacing the constraint with a requirement. Discussion can be found at https://github.com/pypa/pip/issues/8210
updater | <job_592338820>   issue=8210,
updater | <job_592338820> Traceback (most recent call last):
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/bin/pip-compile", line 8, in <module>
updater | <job_592338820>     sys.exit(cli())
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
updater | <job_592338820>     return self.main(*args, **kwargs)
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/click/core.py", line 1055, in main
updater | <job_592338820>     rv = self.invoke(ctx)
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
updater | <job_592338820>     return ctx.invoke(self.callback, **ctx.params)
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/click/core.py", line 760, in invoke
updater | <job_592338820>     return __callback(*args, **kwargs)
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
updater | <job_592338820>     return f(get_current_context(), *args, **kwargs)
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/piptools/scripts/compile.py", line 580, in cli
updater | <job_592338820>     results = resolver.resolve(max_rounds=max_rounds)
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/piptools/resolver.py", line 592, in resolve
updater | <job_592338820>     compatible_existing_constraints=compatible_existing_constraints,
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/piptools/resolver.py", line 625, in _do_resolve
updater | <job_592338820>     check_supported_wheels=not self.options.target_dir,
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
updater | <job_592338820>     collected = self.factory.collect_root_requirements(root_reqs)
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 481, in collect_root_requirements
updater | <job_592338820>     raise InstallationError(problem)
updater | <job_592338820> pip._internal.exceptions.InstallationError: Constraints cannot have extras
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:224:in `run_command'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:282:in `run_pip_compile_command'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:82:in `block (3 levels) in fetch_latest_resolvable_version_string'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:76:in `each'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:76:in `block (2 levels) in fetch_latest_resolvable_version_string'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/shared_helpers.rb:168:in `with_git_configured'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:72:in `block in fetch_latest_resolvable_version_string'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `block in in_a_temporary_directory'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `chdir'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `in_a_temporary_directory'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:71:in `fetch_latest_resolvable_version_string'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:57:in `resolvable?'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker.rb:39:in `latest_resolvable_version'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:74:in `preferred_resolvable_version'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:260:in `preferred_version_resolvable_with_unlock?'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker.rb:108:in `preferred_version_resolvable_with_unlock?'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:252:in `numeric_version_can_update?'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:202:in `version_can_update?'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:44:in `can_update?'
updater | ERROR <job_592338820> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:486:in `requirements_to_unlock'
updater | ERROR <job_592338820> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:259:in `check_and_create_pull_request'
updater | ERROR <job_592338820> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:109:in `check_and_create_pr_with_error_handling'
updater | ERROR <job_592338820> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:82:in `block in run'
updater | ERROR <job_592338820> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:82:in `each'
updater | ERROR <job_592338820> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:82:in `run'
updater | ERROR <job_592338820> /home/dependabot/dependabot-updater/lib/dependabot/update_files_job.rb:17:in `perform_job'
updater | ERROR <job_592338820> /home/dependabot/dependabot-updater/lib/dependabot/base_job.rb:50:in `run'
updater | ERROR <job_592338820> bin/update_files.rb:23:in `<main>'

Smallest manifest that reproduces the issue

requirements-shared.in

play-scraper

requirements-shared.txt. This is what dependabot will generate because it doesn't pass --strip-extras. Notice requests[security]

#
# This file is autogenerated by pip-compile with python 3.7
# To update, run:
#
#    pip-compile --output-file=requirements-shared.txt requirements-shared.in
#
beautifulsoup4==4.11.1
    # via play-scraper
certifi==2022.12.7
    # via requests
charset-normalizer==3.0.1
    # via requests
idna==3.4
    # via requests
lxml==4.9.2
    # via play-scraper
play-scraper==0.6.0
    # via -r requirements-shared.in
requests[security]==2.28.2
    # via
    #   play-scraper
    #   requests-futures
requests-futures==1.0.0
    # via play-scraper
soupsieve==2.3.2.post1
    # via beautifulsoup4
urllib3==1.26.14
    # via requests

requirements-uses.in

-c requirements-shared.txt

click

Note: command below is what dependabot will effectively be running.

$ pip-compile requirements-uses.in --dry-run --resolver=backtracking
/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/pip/_internal/req/req_install.py:877: PipDeprecationWarning: DEPRECATION: Constraints are only allowed to take the form of a package name and a version specifier. Other forms were originally permitted as an accident of the implementation, but were undocumented. The new implementation of the resolver no longer supports these forms. A possible replacement is replacing the constraint with a requirement. Discussion can be found at https://github.com/pypa/pip/issues/8210
  issue=8210,
Traceback (most recent call last):
  File "/home/discord/dependabot-script/ttt123123/venv/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/piptools/scripts/compile.py", line 580, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/piptools/resolver.py", line 592, in resolve
    compatible_existing_constraints=compatible_existing_constraints,
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/piptools/resolver.py", line 625, in _do_resolve
    check_supported_wheels=not self.options.target_dir,
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 481, in collect_root_requirements
    raise InstallationError(problem)
pip._internal.exceptions.InstallationError: Constraints cannot have extras

The text was updated successfully, but these errors were encountered:

deivid-rodriguez · 2023-01-31T13:01:28Z

Hei @nwalsh1995, thanks for reporting this.

I just tried the latest command you present, but with the --strip-extras and it fails with the same error, so I don't think your patch is going to do anything? In general, when updating a file, Dependabot looks at pip-compile's header lines to learn which flags it needs to use to update that file correctly. But in this case, we're not looking at updating the file, but at reading its information to perform updates, so pip-compile flags probably won't help for that?

My guess is that user here is still on a pip version that worked with extras in constraint files, but Dependabot is using a more recent version of pip that does not work. SO we have several options:

I think we have two options here:

Detect the situation and use an older pip version to perform the update. But I'm not sure how to detect which pip version to use, we'd probably need to keep that information ourselves which we generally prefer not to.
Detect the situation, regenerate the constraints file with --strip-extras, and then perform the update using the new constraints file without extras. That should work but the final PR will include changes unrelated to the upgrade (the upgrade of the constraints file to not use extras), which is generally not desired.

It's tricky, I don't think the solution is as simple as adding the --strip-extras flag to all of our pip-compile commands :(

nwalsh1995 · 2023-01-31T18:16:19Z

@deivid-rodriguez can you try with these files instead? Without the PR I expect it to fail, and with the PR its expected to pass. Also, if possible can you tell me how you're testing the changes so that I can reproduce. Our files have extras stripped from requirements.in and the corresponding .txt files (we use --strip-extras to generate the file), so our files are compliant with the newest version of pip, but when dependabot attempts to resolve versions it doesn't include the flag, so I'm surprised the PR doesn't work.

Here are the files, you might have to play with version numbers to force an update of something out of date. Notice all of these files have been generated with --strip-extras. When dependabot runs it will recreate the requirements-shared.txt with an extra (requests[security]) during version resolution because it doesn't include --strip-extras flag. When it moves on to the next filename_to_compile pip-compile will throw an error because there is an extra in -c file.

requirements-shared.in

play-scraper

requirements-shared.txt

#
# This file is autogenerated by pip-compile with Python 3.7
# by the following command:
#
#    pip-compile --resolver=backtracking --strip-extras requirements-shared.in
#
beautifulsoup4==4.11.1
    # via play-scraper
certifi==2022.12.7
    # via requests
charset-normalizer==3.0.1
    # via requests
idna==3.4
    # via requests
lxml==4.9.2
    # via play-scraper
play-scraper==0.6.0
    # via -r requirements-shared.in
requests==2.28.0
    # via
    #   play-scraper
    #   requests-futures
requests-futures==1.0.0
    # via play-scraper
soupsieve==2.3.2.post1
    # via beautifulsoup4
urllib3==1.26.14
    # via requests

requirements-uses.in

-c requirements-shared.txt

click

requirements-uses.txt

#
# This file is autogenerated by pip-compile with Python 3.7
# by the following command:
#
#    pip-compile --resolver=backtracking --strip-extras requirements-uses.in
#
click==8.1.3
    # via -r requirements-uses.in
importlib-metadata==6.0.0
    # via click
typing-extensions==4.4.0
    # via importlib-metadata
zipp==3.12.0
    # via importlib-metadata

deivid-rodriguez · 2023-01-31T18:36:25Z

Sorry, I misunderstood your issue, I thought your constraint files had extras and Dependabot was not being able to deal with them. You're saying that dependabot is adding the extras itself during them update, and then crashing because of that.

Still I could not reproduce, I'm getting the same result, with or without your change. To test I pushed your manifests to a repo, downgrading click to 8.1.2 and play-scraper to 0.5.0 and I got proper updates for both.

I guess your issue happens when upgrading subdependencies? What's your configuration in .dependabot.yml?

nwalsh1995 · 2023-01-31T23:18:45Z

@deivid-rodriguez can you try from this repo? It's closer to our actual usage and dependabot is failing there with the same logs as in the OP issue.

https://github.com/nwalsh1995/depbot-example

deivid-rodriguez · 2023-02-01T00:54:00Z

I see, you have extras in the constraints.in file not in the txt file.

I still get the same error with your patch though. I think what pip is telling us is that it can't resolve if a constraints file (included through -c) includes extras.

nwalsh1995 · 2023-02-01T01:04:57Z

@deivid-rodriguez Another example which is closer to my initial report, this one doesn't contain extras in any of the files, and they are pulled in by dependabots run: https://github.com/nwalsh1995/depbot-example2

How are you running with the patch? It will be helpful for me to reproduce locally to see what's going on.

nwalsh1995 · 2023-02-01T01:07:56Z

I think what pip is telling us is that it can't resolve if a constraints file (included through -c) includes extras.

Yes, but the files don't contain extras anywhere in the constraints file. Somehow extras are being generated.

In example 1 we have constraints in the .in file but the .txt is generated with --strip-extras. Note that extras in the .in file is okay since nothing declares a -c on the .in, only the .txt.
In example 2 we have no constraints but still get constraints error.

I have no explanation of why the suggested PR doesn't work though, will need more info on how to debug that for me to help.

nwalsh1995 · 2023-02-01T01:19:58Z

Okay, I managed to repro the constraint problem locally on my docker instance (dependabot-script's generic-update-script.rb) for example2. Re-ran with the patch and the run finished successfully.

I cannot repro the exception in depbot-example1 using dependabot-script, so I'm not sure what's different between that and what's running on GH.

deivid-rodriguez · 2023-02-01T01:20:44Z

From your PR branch, run bin/docker-dev-shell python, and then from the devcontainer, bin/dry-run.rb pip nwalsh1995/depbot-example2. I haven't yet run that, so not sure if your patch works against the latest repo you posted.

nwalsh1995 · 2023-02-01T18:03:22Z

For depbot-example its reproducible by running pip-compile instead of the whole dependabot stack:

pip-compile -v --build-isolation --allow-unsafe --strip-extras --resolver backtracking --dry-run -P googleapis-common-protos\[grpc\] requirements.in

Internally pip-compile treats the things inside of the requirements.in file as constraints, and we have extras in there so it fails. I'm not sure whether dependabot is using pip-compile incorrectly, or pip-compile has a bug for this path.

#6551 fixes the case in depot-example2 where someone uses pip-compile and has no extras in their requirements.in and a dependency tries to pull it in.

There will have to be another issue/PR for handling manifest files with constraints which is reproducible in depbot-example.

nwalsh1995 · 2023-02-01T19:19:05Z

FYI I've created jazzband/pip-tools#1806 for pip-compile.

nwalsh1995 added the T: bug 🐞 Something isn't working label Jan 30, 2023

nwalsh1995 mentioned this issue Jan 30, 2023

[python][pip-compile] Add --strip-extras by default when resolving versions #6551

Closed

nwalsh1995 changed the title ~~[python][pip-compile] Dependabot cannot update dependencies if the lockfile is used as constraints in other files.~~ [python][pip-compile] Dependabot cannot update dependencies if the lockfile is used as constraints in other manifest files. Jan 30, 2023

nwalsh1995 changed the title ~~[python][pip-compile] Dependabot cannot update dependencies if the lockfile is used as constraints in other manifest files.~~ [python][pip-compile] Dependabot cannot resolve dependency version if the lockfile is used as constraints in other manifest files. Jan 31, 2023

deivid-rodriguez added the L:python:pip-compile Python packages via pip-compile label Jan 31, 2023

thomdixon mentioned this issue Feb 3, 2023

Remove extras from user-supplied constraints in backtracking resolver jazzband/pip-tools#1808

Merged

4 tasks

jeffwidman mentioned this issue Feb 4, 2023

Extract --extra pip-compile command-line argument from requirements.txt header #6406

Open

1 task

StevenMaude mentioned this issue Feb 16, 2023

Production dependencies not getting updated by Dependabot opensafely-core/ehrql#964

Closed

deivid-rodriguez mentioned this issue Mar 1, 2023

Allow pip-tools 6.12.3 #6757

Merged

jeffwidman closed this as completed in #6757 Mar 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[python][pip-compile] Dependabot cannot resolve dependency version if the lockfile is used as constraints in other manifest files. #6550

[python][pip-compile] Dependabot cannot resolve dependency version if the lockfile is used as constraints in other manifest files. #6550

nwalsh1995 commented Jan 30, 2023 •

edited

deivid-rodriguez commented Jan 31, 2023

nwalsh1995 commented Jan 31, 2023 •

edited

deivid-rodriguez commented Jan 31, 2023

nwalsh1995 commented Jan 31, 2023 •

edited

deivid-rodriguez commented Feb 1, 2023

nwalsh1995 commented Feb 1, 2023

nwalsh1995 commented Feb 1, 2023 •

edited

nwalsh1995 commented Feb 1, 2023

deivid-rodriguez commented Feb 1, 2023

nwalsh1995 commented Feb 1, 2023 •

edited

nwalsh1995 commented Feb 1, 2023

[python][pip-compile] Dependabot cannot resolve dependency version if the lockfile is used as constraints in other manifest files. #6550

[python][pip-compile] Dependabot cannot resolve dependency version if the lockfile is used as constraints in other manifest files. #6550

Comments

nwalsh1995 commented Jan 30, 2023 • edited

Is there an existing issue for this?

Package ecosystem

Package manager version

Language version

Manifest location and content before the Dependabot update

dependabot.yml content

Updated dependency

What you expected to see, versus what you actually saw

Suggestion

Alternative to suggestion

Native package manager behavior

Images of the diff or a link to the PR, issue, or logs

Smallest manifest that reproduces the issue

deivid-rodriguez commented Jan 31, 2023

nwalsh1995 commented Jan 31, 2023 • edited

deivid-rodriguez commented Jan 31, 2023

nwalsh1995 commented Jan 31, 2023 • edited

deivid-rodriguez commented Feb 1, 2023

nwalsh1995 commented Feb 1, 2023

nwalsh1995 commented Feb 1, 2023 • edited

nwalsh1995 commented Feb 1, 2023

deivid-rodriguez commented Feb 1, 2023

nwalsh1995 commented Feb 1, 2023 • edited

nwalsh1995 commented Feb 1, 2023

nwalsh1995 commented Jan 30, 2023 •

edited

nwalsh1995 commented Jan 31, 2023 •

edited

nwalsh1995 commented Jan 31, 2023 •

edited

nwalsh1995 commented Feb 1, 2023 •

edited

nwalsh1995 commented Feb 1, 2023 •

edited