pip-compile Does Not Work for vsphere-automation-sdk-python setup.py on GitHub #1054

nguyenlieunhatvy · 2020-01-31T18:56:16Z

pip-compile does not work if I have this in my requirements.in/base.txt

# VMware vSphere Automation SDK for Python
# Note the use of -e to make this editable and preserve the git URL.
-e git+https://github.com/vmware/vsphere-automation-sdk-python.git@v6.9.1#egg=vSphere-Automation-SDK

Link to vsphere-automation-sdk-python GitHub

Environment Versions

OS Type: macOS Catalina 10.15.3
Python version: Python 3.7.2
pip version: pip 20.0.2
pip-tools version: pip-compile, version 4.4.1

Steps to replicate

Create an empty virtual environment: python3 -m venv venv.
Activate the virtual environment.
Run pip install -U pip.
Create a requirements.in/base.txt file.
Add the below lines to the requirements.in/base.txt:

# VMware vSphere Automation SDK for Python
# Note the use of -e to make this editable and preserve the git URL.
-e git+https://github.com/vmware/vsphere-automation-sdk-python.git@v6.9.1#egg=vSphere-Automation-SDK

Run pip-compile requirements.in/base.txt

Expected result

Should successfully compile the requirements.in/base.txt

Actual result

Error:

Could not find a version that matches vapi-common-client>=2.12.0 (from vSphere-Automation-SDK->-r requirements.in/base.txt (line 46))
No versions found
Was https://pypi.org/simple reachable?

The text was updated successfully, but these errors were encountered:

atugushev · 2020-02-02T16:17:01Z

Hello @nguyenlieunhatvy,

Thanks for the detailed information and reproducible steps. The issue is caused when we combine install requirements where we lose some info on dropped requirements, see the NOTE message:

pip-tools/piptools/resolver.py

Lines 61 to 68 in d405bf9

    
           for ireq in source_ireqs[1:]: 
        
               # NOTE we may be losing some info on dropped reqs here 
        
               combined_ireq.req.specifier &= ireq.req.specifier 
        
               combined_ireq.constraint &= ireq.constraint 
        
               # Return a sorted, de-duped tuple of extras 
        
               combined_ireq.extras = tuple( 
        
                   sorted(set(tuple(combined_ireq.extras) + tuple(ireq.extras))) 
        
               )

I've tried to trace Could not find a version that matches vapi-common-client>=2.12.0, see traceback:

  /usr/local/bin/pip-compile(8)<module>()
-> sys.exit(cli())
  /usr/local/lib/python3.7/site-packages/click/core.py(764)__call__()
-> return self.main(*args, **kwargs)
  /usr/local/lib/python3.7/site-packages/click/core.py(717)main()
-> rv = self.invoke(ctx)
  /usr/local/lib/python3.7/site-packages/click/core.py(956)invoke()
-> return ctx.invoke(self.callback, **ctx.params)
  /usr/local/lib/python3.7/site-packages/click/core.py(555)invoke()
-> return callback(*args, **kwargs)
  /usr/local/lib/python3.7/site-packages/click/decorators.py(17)new_func()
-> return f(get_current_context(), *args, **kwargs)
  /usr/local/lib/python3.7/site-packages/piptools/scripts/compile.py(385)cli()
-> results = resolver.resolve(max_rounds=max_rounds)
  /usr/local/lib/python3.7/site-packages/piptools/resolver.py(171)resolve()
-> has_changed, best_matches = self._resolve_one_round()
  /usr/local/lib/python3.7/site-packages/piptools/resolver.py(269)_resolve_one_round()
-> theirs = set(self._group_constraints(sorted(their_constraints, key=str)))
  /usr/local/lib/python3.7/site-packages/piptools/resolver.py(236)_group_constraints()
-> yield combine_install_requirements(ireqs)
> /usr/local/lib/python3.7/site-packages/piptools/resolver.py(60)combine_install_requirements()
-> combined_ireq = copy.deepcopy(source_ireqs[0])

Where source_ireqs is:

[<InstallRequirement object: vapi-common-client (from vmc-client-bindings@ file://localhost//src/vsphere-automation-sdk/lib/vmc-client-bindings/vmc_client_bindings-1.11.0-py2.py3-none-any.whl->vSphere-Automation-SDK->-r requirements.in (line 1)) editable=False>,
 <InstallRequirement object: vapi-common-client (from vmc-draas-client-bindings@ file://localhost//src/vsphere-automation-sdk/lib/vmc-draas-client-bindings/vmc_draas_client_bindings-1.0.0-py2.py3-none-any.whl->vSphere-Automation-SDK->-r requirements.in (line 1)) editable=False>,
 <InstallRequirement object: vapi-common-client>=2.12.0 (from nsx-policy-python-sdk@ file://localhost//src/vsphere-automation-sdk/lib/nsx-policy-python-sdk/nsx_policy_python_sdk-2.3.0.0.3.13851140-py2.py3-none-any.whl->vSphere-Automation-SDK->-r requirements.in (line 1)) editable=False>,
 <InstallRequirement object: vapi-common-client>=2.12.0 (from nsx-python-sdk@ file://localhost//src/vsphere-automation-sdk/lib/nsx-python-sdk/nsx_python_sdk-2.3.0.0.3.13851140-py2.py3-none-any.whl->vSphere-Automation-SDK->-r requirements.in (line 1)) editable=False>,
 <InstallRequirement object: vapi-common-client>=2.12.0 (from nsx-vmc-aws-integration-python-sdk@ file://localhost//src/vsphere-automation-sdk/lib/nsx-vmc-aws-integration-python-sdk/nsx_vmc_aws_integration_python_sdk-2.3.0.0.3.13851140-py2.py3-none-any.whl->vSphere-Automation-SDK->-r requirements.in (line 1)) editable=False>,
 <InstallRequirement object: vapi-common-client>=2.12.0 (from nsx-vmc-policy-python-sdk@ file://localhost//src/vsphere-automation-sdk/lib/nsx-vmc-policy-python-sdk/nsx_vmc_policy_python_sdk-2.3.0.0.3.13851140-py2.py3-none-any.whl->vSphere-Automation-SDK->-r requirements.in (line 1)) editable=False>,
 <InstallRequirement object: vapi-common-client@ file://localhost//src/vsphere-automation-sdk/lib/vapi-common-client/vapi_common_client-2.12.0-py2.py3-none-any.whl from file://localhost//src/vsphere-automation-sdk/lib/vapi-common-client/vapi_common_client-2.12.0-py2.py3-none-any.whl (from vSphere-Automation-SDK->-r requirements.in (line 1)) editable=False>]

Note, we deep copy the source_ireqs[0] and collect there specifier and constraint info from the other requirements, but lose the other data which is URL in this particular case. The URL info is in source_ireq[-1].

Changing

pip-tools/piptools/resolver.py

Line 60 in d405bf9

combined_ireq = copy.deepcopy(source_ireqs[0])

to

combined_ireq = copy.deepcopy(source_ireqs[-1])`

fixes the issue, but it is not the right way to go, we need to figure out how to properly merge the data.

microcat49 · 2021-04-11T19:48:56Z

Hey @atugushev,

Would you mind if I took this issue? I figured out that the information being lost here is the sub libraries come from repos lib folder. Once it loses that information it tries to search pypi for these libraries but can't find them because they're not on there. I feel that I have a good solution for this.

atugushev · 2021-04-12T08:03:26Z

@microcat49 sure 👍🏻

microcat49 · 2021-04-17T16:48:25Z

Pull request #1385

AndydeCleyre · 2021-10-13T00:51:33Z

I tried out your logic fix from #1385, and it does seem to fix the issue.

But the output is full of paths to local temporary checkout of git dependencies.

I guess that is a separate issue, but am wondering if anyone can shine some light on this result vs expectations, whatever they may be:

Output

#
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
#    pip-compile reqsin/base.in
#
-e git+https://github.com/vmware/vsphere-automation-sdk-python.git@v6.9.1#egg=vSphere-Automation-SDK
    # via -r reqsin/base.in
certifi==2021.10.8
    # via requests
cffi==1.14.6
    # via cryptography
charset-normalizer==2.0.7
    # via requests
cryptography==35.0.0
    # via pyopenssl
idna==3.3
    # via requests
lxml==4.6.3
    # via vsphere-automation-sdk
nsx-policy-python-sdk @ file://localhost//home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/src/vsphere-automation-sdk/lib/nsx-policy-python-sdk/nsx_policy_python_sdk-2.3.0.0.3.13851140-py2.py3-none-any.whl
    # via vsphere-automation-sdk
nsx-python-sdk @ file://localhost//home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/src/vsphere-automation-sdk/lib/nsx-python-sdk/nsx_python_sdk-2.3.0.0.3.13851140-py2.py3-none-any.whl
    # via vsphere-automation-sdk
nsx-vmc-aws-integration-python-sdk @ file://localhost//home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/src/vsphere-automation-sdk/lib/nsx-vmc-aws-integration-python-sdk/nsx_vmc_aws_integration_python_sdk-2.3.0.0.3.13851140-py2.py3-none-any.whl
    # via vsphere-automation-sdk
nsx-vmc-policy-python-sdk @ file://localhost//home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/src/vsphere-automation-sdk/lib/nsx-vmc-policy-python-sdk/nsx_vmc_policy_python_sdk-2.3.0.0.3.13851140-py2.py3-none-any.whl
    # via vsphere-automation-sdk
pycparser==2.20
    # via cffi
pyopenssl==21.0.0
    # via vapi-runtime
pyvmomi==7.0.2
    # via vsphere-automation-sdk
requests==2.26.0
    # via
    #   pyvmomi
    #   vapi-runtime
six==1.16.0
    # via
    #   pyopenssl
    #   pyvmomi
    #   vapi-runtime
suds-jurko==0.6
    # via vsphere-automation-sdk
urllib3==1.26.7
    # via requests
vapi-client-bindings @ file://localhost//home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/src/vsphere-automation-sdk/lib/vapi-client-bindings/vapi_client_bindings-3.1.0-py2.py3-none-any.whl
    # via vsphere-automation-sdk
vapi-common-client @ file://localhost//home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/src/vsphere-automation-sdk/lib/vapi-common-client/vapi_common_client-2.12.0-py2.py3-none-any.whl
    # via
    #   nsx-policy-python-sdk
    #   nsx-python-sdk
    #   nsx-vmc-aws-integration-python-sdk
    #   nsx-vmc-policy-python-sdk
    #   vmc-client-bindings
    #   vmc-draas-client-bindings
    #   vsphere-automation-sdk
vapi-runtime @ file://localhost//home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/src/vsphere-automation-sdk/lib/vapi-runtime/vapi_runtime-2.12.0-py2.py3-none-any.whl
    # via
    #   nsx-policy-python-sdk
    #   nsx-python-sdk
    #   nsx-vmc-aws-integration-python-sdk
    #   nsx-vmc-policy-python-sdk
    #   vapi-client-bindings
    #   vapi-common-client
    #   vmc-client-bindings
    #   vmc-draas-client-bindings
    #   vsphere-automation-sdk
vmc-client-bindings @ file://localhost//home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/src/vsphere-automation-sdk/lib/vmc-client-bindings/vmc_client_bindings-1.11.0-py2.py3-none-any.whl
    # via vsphere-automation-sdk
vmc-draas-client-bindings @ file://localhost//home/andy/.local/share/venvs/1b2b5c9cc79c4656dd687f3b665b224c/venv/src/vsphere-automation-sdk/lib/vmc-draas-client-bindings/vmc_draas_client_bindings-1.0.0-py2.py3-none-any.whl
    # via vsphere-automation-sdk

# The following packages are considered to be unsafe in a requirements file:
# setuptools

Fixes jazzband#1054 Co-authored-by: Andrew <harta5039@gmail.com>

atugushev added bug Something is not working resolver Related to dependency resolver labels Feb 1, 2020

atugushev added the help wanted Request help from the community label Feb 12, 2020

atugushev assigned microcat49 Apr 12, 2021

AndydeCleyre mentioned this issue Sep 20, 2021

Fixing issue with losing data about local files #1385

Closed

3 tasks

AndydeCleyre mentioned this issue Oct 12, 2021

Second level dependencies missing on for local file packages on >= 6.3.1 #1505

Closed

AndydeCleyre mentioned this issue Oct 13, 2021

Prevent loss of deep dependencies of local requirements #1506

Closed

6 tasks

AndydeCleyre added a commit to AndydeCleyre/pip-tools that referenced this issue Oct 13, 2021

Integrate microcat49's attribute copying fix for lost local req info

81129a1

Fixes jazzband#1054 Co-authored-by: Andrew <harta5039@gmail.com>

richafrank mentioned this issue Oct 28, 2021

Instantiate a new accumulator InstallRequirement for combine_install_requirements output #1519

Merged

4 tasks

AndydeCleyre added a commit to richafrank/pip-tools that referenced this issue Feb 8, 2022

Integrate microcat49's attribute copying fix for lost local req info

c9b9db4

Fixes jazzband#1054 Co-authored-by: Andrew <harta5039@gmail.com>

atugushev closed this as completed in 41b4d37 May 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pip-compile Does Not Work for vsphere-automation-sdk-python setup.py on GitHub #1054

pip-compile Does Not Work for vsphere-automation-sdk-python setup.py on GitHub #1054

nguyenlieunhatvy commented Jan 31, 2020

atugushev commented Feb 2, 2020 •

edited

microcat49 commented Apr 11, 2021 •

edited

atugushev commented Apr 12, 2021

microcat49 commented Apr 17, 2021

AndydeCleyre commented Oct 13, 2021

pip-compile Does Not Work for vsphere-automation-sdk-python setup.py on GitHub #1054

pip-compile Does Not Work for vsphere-automation-sdk-python setup.py on GitHub #1054

Comments

nguyenlieunhatvy commented Jan 31, 2020

Environment Versions

Steps to replicate

Expected result

Actual result

atugushev commented Feb 2, 2020 • edited

microcat49 commented Apr 11, 2021 • edited

atugushev commented Apr 12, 2021

microcat49 commented Apr 17, 2021

AndydeCleyre commented Oct 13, 2021

atugushev commented Feb 2, 2020 •

edited

microcat49 commented Apr 11, 2021 •

edited