`--trusted-host` not passed to `pip` when installing from a lockfile

[orf]

Issue description

On the latest pipenv release, running pipenv install does not pass --trusted-host to pip.

Running pipenv install xyz does.

Expected result

--trusted-host is passed to pip.

Actual result

> pipenv install --verbose                                
Installing dependencies from Pipfile…
Installing 'aiohttp'▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/7 — 00:00:00
$ ['/Users/tom/.local/share/virtualenvs/tmp.YkTM0kln-68P7VPaZ/bin/pip', 'install', '--verbose', '--upgrade', '--no-deps', '-r', '/var/folders/9w/twrv54nd0v19gsbjhjs3mlhc0000gn/T/pipenv-wgs4narf-requirements/pipenv-h6s3thx1-requirement.txt', '-i', 'http://x.x.co.uk:8080', '--require-hashes']

And:

> pipenv install --verbose pytest
⠋Installing 'pytest'
$ ['/Users/tom/.local/share/virtualenvs/tmp.YkTM0kln-68P7VPaZ/bin/pip', 'install', '--verbose', '--upgrade', 'pytest', '-i', 'http://x.x.co.uk:8080', '--trusted-host', 'x.x.co.uk']

Steps to replicate

Have a Pipfile like this:

[[source]]
url = "http://INTERNAL_PYPI:8080"
verify_ssl = false
name = "xyz"

[packages]
aiohttp = "*"

Notice how the argument is not passed to pip when calling pipenv install.

[orf]

I believe db5a862 by @techalchemy (or some change around that, 7429881 / 683df2f) introduced this. This section:

pipenv/pipenv/core.py

Lines 749 to 761 in ddb40ec

	# Use a specific index, if specified.
	indexes, trusted_hosts, dep = parse_indexes(dep)
	index = None
	extra_indexes = []
	if indexes:
	index = indexes[0]
	if len(indexes) > 0:
	extra_indexes = indexes[1:]
	dep = Requirement.from_line(" ".join(dep))
	if index:
	_index = None
	try:
	_index = project.find_source(index).get("name")

Ends up calling pip_install with index="some index url", whereas pipenv install xyz calls it with index=None. pip_install will call prepare_pip_source_args with just [{'url': 'some index url'}] if index=string, whereas if index=none the correct [{url: ..., verify_ssl: ..., }] object is passed.

It also seems to correctly parse the trusted_hosts in parse_indexes, but does nothing with it.

Currently pipenv is broken for anyone using verify_ssl: false!

[techalchemy]

Thanks for catching this. Is this before the bug fix release goes out! Note that there was another issue on the topic already so it’s helpful to find the actual cause. I think you’re spot on, we just need to actually do something with the correctly parsed trusted host. There is some redundant code there as well, but I’m not confident enough to remove it just now

We have had an endless stream of issues around sources so I’m anxious to have this sorted.

[orf]

We can sort this in a slightly hacky way by just passing trusted_hosts to pip_install, and then from pip_install to prepare_pip_source_args where it does the right thing ™️

I don't think this is the best long term solution, but if it sounds OK I could try and prepare a PR?

[royrusso]

Is there a way to brew install a previous version. Seems like the current version break on trusted hosts, so until this PR is merged... ?

[KrobotP]

Hello. It is seems to me that this bug is back again. I am trying ty install a package from my private repo:

pipenv install --verbose my_package>=4.0.0 --index="http://private.repo.org/x/y/z/simple" --trusted-host private.repo.org

but got this output (trusted-host is missing and command fails):

⠋ Installing...Writing supplied requirement line to temporary file: 'my_package'
⠇ Installing my_package...$ /home/current/me/.local/share/virtualenvs/my_project-acQDxuau/bin/python -m pip install --verbose --upgrade --exists-action=i -r /tmp/pipenv-0nipwnut-requirements/pipenv-6zqlbkle-requirement.txt -i http://private.repo.org/x/y/z/simple --extra-index-url https://pypi.org/simple --extra-index-url https://pypi.org/simple
Error:  An error occurred while installing my_package!
Error text: Using pip 22.1.2 from /home/current/me/.local/share/virtualenvs/my_project-acQDxuau/lib/python3.8/site-packages/pip (python 3.8)
Looking in indexes: http://private.repo.org/x/y/z/simple, https://pypi.org/simple, https://pypi.org/simple

WARNING: The repository located at private.repo.org is not a trusted or secure host and is being ignored. If this repository is available via HTTPS we recommend you use HTTPS instead, otherwise you may silence this warning and allow it anyway with '--trusted-host private.repo.org'.
ERROR: Could not find a version that satisfies the requirement my_package (from versions: none)
ERROR: No matching distribution found for my_package

Using pip 22.1.2 from /home/current/me/.local/share/virtualenvs/my_project-acQDxuau/lib/python3.8/site-packages/pip (python 3.8)
Looking in indexes: http://private.repo.org/x/y/z/simple, https://pypi.org/simple, https://pypi.org/simple
✘ Installation Failed

Installation via pip directly succeeds. It is frustrating that solution provided by the error output is the thing which is not working.

Edit: Using a current version of pipenv:

$ pipenv --version
pipenv, version 2022.6.7

[matteius]

@KrobotP It is plausible that something broke when the new version of pip was vendor'd in. This issue report is ancient, can you do some more digging and open a new report? The last version to have the prior pip was 2022.3.24 so perhaps start by checking that and narrowing down when it could have broke. I wasn't part of the original discussion here to know what happened or what the fix originally was.

[matteius]

Ah actually @KrobotP I think you are running into index restricted packages security enhancement. There are some other closed issue reports that talk more in depth about how to handle the case of multiple indexes. EDIT: Like this one: #5053
Well this maybe is a case where index restricted packages broke it, or maybe the new version of pip, or a combo of the two. I think we do actually need to determine when it broke and open a new issue report about it.

[KrobotP]

@matteius thank you for fast reaction. I have inspected the issue and tryied 2022.3.24 and here are some of my findings:

• It is true that 'my_mackage' depends on another package from same private repo.
• Mentioned problem still persists with 2022.3.24 (I still see as the main issue that "--trusted-host" argument is not passed to the pip call underneath)
• When I try to add our private repo as another source to fresh Pipfile:

[[source]]
url = "http://private.repo.org/x/y/z/simple"
verify_ssl = false
name = "private-pypi"

the package is then installed successfuly (also with dependencies), but pipenv fails with another error:

Installing my_package...
Installing package: my_package
Writing supplied requirement line to temporary file: 'my_package'
⠸ Installing my_package...$ <my_home>/share/virtualenvs/my_project-acQDxuau/bin/python -m pip install --verbose --upgrade --exists-action=i -r /tmp/pipenv-bk_zjaa7-requirements/pipenv-t5vupgq0-requirement.txt -i http://private.repo.org/x/y/z/simple --trusted-host private.repo.org --extra-index-url https://pypi.org/simple --extra-index-url https://pypi.org/simple
✔ Installation Succeeded
Installing --trusted-host...
Traceback (most recent call last):
  File "<my_home>/lib/python3.6/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 102, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "<my_home>/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1654, in parseString
    raise exc
  File "<my_home>/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1644, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "<my_home>/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "<my_home>/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 3417, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "<my_home>/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "<my_home>/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 3739, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "<my_home>/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "<my_home>/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 3400, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "<my_home>/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 1406, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "<my_home>/lib/python3.6/site-packages/pkg_resources/_vendor/pyparsing.py", line 2711, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pkg_resources._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<my_home>/lib/python3.6/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 966, in _parse_name_from_line
    self._requirement = init_requirement(self.line)
  File "<my_home>/lib/python3.6/site-packages/pipenv/vendor/requirementslib/models/utils.py", line 197, in init_requirement
    req = Requirement.parse(name)
  File "<my_home>/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3154, in parse
    req, = parse_requirements(s)
  File "<my_home>/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3099, in parse_requirements
    yield Requirement(line)
  File "<my_home>/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3109, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "<my_home>/lib/python3.6/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 105, in __init__
    f'Parse error at "{ requirement_string[e.loc : e.loc + 8]!r}": {e.msg}'
pkg_resources.extern.packaging.requirements.InvalidRequirement: Parse error at "'--truste'": Expected W:(abcd...)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "<my_home>/lib/python3.6/site-packages/pipenv/__main__.py", line 5, in <module>
    cli()
  File "<my_home>/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "<my_home>/lib/python3.6/site-packages/pipenv/cli/options.py", line 54, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
  File "<my_home>/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "<my_home>/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "<my_home>/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "<my_home>/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "<my_home>/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "<my_home>/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "<my_home>/lib/python3.6/site-packages/pipenv/cli/command.py", line 213, in install
    site_packages=state.site_packages
  File "<my_home>/lib/python3.6/site-packages/pipenv/core.py", line 2028, in do_install
    pkg_requirement = Requirement.from_line(pkg_line)
  File "<my_home>/lib/python3.6/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2674, in from_line
    parsed_line = Line(line)
  File "<my_home>/lib/python3.6/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 171, in __init__
    self.parse()
  File "<my_home>/lib/python3.6/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 1304, in parse
    self.parse_name()
  File "<my_home>/lib/python3.6/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 1027, in parse_name
    name = self._parse_name_from_line()
  File "<my_home>/lib/python3.6/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 969, in _parse_name_from_line
    "Failed parsing requirement from {0!r}".format(self.line)
pipenv.vendor.requirementslib.exceptions.RequirementError: Failed parsing requirement from '--trusted-host'

[KrobotP]

One more update: I read the https://pipenv.pypa.io/en/latest/advanced/#specifying-package-indexes (refered from the issue above) and after modyfying 'my_package' dependency in Pipfile with 'index' option:

my_package= {version=">=4.0.0", index="private_repo"}

However, (kind of 'fun fact') this also worked, when I mystakenly run pipenv install with index="pytorch" set for my dependency (also tried with index="and it still works"). It seems that seting non-default index to something activates all non-default sources..

[KrobotP]

I have tried this also on the actual version (2022.6.7). While setting 'index' option for the package does not work, it works to leave only my private repo as a single source within the Pipfile. What confuses me is that I dont have other dependecies and subdependencies mirrored in my private repo. As I understand this https://pipenv.pypa.io/en/latest/advanced/#specifying-package-indexes it should not work without dependencies from public pypi mirrored to my private one but it does.

[matteius]

@KrobotP I think it wasn't until the 2022.4.8 release that I had really gotten index restricted packages fully taking effect. The problem is what you described:

It is true that 'my_mackage' depends on another package from same private repo.

Because of that you either have to make your default index be the private index, or you have to specify also the other package in the private repo in the Pipfile as well for the purpose of specifying the index. For example:

my_package= {version=">=4.0.0", index="private_repo"}
my_other_package= {version="*", index="private_repo"}

This will marry up the version requirements that the resolver determines with the requirement that my_other_package also be pulled from private_repo. There are some other requests in the backlog about how we can make this something that is specified in the setup.cfg or some other way of improving things.

What confuses me is that I dont have other dependencies and subdependencies mirrored in my private repo. As I understand this https://pipenv.pypa.io/en/latest/advanced/#specifying-package-indexes it should not work without dependencies from public pypi mirrored to my private one but it does.

Does your Pipfile still have the pypi source in it? If so, is it the default or secondary source? I am not sure but if you could open a new ticket about this with an example case that we can explore in more detail that would be helpful. It might just be that you have pypi defined and resolution is allowing it to be found at pypi ... not sure without a specific example. Preferably a new ticket so we can track it and not ping the old thread participants of this closed issue

However, (kind of 'fun fact') this also worked, when I mistakenly run pipenv install with index="pytorch" set for my dependency (also tried with index="and it still works"). It seems that seting non-default index to something activates all non-default sources..

Could you perhaps file a separate new ticket about this with some details of how to reproduce? That seems like something we should look into.