Using: keyrings.google-artifactregistry-auth pipenv update -d fails hard.

[klausmyrseth]

Be sure to check the existing issues (both open and closed!), and make sure you are running the latest version of Pipenv.

Check the diagnose documentation for common issues before posting! We may close your issue if it is very similar to one of them. Please be considerate, or be on your way.

Make sure to mention your debugging experience if the documented solution failed.

Issue description

Google Artifact Repositories use keyrings with their own addon to make git and twine work perfectly, but when using pipenv it fails hard to a point where we cant use pipenv.

Note I also tried with 20.11.15 with same result

Expected result

pipenv update -d should update dependencies and make an update Pipfile.lock

Actual result

(applied_mapper) klaus@new-server ~/src/applied_mapper master> pipenv update -d -v
Running $ pipenv lock then $ pipenv sync.
Locking [dev-packages] dependencies…
Building requirements...
Resolving dependencies...
⠧ Locking...
ROUND 1
✘ Locking Failed!
Current constraints:
aadomain (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 10))
applied-core (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 14))
applied-domain (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 9))
geojson (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 13))
keyring (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 7))
keyrings.google-artifactregistry-auth (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 15))
pbr (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 5))
pylint (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 4))
pytest (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 8))
pytest-flake8 (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 6))
python-dateutil (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 12))
strenum (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 11))

Finding the best candidates:
found candidate aadomain==1.4.42 (constraint was )
Traceback (most recent call last):
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py", line 808, in resolve
results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 180, in resolve
has_changed, best_matches = self._resolve_one_round()
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 260, in _resolve_one_round
best_matches = {self.get_best_match(ireq) for ireq in constraints}
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 260, in
best_matches = {self.get_best_match(ireq) for ireq in constraints}
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 319, in get_best_match
best_match = self.repository.find_best_match(
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 202, in find_best_match
raise NoCandidateFound(ireq, all_candidates, self.finder)
pipenv.patched.piptools.exceptions.NoCandidateFound: Could not find a version that matches applied-core (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 14))
No versions found
Were https://europe-north1-pypi.pkg.dev/new-server/pypi/simple or https://legacy-pypi.appliedautonomy.no:8443/simple or https://pypi.org/simple reachable?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in
main()
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
_main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
results, resolver = resolve(
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
return resolve_deps(
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1378, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1093, in actually_resolve_deps
resolver.resolve()
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py", line 818, in resolve
raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: ERROR: Could not find a version that matches applied-core (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 14))
No versions found
Were https://europe-north1-pypi.pkg.dev/new-server/pypi/simple or https://legacy-pypi.appliedautonomy.no:8443/simple or https://pypi.org/simple reachable?
ROUND 1
Current constraints:
aadomain (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 10))
applied-core (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 14))
applied-domain (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 9))
geojson (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 13))
keyring (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 7))
keyrings.google-artifactregistry-auth (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 15))
pbr (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 5))
pylint (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 4))
pytest (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 8))
pytest-flake8 (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 6))
python-dateutil (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 12))
strenum (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 11))

Finding the best candidates:
found candidate aadomain==1.4.42 (constraint was )
Traceback (most recent call last):
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py", line 808, in resolve
results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 180, in resolve
has_changed, best_matches = self._resolve_one_round()
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 260, in _resolve_one_round
best_matches = {self.get_best_match(ireq) for ireq in constraints}
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 260, in
best_matches = {self.get_best_match(ireq) for ireq in constraints}
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 319, in get_best_match
best_match = self.repository.find_best_match(
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 202, in find_best_match
raise NoCandidateFound(ireq, all_candidates, self.finder)
pipenv.patched.piptools.exceptions.NoCandidateFound: Could not find a version that matches applied-core (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 14))
No versions found
Were https://europe-north1-pypi.pkg.dev/new-server/pypi/simple or https://legacy-pypi.appliedautonomy.no:8443/simple or https://pypi.org/simple reachable?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in
main()
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
_main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
results, resolver = resolve(
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
return resolve_deps(
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1378, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1093, in actually_resolve_deps
resolver.resolve()
File "/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py", line 818, in resolve
raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: ERROR: Could not find a version that matches applied-core (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 14))
No versions found
Were https://europe-north1-pypi.pkg.dev/new-server/pypi/simple or https://legacy-pypi.appliedautonomy.no:8443/simple or https://pypi.org/simple reachable?

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

Its hard to replicate as you need alpha access to artifact repo pypi, but using a Pipfile like this:
[[source]]
name = "applied-backend"
url = "https://europe-north1-pypi.pkg.dev/new-server/pypi/simple"
verify_ssl = true

[[source]]
name = "applied"
url = "https://legacy-pypi.appliedautonomy.no:8443/simple"
verify_ssl = true

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
keyring = ""
"keyrings.google-artifactregistry-auth" = ""
pytest = ""
pytest-flake8 = ""
pbr = ""
pylint = ""
StrEnum = ""
python-dateutil = ""
geojson = ""
aadomain = ""
applied-core = ""
applied-domain = ""

[packages]

[requires]
python_version = "3.9"

then running pipenv update -d you will get the hard fail listed above. I do believe its connected to the keyring as it works as a charm in pip without any drama.

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

$ pipenv --support

Pipenv version: '2020.8.13'

Pipenv location: '/home/klaus/.local/lib/python3.8/site-packages/pipenv'

Python location: '/usr/bin/python3'

Python installations found:

• 3.9.0: /home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4/bin/python3.9
• 3.9.0: /home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4/bin/python3
• 3.9.0: /home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4/bin/python3.9
• 3.9.0: /home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4/bin/python3
• 3.9.0: /usr/bin/python3.9
• 3.9.0: /bin/python3.9
• 3.8.5: /usr/bin/python3.8
• 3.8.5: /usr/bin/python3
• 3.8.5: /bin/python3.8
• 3.8.5: /bin/python3
• 2.7.18: /usr/bin/python2
• 2.7.18: /usr/bin/python2.7
• 2.7.18: /bin/python2
• 2.7.18: /bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.4.0-73-generic',
 'platform_system': 'Linux',
 'platform_version': '#82-Ubuntu SMP Wed Apr 14 17:39:42 UTC 2021',
 'python_full_version': '3.8.5',
 'python_version': '3.8',
 'sys_platform': 'linux'}

System environment variables:

• GJS_DEBUG_TOPICS
• SSH_AUTH_SOCK
• SESSION_MANAGER
• GNOME_TERMINAL_SCREEN
• SSH_AGENT_PID
• XDG_CURRENT_DESKTOP
• LANG
• LC_IDENTIFICATION
• DEFAULTS_PATH
• XDG_SESSION_CLASS
• COLORTERM
• LIBVIRT_DEFAULT_URI
• GPG_AGENT_INFO
• DESKTOP_SESSION
• GJS_DEBUG_OUTPUT
• XDG_MENU_PREFIX
• USER
• QT_IM_MODULE
• LC_MEASUREMENT
• VTE_VERSION
• DBUS_SESSION_BUS_ADDRESS
• PWD
• LC_NUMERIC
• GTK_MODULES
• _
• WINDOWPATH
• XDG_SESSION_DESKTOP
• JOURNAL_STREAM
• QT_ACCESSIBILITY
• HOME
• GNOME_DESKTOP_SESSION_ID
• MANAGERPID
• LC_TIME
• XDG_DATA_DIRS
• GNOME_TERMINAL_SERVICE
• LC_PAPER
• LOGNAME
• MANDATORY_PATH
• XDG_RUNTIME_DIR
• XDG_CONFIG_DIRS
• XDG_SESSION_TYPE
• XMODIFIERS
• PATH
• LC_TELEPHONE
• LC_MONETARY
• SHELL
• GNOME_SHELL_SESSION_MODE
• USERNAME
• INVOCATION_ID
• SHLVL
• XAUTHORITY
• LC_NAME
• IM_CONFIG_PHASE
• TERM
• LC_ADDRESS
• DISPLAY
• GDMSESSION
• ZSH
• PAGER
• LESS
• LSCOLORS
• LS_COLORS
• CLOUDSDK_HOME
• APPLIED_SCRIPT_HOME
• GOOGLE_APPLICATION_CREDENTIALS
• GOOGLE_ACCOUNT_SERVICE_FILE
• NVM_DIR
• NVM_CD_FLAGS
• NVM_BIN
• NVM_INC
• pipfile_dir
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH
• PIPENV_ACTIVE
• VIRTUAL_ENV
• PS1
• PIP_SHIMS_BASE_MODULE
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

• PIPENV_ACTIVE: 1

Debug–specific environment variables:

• PATH: /home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4/bin:/home/klaus/bin:/opt/ghc/bin:/home/klaus/.nvm/versions/node/v15.1.0/bin:/home/klaus/bin:/opt/ghc/bin:/home/klaus/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
• SHELL: /usr/bin/zsh
• LANG: en_US.UTF-8
• PWD: /home/klaus/src/applied_mapper
• VIRTUAL_ENV: /home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4

---

Contents of Pipfile ('/home/klaus/src/applied_mapper/Pipfile'):

[[source]]
name = "applied-backend"
url = "https://europe-north1-pypi.pkg.dev/new-server/pypi/simple"
verify_ssl = true

[[source]]
name = "applied"
url = "https://$legacy-pypi.appliedautonomy.no:8443/simple"
verify_ssl = true

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
keyring = "*"
"keyrings.google-artifactregistry-auth" = "*"
pytest = "*"
pytest-flake8 = "*"
pbr = "*"
pylint = "*"
StrEnum = "*"
python-dateutil = "*"
geojson = "*"
aadomain = "*"
applied-core = "*"
applied-domain = "*"

[packages]

[requires]
python_version = "3.9"

[romhml]

I experienced the same issue and noticed that pipenv is actually able to download packages from Google Artifact Registry, and only failing when updating the lockfile. You can work around this by editing the lockfile file manually, or by doing something like this:

$ pip download --extra-index-url <URL_TO_ARTIFACT_REG> <YOUR_PACKAGE>
$ pipenv install *.whl

This will update the lockfile, but also modify your Pipfile to point towards local archives. Undo the changes to the Pipfile and it will work as expected.

I'm trying to figure out how to properly fix this, I'll reach out if I find anything.

[nibuddg]

I'm experiencing a similar issue and am also trying to find a solution. This workaround of downloading packages first and then installing them from local works, but is not so ideal

[dhendry]

Its my dream to have pipenv+artifact-registry work nicely together (leveraging keychain for secure credential management).

Therin lies a world of rainbows and unicorns.

+1 on fixing this problem. I unfortunately lack the time to dig into this but if somebody could gat a fix in they would have my eternal gratitude.

[Darsstar]

This also fails for Azure DevOps Atifact Feeds with the artifacts-keyring backend, and I'm pretty sure everything that recommends authentication via a keyring backend when using the 2020 resolver. (I have not tried the legacy resolver yet.)
This is because pipenv.utils.resolver.Resolver's pip_options property sets no_input to true, which causes pip to disable keyring lookups (see self.prompting in auth.py in notpip's network module) since it is theoretically allowed for keyring backends to prompt the user.

I'm going to call it a night for today, but I wanted to share what I discovered so far.

[Darsstar]

I did not get as much sleep as I hoped, but at least I came up with a possible long term fix while lying awake: add a --force-keyring flag to pip to convince it to use keyring anyway potentially causing processes to hang if the user incorrectly configured keyring for that use case. (The recommanded way to use it would be by setting an environment variable or pip config set/manually editing a pip config file. Not to pass it via the cli.)

Any thoughts?

@Darsstar I see three options:

• give the user the ability to change the value of no_input by changing the line to something like this pip_options.no_input = self.project.settings.get("disable_pip_input", True). I can confirm this is working.
• patching pip (which is embedded) and changing the L248 of auth.py to allow bypassing it.
• use the below code snipped to fake the authentication and uninstall the keychain

echo -e "machine europe-west3-python.pkg.dev\nlogin oauth2accesstoken\npassword $(gcloud auth print-access-token)" > $HOME/.netrc

you have to replace europe-west3-python.pkg.dev with the registry you'd like to work with.

[savitha-atg]

Any idea when this fix will be released? We're blocked from moving over to GCP artifact registry because of it.

[matteius]

@savitha-atg and cc: @allcloud-jonathan I was hoping we could get some documentation added as part of the PR: https://github.com/pypa/pipenv/pull/5036/files

Have you verified that PR is working for you?

Planning to do a pipenv release tonight, and so its possible we could merge it later today but like I said about the documentation. Also since there are no tests for this change, that should be considered if its possible to add a test that would prevent future regression, even if it was a mock.patch test. If we have to punt on the documentation and/or test for it, which seems less than ideal, lets at least ticket it before merging the other PR.

[savitha-atg]

No, I haven't. I'm very new to pipenv. I'm assuming I'd have to build pipenv from sources to test the fix?

[matteius]

@savitha-atg Its relatively simpler than that with pip. First you would pip uninstall pipenv version that you have -- then you would run:
pip install git+https://github.com/pypa/pipenv.git@allcloud-jonathan:feature/no_input_flag

If you do the preferred user installs of pipenv (which requires having that location on your PATH) then you would just add the --user flag to that command. Then you should be able to test the version with pipenv --version and test with the new option that has been added by adding it you your Pipfile.