Add an interface to allow calling system `keyring` #11589

judahrand · 2022-11-09T19:34:00Z

This PR allows pip to use system wide keyring installations which appear on PATH.

In order to avoid breaking peoples current systems this will still default to using keyring imported from
the local environment. A different keyring installation will only be used if no local keyring is found.

Edit: this may also partially address #8485 as it delays the import of keyring

src/pip/_internal/network/auth.py

judahrand · 2022-11-09T20:39:58Z

I'm not sure there's much to be tested here since we don't actually install keyring for testing if I understand correctly? 🤔

I haven't added any code branches outside KeyRingCli and testing the class itself is somewhat a pointless endeavor - it doesn't do much without keyring. But I'm happy to add whatever you feel is worthwhile.

pfmoore

I've made some general comments, but to be perfectly honest I don't use keyring at all, and I have little experience with what else might be needed here. So just to set expectations, I'm happy to review the code in general, but I'd want other maintainers to comment on whether this is an acceptable feature to add, and approve the implementation.

pfmoore · 2022-11-09T20:42:03Z

src/pip/_internal/network/auth.py

+    @classmethod
+    def set_password(cls, service_name: str, username: str, password: str) -> None:
+        cmd = ["keyring", "set", service_name, username]
+        input_ = password.encode() + b"\n"


I can imagine encoding issues here, especially on Windows.

I think I've solved this using PYTHONIOENCODING

pfmoore · 2022-11-09T21:05:48Z

src/pip/_internal/network/auth.py

-        res = subprocess.run(cmd)
-        if res.returncode:
-            raise RuntimeError(res.stderr)
+        password = res.stdout.decode().strip("\n")


Encoding issues possible here. On Windows, at least, it's not necessarily true that keywring will write its output in the same encoding as the pip process' default encoding.

I think I've solved this using PYTHONIOENCODING

pfmoore · 2022-11-09T21:22:27Z

src/pip/_internal/network/auth.py

    ) -> Optional[KeyRingCredential]:
-        cmd = ["keyring", "get", self._quote(service_name), self._quote(username)]
-        res = subprocess.run(cmd)
+        cmd = ["keyring", "get", service_name, str(username)]


Why str() round username? You don't account for the possibility of None (allowed by the type signature) and str() will do nothing if it's a string.

I've avoided this by only supporting get_password. I think this is the more correct thing to do as this is actually the function which is being called by the CLI.

I've moved to mirroring keyring's default implementation of get_credential which just wraps get_password

pfmoore · 2022-11-09T21:23:57Z

src/pip/_internal/network/auth.py

-        cmd = ["keyring", "get", self._quote(service_name), self._quote(username)]
-        res = subprocess.run(cmd)
+        cmd = ["keyring", "get", service_name, str(username)]
+        res = subprocess.run(cmd, capture_output=True)


You need to support the --no-input option here, so don't try to read stdin if the user supplied that flag.

I don't think I understand this. I'm not reading from stdin here.

Did you mean this line res = subprocess.run(cmd, input=input_)? If so the reason I'm doing this is that the user has already supplied the password through interaction. We're just saving the result here in a callback. Unfortunately, keyring doesn't support passing the password in any other way as far as I can see.

No, the keyring command you run might try to read from stdin. The --no-input flag is specifically to ensure that people can run pip without getting prompted for input, so you need to ensure that if --no-input is specified, you stop the keyring subprocess from reading stdin - probably by passing stdin=subprocess.DEVNULL to the call, assuming the keyring process works correctly if you do that.

pfmoore · 2022-11-09T22:18:41Z

I'm not sure there's much to be tested here since we don't actually install keyring for testing if I understand correctly?

I think we should test this. If the existing keyring code isn't tested, then I see that as a flaw in the original implementation, not a precedent we should follow...

judahrand · 2022-11-09T22:25:07Z

I'm not sure there's much to be tested here since we don't actually install keyring for testing if I understand correctly?

I think we should test this. If the existing keyring code isn't tested, then I see that as a flaw in the original implementation, not a precedent we should follow...

The existing code is tested but keyring is patched to return expected results. Are you suggesting adding some tests which require keyring to be installed? Happy to do that - is there a pattern to follow for tests which rely on external dependencies?

It shouldn't need to ever so no reason to allow it and have to jiggle around the `--no-input` option in `pip`.

uranusjr · 2022-11-10T08:35:16Z

src/pip/_internal/network/auth.py


 try:
    import keyring
 except ImportError:
    keyring = None  # type: ignore[assignment]
+    keyring_path = shutil.which("keyring")
+    if keyring_path is not None:
+        keyring = KeyRingCli(keyring_path)  # type: ignore[assignment]


Instead of trying to fake keyring’s Python interface, it’d probably be easier if we introduce a common abstraction. We can have KeyRingCliProvider and KeyRingPythonProvider wrapping each implementation. This should also help get rid of the ImportError block.

I've acted on this feedback

pfmoore · 2022-11-10T09:39:43Z

src/pip/_internal/network/auth.py

+            cmd,
+            stdin=subprocess.DEVNULL,
+            capture_output=True,
+            env=dict(PYTHONIOENCODING="utf-8"),


I believe setting ENV will remove all other environment variables, including ones that the user might have set to control the keyring command (which might be the right thing to do - I don't know about that) and some essential system variables on Windows.

Rather than using env like this, you need to take a copy of os.environ, modify it, and pass the modified copy into env.

I've acted on this feedback

judahrand · 2022-11-10T11:15:25Z

I'll need to overhaul the tests for this area of the code to fit in with the new abstraction.

judahrand · 2022-11-10T20:05:54Z

I'm not sure there's much to be tested here since we don't actually install keyring for testing if I understand correctly?

I think we should test this. If the existing keyring code isn't tested, then I see that as a flaw in the original implementation, not a precedent we should follow...

I've added some tests by mocking subprocess.run with the expected behavior of the keyring cli.

uranusjr · 2022-11-10T20:44:01Z

src/pip/_internal/network/auth.py

+    python_keyring = KeyRingPythonProvider()
+    if python_keyring.is_available():
+        return python_keyring
+
+    cli_keyring = KeyRingCliProvider()
+    if cli_keyring.is_available():
+        return cli_keyring


Since we only return a provider if the backend is available anyway, I wonder if it’d be easier to do something like

try: import keyring except ImportError: keyring = None # type: ignore[assignment] class KeyRingCliProvider: def __init__(self, cmd: str) -> None: self.keyring = cmd def get_keyring_provider() -> Optional[KeyRingBaseProvider]: if keyring is not None: return KeyRingPythonProvider() cli = shutil.which("keyring") if cli: return KeyRingCliProvider(cli) return None

This avoids a few is_available checks.

I also wonder whether it’d be a good idea to have a KeyRingNullProvider that always fails; this should help localise the KEYRING_DISABLED logic in get_keyring_provider and eliminate some is None checks.

I think there is an advantage to delaying the import keyring until get_keyring_provider is called. It means that if the user has provided a password we won't bother trying to import keyring (which from what I've heard can be very slow!).

I've implemented your idea of having a KeyRingNullProvider, doing away with is_available, and simplifying the KEYRING_DISABLED logic. This necessitates removing the cache on get_keyring_provider but that's probably fine.

src/pip/_internal/network/auth.py

pfmoore · 2022-11-10T21:53:52Z

I've reached the point now where this looks good to me, and I'm assuming that the way it uses the keyring client is OK (I have very limited knowledge of the keyring module). So I've approved it but I'll wait for @uranusjr's further review.

judahrand · 2022-11-10T21:58:26Z

I've reached the point now where this looks good to me, and I'm assuming that the way it uses the keyring client is OK (I have very limited knowledge of the keyring module). So I've approved it but I'll wait for @uranusjr's further review.

Thanks very much, Paul. Appreciate your patience with me. I'm excited to be (potentially -- still early days) making my first contribution to pip. Massive FOSS supporter and very grateful for your work.

pfmoore · 2022-11-10T22:05:36Z

It's been a pleasure 🙂

pradyunsg · 2022-11-11T11:43:41Z

We didn't update the documentation -- @judahrand would you be interested in covering this in https://pip.pypa.io/en/stable/topics/authentication/#keyring-support?

I guess that section would need to be modified to have subheaders for the two mechanisms and the header for subprocess calls can get a

```{versionadded} 23.0
```

at the start of it.

judahrand · 2022-11-11T11:46:16Z

We didn't update the documentation -- @judahrand would you be interested in covering this in https://pip.pypa.io/en/stable/topics/authentication/#keyring-support?

I guess that section would need to be modified to have subheaders for the two mechanisms and the header for subprocess calls can get a
```{versionadded} 23.0
at the start of it.

I'll try to get to this in the next day or two.

pradyunsg · 2022-11-11T11:47:11Z

Awesome, thank you! ^.^

Add an interface to allow calling system keyring

b87ddb9

judahrand force-pushed the keyring-cli branch from 3ddc9f6 to b87ddb9 Compare November 9, 2022 19:34

Add news

edc588c

pfmoore reviewed Nov 9, 2022

View reviewed changes

src/pip/_internal/network/auth.py Outdated Show resolved Hide resolved

judahrand marked this pull request as draft November 9, 2022 20:06

Improve cli interface

4cbae5b

judahrand marked this pull request as ready for review November 9, 2022 20:29

judahrand added 2 commits November 9, 2022 20:38

Raise better exception

efa7f2b

Don't capture output

7e93102

pfmoore requested changes Nov 9, 2022

View reviewed changes

judahrand added 3 commits November 9, 2022 22:31

Handle IO encoding

6ec0af5

Switch to defining get_password

f5c96b1

Set keyring correctly

43abcf0

judahrand force-pushed the keyring-cli branch from d57e834 to 43abcf0 Compare November 9, 2022 22:54

judahrand added 2 commits November 9, 2022 23:13

Use full keyring path

5137ce2

Prevent keyring from ever reading from stdin

4fc2008

It shouldn't need to ever so no reason to allow it and have to jiggle around the `--no-input` option in `pip`.

uranusjr reviewed Nov 10, 2022

View reviewed changes

pfmoore requested changes Nov 10, 2022

View reviewed changes

Abstract provider interface to keyring

888c3b6

judahrand force-pushed the keyring-cli branch from 1ecade4 to 888c3b6 Compare November 10, 2022 10:34

judahrand added 2 commits November 10, 2022 10:36

Take copy of os.environ rather than editing

996d4fa

Import keyring lazily

4f8a613

Get the tests passing again

3a15e01

judahrand requested a review from pfmoore November 10, 2022 20:02

Add tests for new code paths

8d9ea8b

judahrand force-pushed the keyring-cli branch from bd4eb67 to 8d9ea8b Compare November 10, 2022 20:05

uranusjr reviewed Nov 10, 2022

View reviewed changes

Simplify provider interface

c04222f

uranusjr reviewed Nov 10, 2022

View reviewed changes

src/pip/_internal/network/auth.py Show resolved Hide resolved

judahrand added 3 commits November 10, 2022 21:34

Move keyring import

e6e42de

Don't silently fallback to cli

14a3d93

Do fallback but issue a warning

623ac5d

pfmoore approved these changes Nov 10, 2022

View reviewed changes

judahrand requested a review from uranusjr November 10, 2022 21:51

uranusjr approved these changes Nov 10, 2022

View reviewed changes

uranusjr merged commit 1b9cc0f into pypa:main Nov 10, 2022

judahrand mentioned this pull request Nov 10, 2022

Vendor keyring & keyring-subprocess #11399

Closed

judahrand added a commit to judahrand/pip that referenced this pull request Nov 11, 2022

Add documentation on functionality in pypa#11589

89a4a35

SnoopJ mentioned this pull request Nov 15, 2022

Make keyring integration opt-in (closes #8719, #6773) #11215

Closed

github-actions bot locked as resolved and limited conversation to collaborators Nov 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an interface to allow calling system `keyring` #11589

Add an interface to allow calling system `keyring` #11589

judahrand commented Nov 9, 2022 •

edited

judahrand commented Nov 9, 2022 •

edited

pfmoore left a comment

pfmoore Nov 9, 2022

judahrand Nov 9, 2022

pfmoore Nov 9, 2022

judahrand Nov 9, 2022

pfmoore Nov 9, 2022

judahrand Nov 9, 2022

judahrand Nov 10, 2022

pfmoore Nov 9, 2022

judahrand Nov 9, 2022

pfmoore Nov 9, 2022

pfmoore commented Nov 9, 2022

judahrand commented Nov 9, 2022

uranusjr Nov 10, 2022

judahrand Nov 10, 2022

pfmoore Nov 10, 2022

judahrand Nov 10, 2022

judahrand commented Nov 10, 2022

judahrand commented Nov 10, 2022

uranusjr Nov 10, 2022 •

edited

judahrand Nov 10, 2022

pfmoore commented Nov 10, 2022

judahrand commented Nov 10, 2022

pfmoore commented Nov 10, 2022

pradyunsg commented Nov 11, 2022

judahrand commented Nov 11, 2022

pradyunsg commented Nov 11, 2022

Add an interface to allow calling system keyring #11589

Add an interface to allow calling system keyring #11589

Conversation

judahrand commented Nov 9, 2022 • edited

judahrand commented Nov 9, 2022 • edited

pfmoore left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pfmoore commented Nov 9, 2022

judahrand commented Nov 9, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

judahrand commented Nov 10, 2022

judahrand commented Nov 10, 2022

uranusjr Nov 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pfmoore commented Nov 10, 2022

judahrand commented Nov 10, 2022

pfmoore commented Nov 10, 2022

pradyunsg commented Nov 11, 2022

judahrand commented Nov 11, 2022

pradyunsg commented Nov 11, 2022

Add an interface to allow calling system `keyring` #11589

Add an interface to allow calling system `keyring` #11589

judahrand commented Nov 9, 2022 •

edited

judahrand commented Nov 9, 2022 •

edited

uranusjr Nov 10, 2022 •

edited