Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recommended workflow for switching between multiple API tokens? #496

Open
nlhkabu opened this issue Sep 11, 2019 · 28 comments
Open

Recommended workflow for switching between multiple API tokens? #496

nlhkabu opened this issue Sep 11, 2019 · 28 comments
Labels
blocked Issues we can't or shouldn't get to yet question Discussion/decision needed from maintainers

Comments

@nlhkabu
Copy link
Member

nlhkabu commented Sep 11, 2019

Hi folks

I am working on adding instructions to the 'Add API token' page on PyPI (see pypi/warehouse#6615). Currently, the instructions look like this:

API token for an entire PyPI account

We suggest that users create a .pypirc file in their home directory with the token details:

Screenshot from 2019-09-11 07-11-27

Token scoped to a PyPI project

We suggest that users can switch between multiple tokens with <code>twine --repository PROJECT_NAME</code>

Screenshot from 2019-09-11 07-12-09

Questions

  1. Is this accurate?
  2. Is this the recommended workflow?
  3. Am I missing anything that it would be useful to document here?

Thanks :)

@nlhkabu nlhkabu changed the title Recommended workflow for using multiple API tokens? Recommended workflow for switching between multiple API tokens? Sep 11, 2019
@sigmavirus24
Copy link
Member

That looks accurate. I'd say that it's a hack but it's the first thing I thought of when I saw your question.

It would probably be good to have another way of managing these though instead of forcing folks to use --repository

@nlhkabu
Copy link
Member Author

nlhkabu commented Sep 13, 2019

Ok, thanks @sigmavirus24. I will go ahead and get these instructions merged into Warehouse.

@bhrutledge
Copy link
Contributor

@sigmavirus24 and @nlhkabu is it okay to close this? And/or should we open an issue for improving the UX for using Twine with API tokens?

@bhrutledge
Copy link
Contributor

bhrutledge commented Jan 26, 2020

FYI @sigmavirus24 and @nlhkabu (and @di for good measure):

After trying this out for myself, I think the instructions might not be sufficient. Given this ~/.pypirc:

[pypi]
username = __token__

[testpypi]
username = __token__

[example-pkg]
repository = https://test.pypi.org/legacy/
username = __token__
password = pypi-...

I get this output:

$ twine upload --repository example-pkg dist/*
InvalidConfiguration: Missing 'example-pkg' section from the configuration file
or not a complete URL in --repository-url.
Maybe you have a out-dated '~/.pypirc' format?
more info: https://docs.python.org/distutils/packageindex.html#pypirc

It works if add I this to the top of~/.pypirc:

[distutils]
index-servers =
    pypi
    testpypi
    example-pkg

I discovered that twine.utils.get_config only uses values for defined index-servers, which defaults to ["pypi", "testpypi"].

twine/twine/utils.py

Lines 78 to 85 in 330c0a2

# optional configuration values for individual repositories
for repository in index_servers:
for key in [
"username", "repository", "password",
"ca_cert", "client_cert",
]:
if parser.has_option(repository, key):
config[repository][key] = parser.get(repository, key)

A quick search didn't uncover any issues on Warehouse, which surprised me. It seems like a quick addition to the relevant Warehouse docs, but seems to add to the case for improving Twine's handling of tokens.

bhrutledge added a commit to bhrutledge/warehouse that referenced this issue Jan 26, 2020
Per my comment in pypa/twine#496 (comment), it seems a "project" repository must be defined in `index-servers` in order to be usable by twine.
@bhrutledge
Copy link
Contributor

Related: #565 - Not obvious how to use multiple project API tokens with keyring

di pushed a commit to pypi/warehouse that referenced this issue Jan 26, 2020
* Fix example .pypirc for project tokens

Per my comment in pypa/twine#496 (comment), it seems a "project" repository must be defined in `index-servers` in order to be usable by twine.

* Add missing repository option
@di
Copy link
Sponsor Member

di commented Jan 26, 2020

While this works, I agree with @sigmavirus24 that it's a bit of a hack. I think it's probably worthwhile for us to think of a new and better way to do this in the .pypirc file (and maybe standardize the format while we're at it?)

Maybe something like this for repo-wide tokens:

[distutils]
index-servers =
    pypi
    other

[pypi]
token = pypi-ABC...

[other]
repository = http://example.com/pypi
token = pypi-DEF...

And this for project-specific tokens:

[distutils]
index-servers =
    pypi
    other

[other]
repository = http://example.com/pypi

[pypi:example-pkg]
token = pypi-ABC...

[other:example-pkg]
token = pypi-DEF...

@bhrutledge
Copy link
Contributor

Thanks, @di. I wondered how much freedom we might have with the .pypirc format. I like how this removes the need for repository = https://upload.pypi.org/legacy/ in the config for project-scoped tokens.

FYI, there's an evolving discussion that started at pypa/packaging.python.org#297 (comment), where @jaraco and @takluyver have proposed some ideas on how to make this work with keyring and flit.

@di
Copy link
Sponsor Member

di commented Jan 27, 2020

I could be wrong, but I think these extra fields/sections would be ignored by any tool that doesn't know about them, so it should be fully backwards-compatible.

@bhrutledge bhrutledge added the blocked Issues we can't or shouldn't get to yet label May 28, 2020
@bhrutledge
Copy link
Contributor

Marked this and #565 as blocked until the discussion started at pypa/packaging.python.org#297 (comment) is resolved.

@sigmavirus24
Copy link
Member

sigmavirus24 commented May 28, 2020

The other thing to keep in mind is that the ini format is fairly flexible and projects like pytest and flake8 are able to do things like:

option =
   something: value
   something-else: another-value

So we could keep this super simple and have a:

tokens =
   *: pypi<general token>
   twine: pypi<project-token>
   ...

That keeps us from proliferating sections into the file, keeps the implementation simple, and is farily explicit without trying to hack around things. Then we can select the tokens out of there as necessary without doing "repository" based nonsense. Getting this to work with Keyring will be harder, as the discussion Brian linked indicates

@di
Copy link
Sponsor Member

di commented May 28, 2020

@sigmavirus24 How would that handle the same project name with different tokens across multiple repos?

@sigmavirus24
Copy link
Member

sigmavirus24 commented May 28, 2020

@di A more fully fledged example would be:

[distutils]
index-servers =
    pypi
    other
    another

[pypi]
tokens = 
   *: pypi-ABC...
   twine: pypi-BCD...
   etc: pypi-CDE...

[another]
repository = http://example.com/pypi
tokens = 
   *: pypi-DEF...
   twine: pypi-EFG...
   etc: pypi-FGH...

[other]
repository = http://test.example.com/pypi
tokens = 
    *: pypi-GHI...
    etc: pypi-HIJ...

@di
Copy link
Sponsor Member

di commented May 28, 2020

Gotcha, so with that example, if the user just wanted to specify repo-wide tokens (which is probably the most common use case? maybe?) they'd have to write a minimum of:

[distutils]
index-servers =
    pypi
    other

[pypi]
tokens = 
   *: pypi-ABC...

[other]
repository = http://test.example.com/pypi
tokens = 
    *: pypi-GHI...

which feels clunky compared to:

[distutils]
index-servers =
    pypi
    other

[pypi]
token = pypi-ABC...

[other]
repository = http://example.com/pypi
token = pypi-DEF...

@bhrutledge bhrutledge added this to To do in Repository configuration via automation May 29, 2020
@bhrutledge
Copy link
Contributor

From pypa/packaging.python.org#297, it seems like the ideal scenario is that tokens are not stored in .pypirc, but rather in keyring. Does that influence the choice of format?

@sigmavirus24
Copy link
Member

I think you're always going to have people who would prefer not to use keyring. And we'd have to go through a lengthy deprecation period to kill that off which isn't worth the effort frankly.

Gotcha, so with that example, if the user just wanted to specify repo-wide tokens (which is probably the most common use case? maybe?) they'd have to write a minimum of:

The fact that we don't know what the most common use-case is suggests that we don't know enough to continue pushing forward on making the right choice

@di
Copy link
Sponsor Member

di commented May 29, 2020

As of right now, 60% of PyPI users who own/maintain at least package only have one package:

warehouse=> select count(*) from (select roles.user_id, count(roles.project_id) as
count from roles group by user_id) as projects;
 count
--------
 103550
(1 row)

warehouse=> select count(*) from (select roles.user_id, count(roles.project_id) as
count from roles group by user_id) as projects where projects.count = 1;
 count
-------
 62803
(1 row)

@bhrutledge
Copy link
Contributor

bhrutledge commented May 30, 2020

Related (I think): I just put up a draft PR that documents the current spec of .pypirc: pypa/packaging.python.org#734.

Also, we're talking a lot about .pypirc, but I'm curious how this would manifest in configuration via the command line and/or environment variables. For example, does this imply --token and TWINE_TOKEN? It seems friendlier for those to be aligned.

I'm wondering if it might be simpler to keep the current convention, but mitigate the hack by defaulting username to __token__, as suggested in #561.

I think that would also require fewer updates to the docs in Twine, PPUG, and Warehouse.

@sigmavirus24
Copy link
Member

As of right now, 60% of PyPI users who own/maintain at least package only have one package:

Is there a way to tell what % of that 60% use tokens at all?

Here's the point: If we go with token= then we're implicitly endorsing 1 global token that everyone should be using and thus we're going to calcify that use-case. Is the goal of project-scoped tokens not to (beyond enabling CI) to also encourage keeping tokens scoped appropriately to projects?

We should be meeting folks where they are today, certainly, but we should also be striving to push the community forward with the gentlest nudges we can think of.

@bhrutledge
Copy link
Contributor

bhrutledge commented May 30, 2020

@sigmavirus24 I think @di's original suggestion handles project-scoped tokens via additional sections that inherit from the "base" section:

[pypi]
token = pypi-ABC...

[pypi:example-pkg]
token = pypi-DEF...

Another idea:

[pypi]
token = pypi-ABC...
project_tokens = 
    example-pkg: pypi-DEF...

That said, I'd still like to consider sticking with the current username/password hack, such that Twine's default configuration becomes:

[distutils]
index-servers =
    pypi
    testpypi

[pypi]
repository = https://upload.pypi.org/legacy/
username = __token__

[testpypi]
repository = https://test.pypi.org/legacy/
username = __token__

which I think leans more towards something like this:

[pypi]
password = pypi-ABC...

[pypi:example-pkg]
password = pypi-DEF...

@bhrutledge
Copy link
Contributor

Building on my last suggestion, combined with my second proposal in #565 and pypa/packaging.python.org#297 (comment), I think this would allow CLI commands like:

$ keyring set pypi:example-pkg __token__
Password for '__token__' in 'pypi:example-pkg':

$ twine upload -r pypi:example-pkg dist/*

Aside: looking over those other issues makes me wonder if this is conversation should be moved to https://discuss.python.org/c/packaging/.

@di
Copy link
Sponsor Member

di commented Jun 1, 2020

A few things on the password/token "hack":

  1. I'd like us to imagine a world where using password in .pypirc is deprecated or discouraged, and I think that overloading password with the token value would make that hard or impossible.

  2. I think that a default username of __token__ might produce some confusing behavior in some edge cases, for example consider the following:

    [pypi]
    usenrame = my_username
    password = my_password

    Twine now has no way to tell the user their username is "missing" because it happily falls back on the default of __token__, which fails authentication, even though the user has included the correct username/password (unless we want to start warning the user every time Twine uses the default, which I think would get annoying for users who legitimately want to use tokens)

  3. Using token over password allows the user to explicitly tell us "I expect to use tokens" which allows us to potentially do some preflight checking of this value (i.e. whether it's a valid token). Otherwise, we'd have to implicitly infer this through how the user is using the username field.

  4. Having a clear delineation between what is a token and what is a password in configuration will help provide more context to whatever mechanism implements Pluggable credential manager framework proposal [PoC included] #362, which would allow an authentication provider to provide a different value based on what's being requested. E.g.:

    [pypi]
    password_provider = keyring
    
    [other]
    token_provider = some_other_module

    (where we can probably just consider keyring the default password/token provider, but I'm including it here just as an example)

@shon
Copy link

shon commented Jul 27, 2022

I faced the similar issue today

To reproduce

OS: Ubuntu 22.04
Python: 3.10.4
Twine: 4.0.1

$ cat ~/.pypirc

[distutils]
index-servers =
  pypi

[<package-name>]
repository = https://upload.pypi.org/legacy/
username = __token__
password = pypi-<snip>

Twine command

$ twine upload -r <my-package>  dist/*.tar.gz --verbose

INFO     Using configuration from /home/shon/.pypirc                                                                                                           
ERROR    InvalidConfiguration: Missing '<my-package>' section from ~/.pypirc.                                                                                       
         More info: https://packaging.python.org/specifications/pypirc/        

What worked

  • Rename [<my-package>] to [pypi]
  • twine upload -r <my-package> dist/*.tar.gz
    But then obviously I can't use this for other packages.

@bhrutledge
Copy link
Contributor

@shon Can you try adding <my-package> to index-servers in your .pypirc?

[distutils]
index-servers =
  pypi
  <my-package>

[<my-package>]
repository = https://upload.pypi.org/legacy/
username = __token__
password = pypi-<snip>

I think that will allow you to run:

twine upload --verbose -r <my-package>  dist/*.tar.gz

@shon
Copy link

shon commented Jul 27, 2022

Thanks @bhrutledge that worked!
Wasn't very intuitive to me that my-package should be mentioned index-servers.

@bhrutledge
Copy link
Contributor

@shon Me neither, which led to the discussion starting at #496 (comment).

@kentbull
Copy link

kentbull commented Mar 6, 2023

Agree with @shon. I like @bhrutledge and @di's comments on a path forward.

@Doondondon

This comment has been minimized.

@miketheman
Copy link
Member

Chiming in briefly to inquire if there's appetite for removing the need for the entire [distutils] section and index-servers keys. (I slammed face-first into this issue today.)

distutils has been removed from Python as of 3.12+, and twine hasn't used distutils directly for over a decade.

Is there still a benefit to preserving usage of this indirection, vs setting sections and looking for those? (Potentially replace configparser.RawConfigParser which is a legacy form of configparser.ConfigParser )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocked Issues we can't or shouldn't get to yet question Discussion/decision needed from maintainers
Projects
Development

No branches or pull requests

8 participants