setuptools 18.3.2 cannot access PyPI if PyPI password has an embedded %

Originally reported by: malemburg (Bitbucket: malemburg, GitHub: malemburg)

---

If you have a repository entry in .pypirc with a password using % in the password string, setuptools causes a traceback when trying to install packages with Python 3.4:

#!python

      File "./py3/lib/python3.4/site-packages/setuptools/package_index.py", line 427, in find_packages
        self.scan_url(self.index_url + requirement.unsafe_name+'/')
      File "./py3/lib/python3.4/site-packages/setuptools/package_index.py", line 761, in scan_url
        self.process_url(url, True)
      File "./py3/lib/python3.4/site-packages/setuptools/package_index.py", line 304, in process_url
        f = self.open_url(url, "Download error on %s: %%s -- Some packages may not be found!" % url)
      File "./py3/lib/python3.4/site-packages/setuptools/package_index.py", line 700, in open_url
        return open_with_auth(url, self.opener)
      File "./py3/lib/python3.4/site-packages/setuptools/package_index.py", line 894, in _socket_timeout
        return func(*args, **kwargs)
      File "./py3/lib/python3.4/site-packages/setuptools/package_index.py", line 992, in open_with_auth
        cred = PyPIConfig().find_credential(url)
      File "./py3/lib/python3.4/site-packages/setuptools/package_index.py", line 971, in find_credential
        for repository, cred in self.creds_by_repository.items():
      File "./py3/lib/python3.4/site-packages/setuptools/package_index.py", line 957, in creds_by_repository
        return dict(map(self._get_repo_cred, sections_with_repositories))
      File "./py3/lib/python3.4/site-packages/setuptools/package_index.py", line 963, in _get_repo_cred
        self.get(section, 'password').strip(),
      File "<pyrun>/configparser.py", line 773, in get
      File "<pyrun>/configparser.py", line 374, in before_get
      File "<pyrun>/configparser.py", line 423, in _interpolate_some
    configparser.InterpolationSyntaxError: '%' must be followed by '%' or '(', found: '%12345'

I tested this with setuptools 15.2 and this does not have the problem.

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/442

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

That's odd, because package_index.py hasn't changed since 15.2.

$ hg diff -r 15.2 setuptools/version.py 
diff -r df5dc9c7aa75 setuptools/version.py
--- a/setuptools/version.py Sun Apr 26 10:58:40 2015 -0400
+++ b/setuptools/version.py Tue Sep 29 12:27:25 2015 -0400
@@ -1,1 +1,1 @@
-__version__ = '15.2'
+__version__ = '18.3.3'
$ hg diff -r 15.2 setuptools/package_index.py 
$ hg diff -r 15.2 setuptools/compat.py
$ 

Is it possible that when you tested 15.2 that something else was different in your environment from your test against 18.3.2?

Original comment by malemburg (Bitbucket: malemburg, GitHub: malemburg):

---

I just tried to replicate this: it does indeed happen with setuptools 15.2 and the current 18.3.2. I must have done something else between reverting to 15.2 and trying the install again.

Here's the command I tried to run:

#!shell

bin/pip install cryptography

I also checked with a Python 3.4.3 venv: same error.

The install works fine in Python 2.7.10.

BTW: I do wonder why setuptools is interested in my PyPI account details when simply downloading and installing software.

Here's the complete pip output:

#!python

    Complete output from command /home/lemburg/tmp/cryptography/py3/bin/python3.4 -c "import setuptools, tokenize;__file__='/home/lemburg/tmp/cryptography/py3/build/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-739r_jc2-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/lemburg/tmp/cryptography/py3/include/site/python3.4:
    Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "/home/lemburg/tmp/cryptography/py3/build/cryptography/setup.py", line 313, in <module>

    **keywords_with_side_effects(sys.argv)

  File "/usr/local/python-3.4/lib/python3.4/distutils/core.py", line 108, in setup

    _setup_distribution = dist = klass(attrs)

  File "/home/lemburg/tmp/cryptography/py3/lib/python3.4/site-packages/setuptools/dist.py", line 239, in __init__

    self.fetch_build_eggs(attrs.pop('setup_requires'))

  File "/home/lemburg/tmp/cryptography/py3/lib/python3.4/site-packages/setuptools/dist.py", line 263, in fetch_build_eggs

    parse_requirements(requires), installer=self.fetch_build_egg

  File "/home/lemburg/tmp/cryptography/py3/lib/python3.4/site-packages/pkg_resources.py", line 564, in resolve

    dist = best[req.key] = env.best_match(req, self, installer)

  File "/home/lemburg/tmp/cryptography/py3/lib/python3.4/site-packages/pkg_resources.py", line 802, in best_match

    return self.obtain(req, installer) # try and download/install

  File "/home/lemburg/tmp/cryptography/py3/lib/python3.4/site-packages/pkg_resources.py", line 814, in obtain

    return installer(requirement)

  File "/home/lemburg/tmp/cryptography/py3/lib/python3.4/site-packages/setuptools/dist.py", line 313, in fetch_build_egg

    return cmd.easy_install(req)

  File "/home/lemburg/tmp/cryptography/py3/lib/python3.4/site-packages/setuptools/command/easy_install.py", line 581, in easy_install

    self.local_index

  File "/home/lemburg/tmp/cryptography/py3/lib/python3.4/site-packages/setuptools/package_index.py", line 590, in fetch_distribution

    self.find_packages(requirement)

  File "/home/lemburg/tmp/cryptography/py3/lib/python3.4/site-packages/setuptools/package_index.py", line 428, in find_packages

    self.scan_url(self.index_url + requirement.unsafe_name+'/')

  File "/home/lemburg/tmp/cryptography/py3/lib/python3.4/site-packages/setuptools/package_index.py", line 768, in scan_url

    self.process_url(url, True)

  File "/home/lemburg/tmp/cryptography/py3/lib/python3.4/site-packages/setuptools/package_index.py", line 305, in process_url

    f = self.open_url(url, "Download error on %s: %%s -- Some packages may not be found!" % url)

  File "/home/lemburg/tmp/cryptography/py3/lib/python3.4/site-packages/setuptools/package_index.py", line 702, in open_url

    return open_with_auth(url, self.opener)

  File "/home/lemburg/tmp/cryptography/py3/lib/python3.4/site-packages/setuptools/package_index.py", line 901, in _socket_timeout

    return func(*args, **kwargs)

  File "/home/lemburg/tmp/cryptography/py3/lib/python3.4/site-packages/setuptools/package_index.py", line 999, in open_with_auth

    cred = PyPIConfig().find_credential(url)

  File "/home/lemburg/tmp/cryptography/py3/lib/python3.4/site-packages/setuptools/package_index.py", line 978, in find_credential

    for repository, cred in self.creds_by_repository.items():

  File "/home/lemburg/tmp/cryptography/py3/lib/python3.4/site-packages/setuptools/package_index.py", line 964, in creds_by_repository

    return dict(map(self._get_repo_cred, sections_with_repositories))

  File "/home/lemburg/tmp/cryptography/py3/lib/python3.4/site-packages/setuptools/package_index.py", line 970, in _get_repo_cred

    self.get(section, 'password').strip(),

  File "/usr/local/python-3.4/lib/python3.4/configparser.py", line 773, in get

    d)

  File "/usr/local/python-3.4/lib/python3.4/configparser.py", line 374, in before_get

    self._interpolate_some(parser, option, L, value, section, defaults, 1)

  File "/usr/local/python-3.4/lib/python3.4/configparser.py", line 423, in _interpolate_some

    "found: %r" % (rest,))

configparser.InterpolationSyntaxError: '%' must be followed by '%' or '(', found: '%12345'

----------------------------------------
Cleaning up...
Command /home/lemburg/tmp/cryptography/py3/bin/python3.4 -c "import setuptools, tokenize;__file__='/home/lemburg/tmp/cryptography/py3/build/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-739r_jc2-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/lemburg/tmp/cryptography/py3/include/site/python3.4 failed with error code 1 in /home/lemburg/tmp/cryptography/py3/build/cryptography
Storing debug log for failure in /home/lemburg/.pip/pip.log

Original comment by malemburg (Bitbucket: malemburg, GitHub: malemburg):

---

Just verified: commenting out the entry with the % in the password result in the install command to run through fine.

It is interesting that all repo entries from the .pypirc are read. The one with the % wasn't the standard PyPI entry one.

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

In that case, since it's not a regression, we can treat it just like a bug. I do think it should be fixed. Maybe it be fixed by simply using SafeConfigParser instead. It'd be nice to have a unit test for this failure too.

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Add test capturing InterpolationSyntaxError on Python 3. Ref #442

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Use RawConfigParser instead of SafeConfigParser in PyPIConfig class. Interpolated values are no longer supported. Since backward compatibility could not be retained in either case, prefer the simpler, more direct format. Ref #442.

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Use SafeConfigParser in PyPIConfig file. Allows percent signs to be specified using two percent signs. Fixes #442.

Original comment by rpodgorny (Bitbucket: rpodgorny, GitHub: rpodgorny):

---

are you sure using rawconfigparser is a good idea? the docs say:

Legacy variant of the ConfigParser with interpolation disabled... (mind the "legacy" word)

also:

Note Consider using ConfigParser instead which checks types of the values to be stored internally. If you don’t want interpolation, you can use ConfigParser(interpolation=None).

cheers!

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I'm happy to share my rationale. I did read the docs, and was considering using ConfigParser(interpolation=None) when I found that the interpolation parameter doesn't exist in Python 2.7, so that's a non-option. When we drop support for Python 2.7, I'll switch to the preferred usage for Python 3.

I considered leaving interpolation on and using the SafeConfigParser on Python 2.7, which would at least bring the behavior into parity with Python 3, but on further consideration, I suspect that interpolation is never used in .pypirc, was never documented to support that functionality, and has the undesirable effect of requiring values containing % to be escaped, which is awkward (requires documentation, may be surprising). Furthermore, given that the primary usage of .pypirc is still dominantly Python 2.7 and the behavior there is to accept % unescaped (as first revealed by this bug), using RawConfigParser is probably the least disruptive and using SafeConfigParser would be a more disruptive and also backward-incompatible.

I can imagine scenarios where this backward-incompatible change could cause substantial disruption, and I'm gambling on the hope that those scenarios don't exist in practice. If someone does encounter a real-world example of why this approach is problemmatic, I'd like the feedback and I'll consider another approach.

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

This change was released in 19.0 and seems to be stable. I'm going to recommend that distutils adopt the same fix.