3.7.3rc1 Install Certificates fails on macOS

[dimaqq]

BPO	36340
Nosy	@ronaldoussoren, @ned-deily, @dimaqq
Superseder	bpo-36344: install_certificates.command too complicated
Files	Screenshot 2019-03-19 at 14.08.31.png

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ned-deily'
closed_at = <Date 2019-03-24.20:46:00.225>
created_at = <Date 2019-03-18.09:06:23.296>
labels = ['OS-mac', '3.7', 'expert-installation']
title = '3.7.3rc1 Install Certificates fails on macOS'
updated_at = <Date 2019-03-24.20:46:00.224>
user = 'https://github.com/dimaqq'

bugs.python.org fields:

activity = <Date 2019-03-24.20:46:00.224>
actor = 'ned.deily'
assignee = 'ned.deily'
closed = True
closed_date = <Date 2019-03-24.20:46:00.225>
closer = 'ned.deily'
components = ['Installation', 'macOS']
creation = <Date 2019-03-18.09:06:23.296>
creator = 'Dima.Tisnek'
dependencies = []
files = ['48218']
hgrepos = []
issue_num = 36340
keywords = []
message_count = 5.0
messages = ['338179', '338181', '338316', '338319', '338753']
nosy_count = 3.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'Dima.Tisnek']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '36344'
type = None
url = 'https://bugs.python.org/issue36340'
versions = ['Python 3.7']

[dimaqq]

I've just installed Python 3.7.3rc1 for macOS 10.9 or later from the macOS 64-bit installer.

I've clicked the Install Certificates.command, which opened a Terminal, ran and failed with:

Last login: Mon Mar 18 16:36:21 on ttys010
Welcome to fish, the friendly interactive shell
… ~> /Applications/Python\ 3.7/Install\ Certificates.command ; exit;
 -- pip install --upgrade certifi
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip/__main__.py", line 16, in <module>
    from pip._internal import main as _main  # isort:skip # noqa
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip/_internal/__init__.py", line 19, in <module>
    from pip._vendor.urllib3.exceptions import DependencyWarning
ModuleNotFoundError: No module named 'pip._vendor.urllib3.exceptions'
Traceback (most recent call last):
  File "<stdin>", line 44, in <module>
  File "<stdin>", line 25, in main
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7', '-E', '-s', '-m', 'pip', 'install', '--upgrade', 'certifi']' returned non-zero exit status 1.

[Process completed]

I was able to run the same command from my regular shell (iTerm2, fish, $PATH addons, etc.) and that completed just fine and installed certifi==2019.3.9

Either there's just something wrong with my system (what?) or the initial experience for average user is broken (unlikely?)...

[dimaqq]

More system info:
I've had 3.7.2 installed from official installer prior to this.
Before that, I had 3.7.1 installed from official installer.
I also have 3.8a2. 3.6.8, 2.7.15 on the system, as well as an odd version from Homebrew.

My system site-packages for 3.7 now (after manual fix) contains ceritifi, pip (19.0.3), setuptools, pkg_resources, easy_install.py and README.

[ned-deily]

At first glance, I'm not sure what happened here; we do try to make Install Certificates as bulletproof as possible. As you probably know, clicking on the file causes it to be opened with the macOS application that Launch Services determines is appropriate. By default, .command files are associated with Terminal.app (which can be verified by using the Finder's Get Info command on the .command file). Now apparently you have installed the fish shell (something Apple doesn't ship with macOS) and have changed Terminal.app's preferences or your user account to use fish as the default shell. The Install Certificates command is a shebang line that should cause it to be executed with the newly-installed Python and that seems to have happened. But then for some reason, the vendored urllib3 could not be imported correctly. If you were able to successfully run "Install Certificates.command" from a regular terminal window - without having reinstalled pip - that sounds like some sort of permissions problems which should not happen. Or some other sort of shell startup difference (although the script invokes python with -E to ignore PYTHON* env vars). Ah, but I now notice you say your normal terminal window is an iTerm2 one. So, if when double-clicking, the command file runs under Terminal.app but when you run in manually, it's under iTerm2, there *might* be some discrepancy there - I'm not sure what. One easy thing to try: now that certifi was successfully installed, what happens if you try rerunning the command by double-clicking it? Does it still fail in the same way? If so, it would be interesting to get more info on the environment the failing command is running in. Perhaps the easiest way to do that would be to make a copy of the command file and modify it to do:

    subprocess.check_call([sys.executable,
        "-E", "-s", "-m", "test.pythoninfo"])

just prior to the check_call to install certifi. As it stands, I'm unable to reproduce the failure with a vanilla macOS system without intentionally modifying permissions etc.

[dimaqq]

I've figured out what's going on:

When Installer runs, it asks for user's su passwords, does a bunch of stuff, and then starts "Running package scripts".

While it's "running scripts", towards the end of that process, with "about one minute remaining", the Finder window pops up.

If "Install Certificates.command" is activated at that time, pip fails.

However, if user waits for "running scripts" to complete, the pip succeeds.

The "race" window is less than half a minute.

Below are the examples of site packages listing during the race window:

Tue Mar 19 14:20:50 JST 2019
total 8.0K
drwxr-xr-x  28 root admin  896 Mar 19 14:20 setuptools/
drwxrwxr-x   7 root admin  224 Mar 19 14:20 ./
drwxr-xr-x  10 root admin  320 Mar 19 14:20 setuptools-40.8.0.dist-info/
drwxr-xr-x   3 root admin   96 Mar 19 14:20 __pycache__/
-rw-r--r--   1 root admin  126 Mar 19 14:20 easy_install.py
drwxrwxr-x 208 root admin 6.5K Mar 13 05:13 ../
-rw-rw-r--   1 root admin  119 Mar 13 05:13 README.txt

Tue Mar 19 14:20:51 JST 2019
total 8.0K
drwxr-xr-x  11 root admin  352 Mar 19 14:20 setuptools-40.8.0.dist-info/
drwxr-xr-x   7 root admin  224 Mar 19 14:20 pkg_resources/
drwxrwxr-x   8 root admin  256 Mar 19 14:20 ./
drwxr-xr-x  42 root admin 1.4K Mar 19 14:20 setuptools/
drwxr-xr-x   3 root admin   96 Mar 19 14:20 __pycache__/
-rw-r--r--   1 root admin  126 Mar 19 14:20 easy_install.py
drwxrwxr-x 208 root admin 6.5K Mar 13 05:13 ../
-rw-rw-r--   1 root admin  119 Mar 13 05:13 README.txt

Tue Mar 19 14:20:52 JST 2019
total 8.0K
drwxr-xr-x   6 root admin  192 Mar 19 14:20 pip/
drwxrwxr-x   9 root admin  288 Mar 19 14:20 ./
drwxr-xr-x  11 root admin  352 Mar 19 14:20 setuptools-40.8.0.dist-info/
drwxr-xr-x   7 root admin  224 Mar 19 14:20 pkg_resources/
drwxr-xr-x  42 root admin 1.4K Mar 19 14:20 setuptools/
drwxr-xr-x   3 root admin   96 Mar 19 14:20 __pycache__/
-rw-r--r--   1 root admin  126 Mar 19 14:20 easy_install.py
drwxrwxr-x 208 root admin 6.5K Mar 13 05:13 ../
-rw-rw-r--   1 root admin  119 Mar 13 05:13 README.txt

Tue Mar 19 14:20:53 JST 2019
total 8.0K
drwxr-xr-x   3 root admin   96 Mar 19 14:20 __pycache__/
drwxr-xr-x   9 root admin  288 Mar 19 14:20 pip-19.0.3.dist-info/
drwxrwxr-x  10 root admin  320 Mar 19 14:20 ./
drwxr-xr-x   7 root admin  224 Mar 19 14:20 pip/
drwxr-xr-x  11 root admin  352 Mar 19 14:20 setuptools-40.8.0.dist-info/
drwxr-xr-x   7 root admin  224 Mar 19 14:20 pkg_resources/
drwxr-xr-x  42 root admin 1.4K Mar 19 14:20 setuptools/
-rw-r--r--   1 root admin  126 Mar 19 14:20 easy_install.py
drwxrwxr-x 208 root admin 6.5K Mar 13 05:13 ../
-rw-rw-r--   1 root admin  119 Mar 13 05:13 README.txt

I suspect that this is not a big deal in practice, because a new user would not know to run the command until instructed in the "installation completed" screen.

Meanwhile, "experienced / trigger-happy" users probably know to run the command again :)

P.S. I'm also wondering if it matters that installer asks for current user's su password and after a good installation, most content of site-packages is owned by root:admin, while certifi is owned by current user... For example __pycache__ not being writable by user.

[ned-deily]

Ah, thanks for the further analysis, that makes more sense! The current requirement to manually launch Install Certificates is obviously less than ideal so I think the best way to avoid the race condition here is to eliminate the manual step and have the certificates installed automatically. I've re-opened bpo-36344 to do that and am thus closing this issue as duplicate of it.