How to pass --trusted-host or get brew to use corporate certificate passed to pip via $PIP_CERT?

[jwhendy]

Output of brew config

% brew config
HOMEBREW_VERSION: 4.2.10
ORIGIN: https://github.com/Homebrew/brew
HEAD: c6d959218f143cd17b1fc3e0f10f143cbd273528
Last commit: 4 days ago
Core tap JSON: 01 Mar 01:30 UTC
Core cask tap JSON: 01 Mar 01:30 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_REPOSITORY: /opt/homebrew/Homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 12
Homebrew Ruby: 3.1.4 => /opt/homebrew/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/bin/ruby
CPU: dodeca-core 64-bit arm_blizzard_avalanche
Clang: 15.0.0 build 1500
Git: 2.39.3 => /Library/Developer/CommandLineTools/usr/bin/git
Curl: 8.4.0 => /usr/bin/curl
macOS: 14.3.1-arm64
CLT: 15.1.0.0.1.1700200546
Xcode: N/A
Rosetta 2: false

Output of brew doctor

% brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Unbrewed dylibs were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected dylibs:
  /usr/local/lib/libMsgCom.dylib
  /usr/local/lib/libSysCtlCom.dylib
  /usr/local/lib/libwep_airdrop.dylib
  /usr/local/lib/libwep_burn.dylib
  /usr/local/lib/libwep_cbcarbon.dylib
  /usr/local/lib/libwep_cbcocoa.dylib
  /usr/local/lib/libwep_chrome.dylib
  /usr/local/lib/libwep_dutil.dylib
  /usr/local/lib/libwep_ff.dylib
  /usr/local/lib/libwep_icloud.dylib
  /usr/local/lib/libwep_mail.dylib
  /usr/local/lib/libwep_post.dylib
  /usr/local/lib/libwep_pref.dylib
  /usr/local/lib/libwep_printer.dylib
  /usr/local/lib/libwep_proxy.dylib
  /usr/local/lib/libwep_screen.dylib

Note: I'm posting in discussions because brew doesn't say "Your system is ready to brew." It also doesn't say there's a problem. Can one really not post a github issues due to the above warnings?

Description of issue

Issue #15764 is exactly what I'm running into, but after discussion of alternative certs, the discussion went stale.

I was trying to brew install qmk/qmk/qmk, and it fails at this point:

python3
-m
pip
--python=/opt/homebrew/Cellar/qmk/1.1.2_4/libexec/bin/python
install
--verbose
--no-deps
--no-binary=:all:
--ignore-installed
--no-compile
/private/tmp/qmk--argcomplete-20240301-3143-teqfl5/argcomplete-3.2.2

Using pip 24.0 from /opt/homebrew/lib/python3.12/site-packages/pip (python 3.12)
Processing /private/tmp/qmk--argcomplete-20240301-3143-teqfl5/argcomplete-3.2.2
  Installing build dependencies: started
  Running command pip subprocess to install build dependencies
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))': /simple/setuptools/
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))': /simple/setuptools/
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))': /simple/setuptools/
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))': /simple/setuptools/
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))': /simple/setuptools/
  Could not fetch URL https://pypi.org/simple/setuptools/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/setuptools/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))) - skipping
  ERROR: Could not find a version that satisfies the requirement setuptools>=67.7.2 (from versions: none)
  ERROR: No matching distribution found for setuptools>=67.7.2
  Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))) - skipping
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /opt/homebrew/Cellar/qmk/1.1.2_4/libexec/bin/python /opt/homebrew/lib/python3.12/site-packages/pip/__pip-runner__.py install --ignore-installed --no-user --prefix /private/tmp/pip-build-env-e5u8z1mo/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=67.7.2' 'setuptools_scm[toml]>=6.2'
  cwd: [inherit]
  Installing build dependencies: finished with status 'error'
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

As stated in the linked issue, it has something to do with certificates and the corporate firewall. I've run into this before, and one can use something like --trusted-host (see this SO post), and my company provided a cert bundle, which I have pip pointed to using PIP_CERT=/path/to/bundle.pem in my ~/.zshrc.

I'm wondering if anyone can suggest how to pass arguments or this bundle to pip when it's building dependencies like this? Or kindly let me know I don't understand the problem and it's something else.

[ZhongRuoyu]

Can you try the following? From the caveats in the output of brew info openssl@3:

A CA file has been bootstrapped using certificates from the system
keychain. To add additional certificates, place .pem files in
  /opt/homebrew/etc/openssl@3/certs

and run
  /opt/homebrew/opt/openssl@3/bin/c_rehash

[jwhendy]

@ZhongRuoyu excellent suggestion, that sounded like just what might fix it. Unfortunately, I still get the same issue after making the directory, copying the file there, and running that command. While the linked issue suggests this is about certs, I should say I'm open to other troubleshooting/debugging steps as I really don't know. It's just the closest hit I get when googling the error output. Thanks for the assistance.

[jwhendy]

Update, for what it's worth, I ran with --keep-tmp and --debug so that the temp directories remained and it paused upon failing. Then I opened up a new terminal so I could run the exact commands it failed on (seemed like trying to pull in setuptools as part of argcomplete?), and it worked fine.

So, still not clear what exactly the issue is when doing that command as part of a dependency vs running it directly? What user does brew run as, and would that see $PIP_CERT?

[Jerry1144]

I opened up a new terminal

brew strips a lot of environment variables from the build shell. It was pretty much the reason it can build most things without much hassle. By using your own terminal session you defeated this filtering and allowed your environment variables through.

I guess you have to modify the formula yourself to add a $PIP_CERT variable during install, docs here. You don't have to do that every time brew update. If all you did was to add one line, git is usually clever enough to rebase your changes onto the newest formula.

[wce3]

I have the same issue which is driving me crazy. I'm trying to upgrade glib. For sure, the problem is with my ssl-intercepting proxy. I just can't find the right certificate path to add the proxy cert.

==> Upgrading glib
  -> 2.80.2
/usr/bin/env tar --extract --no-same-owner --file /Users/s5998/Library/Caches/Homebrew/downloads/4e472e59e4e9dace932583b9f83f2978c83a05b9bec6f012097e985afb2e74fa--glib-2.80.2.tar.xz --directory /private/tmp/homebrew-unpack20240603-59334-vv33hf
/usr/bin/env cp -pR /private/tmp/homebrew-unpack20240603-59334-vv33hf/glib-2.80.2/. /private/tmp/glib-20240603-59334-gvt4d0/glib-2.80.2
==> Patching
cp -p /Users/s5998/Library/Caches/Homebrew/downloads/3cff797fba2e962ee6871405f3f9c105575e25520c4eeadd25ebfe6c746d3f15--hardcoded-paths.diff /private/tmp/glib--patch-20240603-59334-uk3ib8/hardcoded-paths.diff
==> Applying hardcoded-paths.diff
patch -g 0 -f -p1 -i /private/tmp/glib--patch-20240603-59334-uk3ib8/hardcoded-paths.diff
patching file 'gio/xdgmime/xdgmime.c'
patching file 'glib/gutils.c'
/usr/bin/env tar --extract --no-same-owner --file /Users/s5998/Library/Caches/Homebrew/downloads/4500f418312d30e798642bced6da5165bf8d975b90f31bd626b7e189656f3acc--packaging-24.0.tar.gz --directory /private/tmp/homebrew-unpack20240603-59334-6a7api
/usr/bin/env cp -pR /private/tmp/homebrew-unpack20240603-59334-6a7api/packaging-24.0/. /private/tmp/glib--packaging-20240603-59334-5xzm77/packaging-24.0
==> python3.12 -m pip install --target /Users/s5998/homebrew/Cellar/glib/2.80.2/share/glib-2.0 --verbose --no-deps --no-binary=:all: --ignore-installed --no-compile .
Using pip 24.0 from /Users/s5998/homebrew/lib/python3.12/site-packages/pip (python 3.12)
Processing /private/tmp/glib--packaging-20240603-59334-5xzm77/packaging-24.0
  Installing build dependencies: started
  Running command pip subprocess to install build dependencies
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))': /simple/flit-core/
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))': /simple/flit-core/
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))': /simple/flit-core/
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))': /simple/flit-core/
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))': /simple/flit-core/
  Could not fetch URL https://pypi.org/simple/flit-core/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/flit-core/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))) - skipping
  ERROR: Could not find a version that satisfies the requirement flit_core>=3.3 (from versions: none)
  ERROR: No matching distribution found for flit_core>=3.3

• I added the certs to /private/etc/ssl/certs and hashed them. This satisfies python (below), but when homebrew runs pip install, it still doesn't seem to be using them.
• I also added them to homebrew/etc/openssl@3/certs, also without any luck.
• added them to homebrew/etc/ca-certificates/cert.pem and ran
  "HOMEBREW_FORCE_BREWED_CA_CERTIFICATES=1 brew upgrade -v glib": also didn't work.

I'm trying to modify the formula, but also no success yet.

$ python3 -c "import ssl; print(ssl.get_default_verify_paths())"
DefaultVerifyPaths(cafile='/private/etc/ssl/cert.pem', capath='/private/etc/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/private/etc/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/private/etc/ssl/certs')

[wce3]

The python3 above was not from homebrew. homebrew upgrade uses this one (where the certificates should also be installed):

$ python3.12 -c "import ssl; print(ssl.get_default_verify_paths())"
DefaultVerifyPaths(cafile='/Users/xxxxx/homebrew/etc/openssl@3/cert.pem', capath='/Users/xxxxx/homebrew/etc/openssl@3/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/Users/xxxxx/homebrew/etc/openssl@3/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/Users/xxxxx/homebrew/etc/openssl@3/certs')

[wce3]

as must have been the case, homebrew pip brings it's own cacerts. It would be good if one could get it to use the system cacerts or the openssl or the ca-certificates.....
For now I just patched it to get on with updating: homebrew/lib/python3.12/site-packages/pip/_vendor/certifi/cacert.pem