certbot-dns-google does not honor no_proxy environment variable

[MaT1g3R]

My operating system is (include version):

Ubuntu 18.04

I installed Certbot with (certbot-auto, OS package manager, pip, etc):

pip

I ran this command and it produced this output:

We have a squid proxy running at our network edge; the local computer has a service account with permissions to manage DNS. The squid computer does not.

export no_proxy=metadata.google.internal
export http_proxy=http://squid:3128
export https_proxy=http://squid:3128
certbot certonly  -d $DOMAIN  --no-eff-email --config-dir ./conf --work-dir ./work --logs-dir ./logs --dns-google -m $EMAIL --agree-tos  --dns-google-propagation-seconds 120

2019-07-23 16:08:39,839:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
File "/home/ubuntu/venv/lib/python3.6/site-packages/certbot/auth_handler.py", line 69, in handle_authorizations
resps = self.auth.perform(achalls)
File "/home/ubuntu/venv/lib/python3.6/site-packages/certbot/plugins/dns_common.py", line 48, in perform
self._setup_credentials()
File "/home/ubuntu/venv/lib/python3.6/site-packages/certbot_dns_google/dns_google.py", line 57, in _setup_credentials
_GoogleClient.get_project_id()
File "/home/ubuntu/venv/lib/python3.6/site-packages/certbot_dns_google/dns_google.py", line 301, in get_project_id
r, content = http.request(url, headers=METADATA_HEADERS)
File "/home/ubuntu/venv/lib/python3.6/site-packages/httplib2/__init__.py", line 1953, in request
cachekey,
File "/home/ubuntu/venv/lib/python3.6/site-packages/httplib2/__init__.py", line 1618, in _request
conn, request_uri, method, body, headers
File "/home/ubuntu/venv/lib/python3.6/site-packages/httplib2/__init__.py", line 1524, in _conn_request
conn.connect()
File "/home/ubuntu/venv/lib/python3.6/site-packages/httplib2/__init__.py", line 1194, in connect
self.sock.connect((self.host, self.port) + sa[2:])
File "/home/ubuntu/venv/lib/python3.6/site-packages/httplib2/socks.py", line 502, in connect
self.__negotiatehttp(destpair[0], destpair[1])
File "/home/ubuntu/venv/lib/python3.6/site-packages/httplib2/socks.py", line 463, in __negotiatehttp
raise HTTPError((statuscode, statusline[2]))
httplib2.socks.HTTPError: (403, b'Forbidden')

Certbot's behavior differed from what I expected because:

We expect the no_proxy env var is honored.
This is an issue with httplib2, we've already sent a PR to them here: httplib2/httplib2#140

We suggest switching to the request library instead, since it seems like the library is better maintained.

[adferrand]

Hello @MaT1g3R, and thanks a lot for your report.

Indeed the fix seems quite straight-forward from the certbot codebase. However, I doubt it will fix the issue, since googleapiclient third-party module, still rely on http2lib, and so no_proxy will still not be honored on the other calls to the Google API.

Instead of modifying the code, we could wait for the new http2lib to be released, since I see the PR you created for it is in its final step to be merged. At this point, we can require a sufficiently up-to-date version of http2lib in certbot-dns-google/setup.py.

What do you think of this?

[MaT1g3R]

Sure that seems reasonable.

Note the line that failed seems to be this line:

certbot/certbot-dns-google/certbot_dns_google/dns_google.py

Line 301 in 03cf5d1

r, content = http.request(url, headers=METADATA_HEADERS)

Which does not use googleapiclient and uses httplib2 directly.

FWIW, I filed an issue upstream here: googleapis/google-api-python-client#726