kubernetes_asyncio major version release breaks: NO_PROXY env var not honoured

[michcio1234]

What happened: dask-kubernetes cannot connect to K8s API when HTTPS proxy is used in the system.

I have HTTPS_PROXY, https_proxy, HTTP_PROXY and http_proxy environment variables set. They point to Squid proxy.

I also have NO_PROXY env var set, which contains 172.20.0.0/16.

This configuration used to work - dask-kubernetes was able to communicate with K8s API, not going through proxy. Now it tries to go through proxy. This fails, because proxy is outside of the cluster (IIUC, dask-kubernetes with the default in-cluster config connects to K8s API via internal endpoint, not through public internet). I am getting this traceback:

  (...)
    cluster = KubeCluster(*args, **kwargs)                                                                                                                                                                                                    
  File "/opt/conda/lib/python3.7/site-packages/dask_kubernetes/core.py", line 466, in __init__                                                                                                                                                
    super().__init__(**self.kwargs)                                                                                                                                                                                                           
  File "/opt/conda/lib/python3.7/site-packages/distributed/deploy/spec.py", line 277, in __init__                                                                                                                                             
    self.sync(self._start)                                                                                                                                                                                                                    
  File "/opt/conda/lib/python3.7/site-packages/distributed/deploy/cluster.py", line 185, in sync                                                                                                                                              
    return sync(self.loop, func, *args, **kwargs)                                                                                                                                                                                             
  File "/opt/conda/lib/python3.7/site-packages/distributed/utils.py", line 326, in sync                                                                                                                                                       
    raise exc.with_traceback(tb)                                                                                                                                                                                                              
  File "/opt/conda/lib/python3.7/site-packages/distributed/utils.py", line 309, in f                                                                                                                                                          
    result[0] = yield future                                                                                                                                                                                                                  
  File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 762, in run                                                                                                                                                              
    value = future.result()                                                                                                                                                                                                                   
  File "/opt/conda/lib/python3.7/site-packages/dask_kubernetes/core.py", line 595, in _start                                                                                                                                                  
    await super()._start()                                                                                                                                                                                                                    
  File "/opt/conda/lib/python3.7/site-packages/distributed/deploy/spec.py", line 306, in _start                                                                                                                                               
    self.scheduler = await self.scheduler                                                                                                                                                                                                     
  File "/opt/conda/lib/python3.7/site-packages/distributed/deploy/spec.py", line 66, in _                                                                                                                                                     
    await self.start()                                                                                                                                                                                                                        
  File "/opt/conda/lib/python3.7/site-packages/dask_kubernetes/core.py", line 182, in start                                                                                                                                                   
    await super().start(**kwargs)                                                                                                                                                                                                             
  File "/opt/conda/lib/python3.7/site-packages/dask_kubernetes/core.py", line 78, in start                                                                                                                                                    
    self.namespace, self.pod_template                                                                                                                                                                                                         
  File "/opt/conda/lib/python3.7/site-packages/kubernetes_asyncio/client/api_client.py", line 189, in __call_api                                                                                                                              
    _request_timeout=_request_timeout)                                                                                                                                                                                                        
  File "/opt/conda/lib/python3.7/site-packages/kubernetes_asyncio/client/rest.py", line 229, in POST                                                                                                                                          
    body=body))                                                                                                                                                                                                                               
  File "/opt/conda/lib/python3.7/site-packages/kubernetes_asyncio/client/rest.py", line 170, in request                                                                                                                                       
    r = await self.pool_manager.request(**args)                                                                                                                                                                                               
  File "/opt/conda/lib/python3.7/site-packages/aiohttp/client.py", line 536, in _request                                                                                                                                                      
    req, traces=traces, timeout=real_timeout                                                                                                                                                                                                  
  File "/opt/conda/lib/python3.7/site-packages/aiohttp/connector.py", line 542, in connect                                                                                                                                                    
    proto = await self._create_connection(req, traces, timeout)                                                                                                                                                                               
  File "/opt/conda/lib/python3.7/site-packages/aiohttp/connector.py", line 905, in _create_connection                                                                                                                                         
    _, proto = await self._create_proxy_connection(req, traces, timeout)                                                                                                                                                                      
  File "/opt/conda/lib/python3.7/site-packages/aiohttp/connector.py", line 1293, in _create_proxy_connection                                                                                                                                  
    headers=resp.headers,                                                                                                                                                                                                                     
aiohttp.client_exceptions.ClientHttpProxyError: 503, message='Service Unavailable', url=URL('<MY PROXY ADDRESS HERE>')

This seems to be related to recent update of kubernetes_asyncio to major version 22. They state in the changelog:

feat(python-asyncio): add support for proxy config using system env vars

What you expected to happen: NO_PROXY env var should be honoured. Connection should not go through proxy.

Minimal Complete Verifiable Example:
I can come up with it later.

Anything else we need to know?:

Environment:

• dask: 2021.7.2
• dask-core: 2021.7.2
• dask-kubernetes: 2021.3.1
• Python version: 3.7.4
• Operating System: Alpine linux inside Docker
• Install method (conda, pip, source): dask-kubernetes installed via pip, dask installed via conda

[jacobtomlinson]

Thanks for raising this. If we pinned kubernetes_asyncio to <22 would that solve this for you?

[michcio1234]

Yes. In my deployment I pinned it to 21.7.* (which is the last release before 22) and it solved the problem. Please notice though that I am using 2021.3.1 version of dask-kubernetes (didn't try updating yet).

[washcycle]

Just came here to post this issue.

I'm on AKS.

Workaround was downgrading the kubernetes_asyncio after dask is installed.

pip install kubernetes_asyncio==21.7.0

I have tons of examples.

this was the error I got

kubernetes_asyncio.client.exceptions.ApiException: (401)
Reason: Unauthorized
HTTP response headers: <CIMultiDictProxy('Audit-Id': 'bffd2fbf-82ac-44db-81dd-48682b60d21f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 22 Mar 2022 13:50:29 GMT', 'Content-Length': '129')>
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

[jacobtomlinson]

Could you try kubernetes_asyncio==22.6.1. They have made a few fixes upstream.

[washcycle]

Could you try kubernetes_asyncio==22.6.1. They have made a few fixes upstream.

I'll do it when I get to work this AM.

• Test kubernetes_asyncio==22.6.1

UPDATE:

I confirmed that latest version works!

[jacobtomlinson]

This project no longer uses kubernetes_asyncio so this can be closed.