kubernetes-asyncio 22.6 breaks DaskExecutor with dask_kubernetes.

[mkarbo]

If anyone else is experiencing issues in prefect using dask + kubernetes with serviceaccounts, if you are experiencing issues today, it might be because of a new version of kubernetes-asyncio. See this issue for details.

---

After rebuilding images, I got the following error:

...
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/dask_kubernetes/core.py", line 474, in __init__
    super().__init__(**self.kwargs)
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/distributed/deploy/spec.py", line 260, in __init__
    self.sync(self._start)
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/distributed/utils.py", line 311, in sync
    return sync(
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/distributed/utils.py", line 378, in sync
    raise exc.with_traceback(tb)
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/distributed/utils.py", line 351, in f
    result = yield future
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/dask_kubernetes/core.py", line 603, in _start
    await super()._start()
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/distributed/deploy/spec.py", line 293, in _start
    self.scheduler = await self.scheduler
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/distributed/deploy/spec.py", line 59, in _
    await self.start()
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/dask_kubernetes/core.py", line 182, in start
    await super().start(**kwargs)
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/dask_kubernetes/core.py", line 87, in start
    raise e
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/dask_kubernetes/core.py", line 77, in start
    self._pod = await self.core_api.create_namespaced_pod(
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/kubernetes_asyncio/client/api_client.py", line 192, in __call_api
    raise e
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/kubernetes_asyncio/client/api_client.py", line 185, in __call_api
    response_data = await self.request(
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/kubernetes_asyncio/client/rest.py", line 223, in POST
    return (await self.request("POST", url,
  File "/pipelines/flow_test/.venv/lib/python3.9/site-packages/kubernetes_asyncio/client/rest.py", line 180, in request
    raise ApiException(http_resp=r)
kubernetes_asyncio.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: <...>
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot create resource \"pods\" in API group \"\" in the namespace \"prefect\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

To fix it, I added 'kubernetes-asyncio<22.6' as a constraint in my packages. I don't know where the error lies, whether in prefect, dask_kubernetes or in kubernetes-asyncio - just posting this for awareness.

[ddelange]

Looks like purely upstream ref dask/dask-kubernetes#419 / dask/dask-kubernetes#420 / tomplus/kubernetes_asyncio#187

[ddelange]

I think this issue can be closed as 22.6.1 has been released and fixes the issue

[zanieb]

Thanks!