Project's test are failing - help to debug greatly appreciated

[consideRatio]

It appears that we have started to get failures in the CI system without actually making a change to the repository, so something changed elsewhere which caused these failures.

But what?

• Last successful run: https://github.com/dask/dask-gateway/actions/runs/4655309201/jobs/8237830130
• The failure following that: https://github.com/dask/dask-gateway/actions/runs/4863824399/jobs/8672055268

Difference in python test dependencies installed (python 3.10 test):

-alembic              1.10.4
+alembic              1.10.3
-attrs                23.1.0
+attrs                22.2.0
-cryptography         40.0.2
+cryptography         40.0.1
-dask                 2023.4.1
+dask                 2023.3.2
-distributed          2023.4.1
+distributed          2023.3.2.1
-grpcio               1.54.0
+grpcio               1.53.0
+HeapDict             1.0.1
-importlib-metadata   6.6.0
+importlib-metadata   6.3.0
-ipython              8.12.1
+ipython              8.12.0
-jupyter_client       8.2.0
+jupyter_client       8.1.0
-jupyterhub           4.0.0
+jupyterhub           3.1.1
-kubernetes-asyncio   24.2.3
+kubernetes-asyncio   24.2.2
-numpy                1.24.3
+numpy                1.24.2
-packaging            23.1
+packaging            23.0
-pandas               2.0.1
+pandas               2.0.0
-partd                1.4.0
+partd                1.3.0
-pip                  22.0.4
+pip                  23.0.1
-platformdirs         3.5.0
+platformdirs         3.2.0
-protobuf             4.22.3
-psutil               5.9.5
+protobuf             4.22.1
+psutil               5.9.4
-Pygments             2.15.1
+Pygments             2.14.0
-pytest               7.3.1
+pytest               7.3.0
-requests             2.29.0
-ruamel.yaml          0.17.22
+requests             2.28.2
+ruamel.yaml          0.17.21
-SQLAlchemy           2.0.12
+SQLAlchemy           2.0.9
-tornado              6.3.1
+tornado              6.2
-trustme              1.0.0
+trustme              0.9.0
-yarl                 1.9.2
+yarl                 1.8.2
-zict                 3.0.0
+zict                 2.2.0

Suspciuos difference in behavior, from setup.py focus (working)...

Obtaining file:///home/runner/work/dask-gateway/dask-gateway/dask-gateway (from -r tests/requirements.txt (line 28))
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Obtaining file:///home/runner/work/dask-gateway/dask-gateway/dask-gateway-server (from -r tests/requirements.txt (line 29))
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'

To a pyproject.toml focus (not working). but note that the python packages are put in dedicated folders, and they don't contain a pyproject.toml - the root directory does to provide config for black/isort etc.

Obtaining file:///home/runner/work/dask-gateway/dask-gateway/dask-gateway (from -r tests/requirements.txt (line 28))
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Checking if build backend supports build_editable: started
  Checking if build backend supports build_editable: finished with status 'done'
  Getting requirements to build editable: started
  Getting requirements to build editable: finished with status 'done'
  Preparing editable metadata (pyproject.toml): started
  Preparing editable metadata (pyproject.toml): finished with status 'done'
Obtaining file:///home/runner/work/dask-gateway/dask-gateway/dask-gateway-server (from -r tests/requirements.txt (line 29))
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Checking if build backend supports build_editable: started
  Checking if build backend supports build_editable: finished with status 'done'
  Getting requirements to build editable: started
  Getting requirements to build editable: finished with status 'done'
  Preparing editable metadata (pyproject.toml): started
  Preparing editable metadata (pyproject.toml): finished with status 'done'

[klucar]

The new package is "HeapDict", which was last updated 2019. Looks like dask/cachey pulls in HeapDict, not sure how dask/cachey is getting in here, it hasn't been touched since 2020 and only depends on HeapDict.

https://libraries.io/pypi/HeapDict
https://github.com/dask/cachey

[klucar]

dask/zict used to depend on it, maybe this is more promissing.

dask/zict@1e4f2b8

[klucar]

Well this all seems coincident with Dask dropping support for python 3.8.

Try making dask and distributed depend on >=2022.4.0,<2023.5.1 or just drop support for 3.8 here too

[consideRatio]

I've concluded that the issue is that the golang binary isn't being built in #739 where I added an explicit check for this.

I think its related to how we make it build through stuff going on in setup.py. I suspect a modern version of pip or setuptools etc made this no longer work.

Here is the changelog for 23.1 that was released april 15th. Last test succeeded april 10th, so it seems very relevant https://pip.pypa.io/en/stable/news/#v23-1