[BUG] - Dask environment not working due to recent package release

[viniciusdc]

OS system and architecture in which you are running QHub

Linux

Expected behavior

When executing the simple dask-gateway pipeline we should be able to susceffuly launch a dask scheduler pod and a cluster instance.Succesful

Actual behavior

Executing the cluster = gateway.new_cluster() block from the above docs, leads to an unexpected error during spawning of the Dask scheduler. Upon further inspection of the scheduler logs, we can see the same error message as shown in this issue.

Even though we currently pin the necessary version so dask within qhub-dask meta-package, we still need to add a new exception to avoid the new version of click which was the root cause of the problem.

Possible Solution

Add the following to the dask environment upon initialization

 - click ~= 8.1.0 # Or downgrade and pin to 8.0.2

How to Reproduce the problem?

In the description

Command output

No response

Versions and dependencies used.

Latest qhub on main, but the environment build is independent of cli version

Compute environment

No response

Integrations

No response

Anything else?

No response

[iameskild]

Until the issue is resolved upstream, we can pin click in our qhub-dask meta package.
conda-forge/qhub-dask-feedstock#11

[viniciusdc]

Until the issue is resolved upstream, we can pin click in our qhub-dask meta package. conda-forge/qhub-dask-feedstock#11

Just I quick commented that I only spotted now, we didn't update qhub-dask in the dask-env config to 0.4.0 when we released: https://github.com/Quansight/qhub/blob/56ffa7b8607acd8172b9fb1025bbfeedb80ec6c4/qhub/initialize.py#L214, should this go to post?

[iameskild]

That's a good catch! I would say say yes since it's a minor change that should've been caught earlier and will ensure that click related issue is actually addressed.