HTCondor CI is failing

[guillaumeeb]

I've put some time in trying to debug HTCondor CI (see #562 (comment)).

If it did work on my laptop, I've not been able to make it works in github Actions, don't know why yet.

Some details:

• I've updated Docker images to run the correct Python version (same as other CI environments),
• I've updated docker-compose setup with a configuration on HTCondor negociator Interval, the time between to scheduling cycle, as the default of 60s made it possible tests fail because of that, or made tests really long.
• I've added some debugging docker commands to try to understand what's happening.

From the last run here, we can see from the Cleanup sections and condor_history command output that the HTCondor queuing system is working. The problem seems to come from the worker jobs which complete really fast, Dask workers never connect to Scheduler.

I guess we would need to see the jobs stdout/stderr to debug further the problem.

cc @riedel @mivade @jolange.

[jolange]

Thanks @guillaumeeb ! Yes, this looks like the worker jobs are failing directly.
The command is starting with a semicolon now:

 Arguments = "-c '; /opt/anaconda/bin/python -m distributed.cli.dask_worker tcp://172.18.0.4:33755 --nthreads 1 --memory-limit 100.00MiB --name HTCondorCluster-1 --nanny --death-timeout 60'"

I certainly broke this with my changes, I guess because env_extra does not stay None due to this line.
I'm not sure if this would cause this to fail, but I'll fix it first before looking further into this.

Otherwise, seeing stderr would help a lot, probably -- I remember that I had to do some manual downgrades of click for some python versions. But I tried a number of setups, so I don't remember really...

[guillaumeeb]

Closed by #570.