New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix command template for empty env_extra
in HTCondor
#570
Conversation
Due to config read with `default=[]`, `env_extra` will not stay `None` but become an empty list. This resulted in a command template starting with a semicolon. Possibly related to dask#568
Now I am confused:
|
This is really shaky: after adding debug output (f03f257), the CI for HTCondor was successfull, but failed again later. Now, this is an example stderr of a worker (from this run):
@guillaumeeb Is the problem simply |
@jolange I think you're on something!! It looks like 100MiB is not enough for running a Dask Worker! However, I believe the default Condor setup is only 1GB available on each condor worker node, so you should use a number lower than that. Maybe try with 500MiB to be safe? |
Ah, thanks, I was just trying to find out what the available memory could be. With 2GiB the job did not start to run, so that seemed too much ;-) I'm trying with 500GiB now. |
With 500GiB it worked without the warning in stderr and I also had a successful CI run for HTCondor. |
Just tried a complement fix on your branch, hope it's okay. The second test was probably fragile too because it also used only 100MiB for worker jobs. If that test fails and workers are note cleaned up, other test will fail. |
Okay, HTCondor CI is green, nice 👏. Thanks a lot @jolange! I will just make another commit here to re-add some of the debug tricks you used, it could be nice later on to have worker logs again! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All is green, outputs is detailed, again, thanks a lot for the work here @jolange!
Nice, thanks! |
Due to the config read with
default=[]
,env_extra
will not stayNone
but become an empty list. This resulted in a command template starting with a semicolon.By first merging
env_extra
and_command_template
to a single list, this is avoided.Possibly related to #568, cc @guillaumeeb