Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
This implements HTCondorCluster, as requested in #100. This allows using HTCondor as a resource manager, provided you have a shared file system.
In order to minimize code, I ended up using the command-line tools instead of the Python bindings and only submitting one job at a time (i.e. run
Some differences usage-wise from the other resource managers:
I tested this with Python 2.7 and Python 3.6, but only on a single machine. Looks like some of the other resource managers use docker-compose to set up test clusters so I'll try to get something like that going for HTCondor too.
guillaumeeb left a comment
Many many thanks @matyasselmeci! This is great to finally have this!
HTCondor looks really specific though...
I like that! From what I understand, this makes HTCondor implementation closer to others.
Yep, but obviously this can wait for another PR!
I've a few comments/suggestions/questions on the changes, but overall this looks really good! I've a small concern on overriding
For the test failure, apparently you've got a flake8 check error.
Thanks @matyasselmeci for the work, this looks quite good to me.
It would also be good if somebody actually using HTCondor could give this a try!
If nobody shows up in the following days, I'm still happy to merge this as a first implementation for HTCondor, and we'll get feedback later on as people find time to play with it!
jhamman left a comment
I took a quick pass through this PR. Thanks @matyasselmeci for pulling it together. I've not personally used HTCondor before so I'm just going to take you at your word on the specifics of the job script implementation. I'd like to see this go it and let people try it out.
Of course, at some point it would be nice to get a docker test environment for this cluster setup but I think that come come later.