Enable pilot job submission for HTCondor service

[rafaelfsilva]

• Implementation of pilot job support
• Update documentation
• Implement tests

[henricasanova]

While trying a test for this, I become a bit confused. It looks like the HT service will simply start jobs on the next available sub-service (e.g., a BareMetalComputeService). But what happens is that a PilotJob would simply be sent to the next service. And if that service doesn't support pilot jobs, then there is an "doesn' t support pilot job". So I am not sure what it means for HTCondor to support pilot jobs actually.. The idea is that it simply relays pilot job requests to one of the subservices? But then it should filter out those services that do not support pilot jobs (and if none does, then too bad, failure). And then, when multiple do, how do we pick?

[rafaelfsilva]

I see your confusion. Actually, HTCondor does not make difference between pilot jobs or regular jobs. However, since our implementation makes this clear difference (and callbacks are handled differently). HTCondor matches the first available service to submit the job (assuming all requirements are met), thus no smart decision is made here. If someone wants a very specific dataset, they need to pass it as part of a requirement. Does this make sense to you?

[henricasanova]

Ok, so I'll just change the code in HTCondor so that if a pilot job is submitted, those sub-services that do not support pilot jobs are not considered (and if none does, then some error). Not sure what you mean by "dataset"?

[rafaelfsilva]

yes, sounds good. "dataset" is most probably a typo ;) I believe I meant to type "behavior"

[henricasanova]

I have improved some of the implementation and error/catching based on the above. I am closing this issue and opening a new one.