Kubernetes job submission support and contributions

[pllopis]

Hi,

In the SRCNet SKA project we’re looking into using ReFrame for some benchmarking work. However, there’s a few things that we would need to add to ReFrame to achieve proper integration with our systems, and this is ticket is mostly to reach out and check if there’s a way we can align and make this contribution such that it would be accepted upstream, and to avoid duplicate work.

The first thing we’re interested in adding would be support for new batch systems. We're starting with Kubernetes (although down the line we’d add support for our bespoke SRCNet Execution APIs which are still in development).
Especially for Kubernetes, given its popularity, I wanted to make sure there isn’t ongoing effort already (in which case we’d be interested to test if it’s ready, or contribute to this effort).

And given the very different semantics of a traditional batch system (script based) vs Kubernetes (containers, yaml templates, PVCs,..) I wanted to know if you already have an opinion on how to fit Kubernetes into the ReFrame model/abstractions.
Do you have any existing opinions on how whether and how to add Kubernetes support into ReFrame? Or any existing effort?

For instance we found this fork: https://github.com/BigBalloon8/reframe/tree/k8s which adds kubernetes support but don’t know if you’re aware of this and if this has been tried to merge upstream. Would you consider this fork to be an acceptable contribution, or do you foresee that it would have to be done in a different way (in which case we would be interested to carry out part of the work).

Thanks!

[vkarak]

Hi,

There is no work for a Kubernetes backend under way right now. We would consider accepting a contribution if it is well founded and proves helpful to users. We are aware of the work from the EPCC folks (I think this is the paper: https://arxiv.org/pdf/2404.10536v2), but we haven't had an upstream request and we haven't looked into the details.

Another thing that might be not so obvious is that since ReFrame 4.4 you can create custom scheduler backends and register them to the framework without the need to fork and modify ReFrame. You can import your scheduler backend in the configuration file and use it, very similar to how custom parallel launchers can be integrated. We plan to write a tutorial on how to add custom scheduler backends, too.

I'd say that, as a first step, try to write your backend as a separate module and we'd be happy to provide help and feedback. We could then consider a possible upstream integration.

Cheers.

[pllopis]

Thanks for the reply!
I guess you were referring to these docs for implementing custom launchers?
That sounds like a great way to start indeed. We mostly wanted to avoid ending up with a ReFrame fork that's hard to maintain.

We would start by trying existing code in that other fork, and then evaluating if that works for our use cases. Thanks for the feedback.