New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement SLURM support #38

Open
samuell opened this Issue Jan 26, 2017 · 9 comments

Comments

Projects
None yet
2 participants
@samuell
Member

samuell commented Jan 26, 2017

No description provided.

@valentin-krasontovitsch

This comment has been minimized.

Show comment
Hide comment
@valentin-krasontovitsch

valentin-krasontovitsch Jan 15, 2018

Considering moving away from ruffus written in python to a golang pipeline framework, and slurm is what we use for batching... is there any prognosis on this? a roadmap? what would have to be done - are you accepting merge requests?

Considering moving away from ruffus written in python to a golang pipeline framework, and slurm is what we use for batching... is there any prognosis on this? a roadmap? what would have to be done - are you accepting merge requests?

@valentin-krasontovitsch

This comment has been minimized.

Show comment
Hide comment
@valentin-krasontovitsch

valentin-krasontovitsch Jan 15, 2018

Wait, it's already perfectly possible to use slurm just by using a go slurm library and writing go code directly in the workflow, correct? What is this issue about, precisely? I'm confused ^^

Wait, it's already perfectly possible to use slurm just by using a go slurm library and writing go code directly in the workflow, correct? What is this issue about, precisely? I'm confused ^^

@samuell

This comment has been minimized.

Show comment
Hide comment
@samuell

samuell Jan 16, 2018

Member

Hi @valentin-krasontovitsch , thanks for checking out scipipe.

What we are doing right now, while pondering the best way to add proper SLURM support, is to just use the prepend feature, and add the SLURM salloc command ther (as shown in http://scipipe.org/howtos/hpc/). It actually works very robustly, but it would be sure great to have something a tad bit more integrated.

I will check the go SLURM library (wasn't aware of it yet :) ), but I'm also leaning towards keeping everything as simple as possible, fighting a bit to avoid dependencies etc ... as I think this will help make scipipe easier to maintain in the long run, with limited academic funding :P

Member

samuell commented Jan 16, 2018

Hi @valentin-krasontovitsch , thanks for checking out scipipe.

What we are doing right now, while pondering the best way to add proper SLURM support, is to just use the prepend feature, and add the SLURM salloc command ther (as shown in http://scipipe.org/howtos/hpc/). It actually works very robustly, but it would be sure great to have something a tad bit more integrated.

I will check the go SLURM library (wasn't aware of it yet :) ), but I'm also leaning towards keeping everything as simple as possible, fighting a bit to avoid dependencies etc ... as I think this will help make scipipe easier to maintain in the long run, with limited academic funding :P

@samuell

This comment has been minimized.

Show comment
Hide comment
@samuell

samuell Jan 16, 2018

Member

Ah, but yes, you should be able to call Go/SLURM library code from within your custom Go-functions, for sure, yes!

Member

samuell commented Jan 16, 2018

Ah, but yes, you should be able to call Go/SLURM library code from within your custom Go-functions, for sure, yes!

@samuell

This comment has been minimized.

Show comment
Hide comment
@samuell

samuell Jan 16, 2018

Member

Otherwise, I'm right now looking hard into introducing a bit more of modularity / pluggability in the code, so that it would be easier for anyone to add custom "runners", according to some interface, without needing to rewrite the core. Hope to get this fixed before 1.0 in a few months.

Member

samuell commented Jan 16, 2018

Otherwise, I'm right now looking hard into introducing a bit more of modularity / pluggability in the code, so that it would be easier for anyone to add custom "runners", according to some interface, without needing to rewrite the core. Hope to get this fixed before 1.0 in a few months.

@valentin-krasontovitsch

This comment has been minimized.

Show comment
Hide comment
@valentin-krasontovitsch

valentin-krasontovitsch Jan 18, 2018

I'm not sure about the slurm library myself, have only looked at it shortly, but it seems like a mature project (just judging by the fact that they seem to have a version 1 and a version 2 ^^ ) - always hard to weigh wether you want to introduce a dependency and maintain that, or write it yourself and maintain that...

but I'm just wondering what kind of features your thinking of that would make slurm more "integrated"? Or more abstractly, any way of running commands / any runner? have you already compiled a (mental) list?

cause at the moment, as you said as well, everybody's just free to use whatever library they want in custom go functions, right? what is that lacking in "integration" compared to the "normal" way of running a command?

I'm not sure about the slurm library myself, have only looked at it shortly, but it seems like a mature project (just judging by the fact that they seem to have a version 1 and a version 2 ^^ ) - always hard to weigh wether you want to introduce a dependency and maintain that, or write it yourself and maintain that...

but I'm just wondering what kind of features your thinking of that would make slurm more "integrated"? Or more abstractly, any way of running commands / any runner? have you already compiled a (mental) list?

cause at the moment, as you said as well, everybody's just free to use whatever library they want in custom go functions, right? what is that lacking in "integration" compared to the "normal" way of running a command?

@samuell

This comment has been minimized.

Show comment
Hide comment
@samuell

samuell Jan 23, 2018

Member

@valentin-krasontovitsch

I'm not sure about the slurm library myself, have only looked at it shortly, but it seems like a mature project (just judging by the fact that they seem to have a version 1 and a version 2 ^^ ) - always hard to weigh wether you want to introduce a dependency and maintain that, or write it yourself and maintain that...

Indeed.

but I'm just wondering what kind of features your thinking of that would make slurm more "integrated"? Or more abstractly, any way of running commands / any runner? have you already compiled a (mental) list?

Having data stored in a bit more structured way would help with things like structured logging.

It might also make it easier to provide e.g. a little smarter ways to calculate good "guesstimates" for the time limits to put on the SLURM job, for example based on input data size multiplied by some factor, that might be either manually set or mined from logging data.

(Providing good max running times can in our experience be a bit of a manual hassle ... you don't want to put too long times so your jobs have problems getting started quickly enough, and not too short, so that jobs will crash before finishing ... and this while running times can vary wildly depending on input datasize, etc).

cause at the moment, as you said as well, everybody's just free to use whatever library they want in custom go functions, right? what is that lacking in "integration" compared to the "normal" way of running a command?

That's a very good point though! The custom Go function feature might be exactly the right place allow custom runners. Perhaps just putting the runner in a struct with a few metadata fields would be enough to provide a good generic solution.

Member

samuell commented Jan 23, 2018

@valentin-krasontovitsch

I'm not sure about the slurm library myself, have only looked at it shortly, but it seems like a mature project (just judging by the fact that they seem to have a version 1 and a version 2 ^^ ) - always hard to weigh wether you want to introduce a dependency and maintain that, or write it yourself and maintain that...

Indeed.

but I'm just wondering what kind of features your thinking of that would make slurm more "integrated"? Or more abstractly, any way of running commands / any runner? have you already compiled a (mental) list?

Having data stored in a bit more structured way would help with things like structured logging.

It might also make it easier to provide e.g. a little smarter ways to calculate good "guesstimates" for the time limits to put on the SLURM job, for example based on input data size multiplied by some factor, that might be either manually set or mined from logging data.

(Providing good max running times can in our experience be a bit of a manual hassle ... you don't want to put too long times so your jobs have problems getting started quickly enough, and not too short, so that jobs will crash before finishing ... and this while running times can vary wildly depending on input datasize, etc).

cause at the moment, as you said as well, everybody's just free to use whatever library they want in custom go functions, right? what is that lacking in "integration" compared to the "normal" way of running a command?

That's a very good point though! The custom Go function feature might be exactly the right place allow custom runners. Perhaps just putting the runner in a struct with a few metadata fields would be enough to provide a good generic solution.

@samuell

This comment has been minimized.

Show comment
Hide comment
@samuell

samuell Feb 2, 2018

Member

Just wanted to report that I think I have now seen the light regarding how to implement the required pluggability. It goes quite along the lines of your way of just using the customExexute method, @valentin-krasontovitsch (Thanks much for pushing me in the right direction!), but I also realized that we can make this in an even more documented and structured way, by allowing to plug in your custom "Executor".

This "Executor" will just be a Go interface of type:

type Executor interface {
    Execute(t *scipipe.Task)
}

... possibly with any other minor methods.

Then, one could have just create different executors, using structs, with their respective factory methods ... so one could just go, in the workflow:

foo := wf.NewProc("foo", "echo foo > {o:out}")
foo.Executor = NewSlurmExecutor("b2013222", "core", "2", "1:00:00", "MyFooJob")

... which would be implemented along the lines of (mostly notes for myself):

type SlurmExecutor struct {
    account string
    partition string
    numCores int
    runTime string
    jobName string
}

func (e *SlurmExecutor) Execute(t *scipipe.Task) {
    slurmCmd := fmt.Sprintf("salloc -A %s -p %s -n %s -t %s -J %s", e.account, e.partition, e.numCores, e.runTime, e.jobName)
    command := slurmCmd + " " + t.Command
    // Execute command
}

func NewSlurmExecutor(account string, partition string ...) *SlurmExecutor {
    executor := &SlurmExecutor{
        account: account,
        // ... etc etc ...
    }
    return executor
}

Will have a look at this shortly.

Member

samuell commented Feb 2, 2018

Just wanted to report that I think I have now seen the light regarding how to implement the required pluggability. It goes quite along the lines of your way of just using the customExexute method, @valentin-krasontovitsch (Thanks much for pushing me in the right direction!), but I also realized that we can make this in an even more documented and structured way, by allowing to plug in your custom "Executor".

This "Executor" will just be a Go interface of type:

type Executor interface {
    Execute(t *scipipe.Task)
}

... possibly with any other minor methods.

Then, one could have just create different executors, using structs, with their respective factory methods ... so one could just go, in the workflow:

foo := wf.NewProc("foo", "echo foo > {o:out}")
foo.Executor = NewSlurmExecutor("b2013222", "core", "2", "1:00:00", "MyFooJob")

... which would be implemented along the lines of (mostly notes for myself):

type SlurmExecutor struct {
    account string
    partition string
    numCores int
    runTime string
    jobName string
}

func (e *SlurmExecutor) Execute(t *scipipe.Task) {
    slurmCmd := fmt.Sprintf("salloc -A %s -p %s -n %s -t %s -J %s", e.account, e.partition, e.numCores, e.runTime, e.jobName)
    command := slurmCmd + " " + t.Command
    // Execute command
}

func NewSlurmExecutor(account string, partition string ...) *SlurmExecutor {
    executor := &SlurmExecutor{
        account: account,
        // ... etc etc ...
    }
    return executor
}

Will have a look at this shortly.

@valentin-krasontovitsch

This comment has been minimized.

Show comment
Hide comment
@valentin-krasontovitsch

valentin-krasontovitsch Apr 12, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment