[WIP] Adding a nnpdf backend #57
Conversation
| # Fail without piety | ||
| raise ValueError("There are replicas in the range considered") | ||
|
|
||
| def include_arguments(self, current_arguments): |
There was a problem hiding this comment.
To check I understand - if you ever wanted to run NNPDF with Arc or DIRAC, would your include_arguments lead to command-line arguments for your executable like --memsize X?
@marianheil and I were discussing that supporting arbitrary arguments across the board might be quite useful, especially if more programs are going to be added to pyHepGrid, so we don't have to add all programs' variables to all users' headers. But it looks like they'd be ignored for SLURM, unless specifically added as a {key} to the .sh template, but for Arc/Dirac they'd be parsed and passed as additional --X Y command-line flags.
Ideally, for those programs we intend to be able to run on all the Backends, each runcard should lead to the same command-line arguments being passed regardless of which Backend it's being submitted to.
Any idea how we could both manage that, and arbitrary arguments? (Probably related to #17 )
There was a problem hiding this comment.
To check I understand - if you ever wanted to run NNPDF with Arc or DIRAC, would your
include_argumentslead to command-line arguments for your executable like--memsize X?
Yes. Although it will fail because I'm trusting in the existence of some arguments that are there only for slurm, but yes, that's the idea.
@marianheil and I were discussing that supporting arbitrary arguments across the board might be quite useful, especially if more programs are going to be added to
pyHepGrid, so we don't have to add all programs' variables to all users' headers. But it looks like they'd be ignored for SLURM, unless specifically added as a{key}to the.shtemplate, but for Arc/Dirac they'd be parsed and passed as additional--X Ycommand-line flags.
There are a few things that are going to be always backend dependent. In principle it might be possible to do it more general but disentangling all of that would be quite a task. I've tried in the past and got bored before achieving anything :__
In any case, this is particularly true for slurm. I remember we had some problems when giving options through command line arguments so we had to have them hard-coded on the scripts (which lead to having to have several different scripts). Don't remember whether this was on the IPPP batch.
Any idea how we could both manage that, and arbitrary arguments? (Probably related to #17 )
I'd say #58 is not a bad compromise: each of the programs must decide/know how to treat the arguments.
add a include_arguments method to the program interface
Allow loading custom runmode without modifying `pyHepGrid` Code
This should allow submission/testing from any folder
|
What is the status on here? Is this ready to merge? This MR includes a few extra features which would be nice to have in master. |
|
Ah, feel free to merge. This is in working condition. |
| util.spCall(["./{0}".format(header.runfile)] + nnlojob_args) | ||
| runfile = os.path.basename(header.runfile) | ||
| util.spCall(["chmod","+x",runfile]) | ||
| util.spCall(["./{0}".format(runfile)] + nnlojob_args) |
There was a problem hiding this comment.
Question: what happens if header.runfile is not in the local directory? (or is it always?)
There was a problem hiding this comment.
This is exactly what should changed. Before to run test mode you had to have the runfile in your current directory. Now you can also specify an absolute path. In the lines you highlighted we only get the file name the copy happens in:
pyHepGrid/src/pyHepGrid/src/test_nnlojob.py
Lines 15 to 16 in 9fe0cf2
For the actual submission this already worked before, since we just pass header.runfile to Arc/Dirac/Slurm and they can handle absolute and relative paths.
There was a problem hiding this comment.
So does the os.chdir in setup() persist after it exits?
There was a problem hiding this comment.
What do you mean with "persist"? Within python yes, but it doesn't change your "main shell" from which you are running pyHepGrid. Nothing change in this regard. We also don't have to change back in pyHepGrid after run_test, since pyHepGrid finishes afterwards.
There was a problem hiding this comment.
Yes, within python - I assumed it would basically work like a local variable in the setup scope, but it looks like it does persist.
Nowadays I'm only using slurm but I am using it in several different systems so might as well add it as a public backend (even if I think I will be the only one using it for now :__)
Todo: