(Edited) Option to log context

[LudvigOlsen]

It could be nice to have an option for logging the target context before executing the call. E.g. to have easy access to the JobID when going through the log of a failed target, and being able to see the called command for debugging or rerunning interactively.

As previously mentioned, I have a pretty large workflow that takes quite a while to do anything with. If I just want to run a job interactively with all the paths and settings filled in by the workflow script, it would be nice to know the exact call (e.g. a python script with lots of paths, etc.).

I've come up with the following wrapper function, that gives me this information. Could something like this be useful to integrate in gwf? This version only works for the slurm backend currently, but it illustrates the idea:

def log_context(call: str):
    escaped_call = call.replace('"', r"\"").replace("'", r"\'").strip()
    return f"""
printf '%s\n' '---'
date
printf "JobID: $SLURM_JOB_ID \n"
printf "Command:\n{escaped_call}\n"
printf '%s' '---\n\n'
{call}
"""

gwf.target('job', inputs=[input_file], outputs=[...]) << log_context("""...""")

Here's an example .stdout output:

[LudvigOlsen]

I edited the issue to be about more general context logging instead of just the command.

[dansondergaard]

You can use gwf info <target name> to get the "finalized" target script with all variables etc. filled out, which you can then copy and run :-)

[LudvigOlsen]

Sure but it builds the graph to get me that information. No problem with a small workflow, big annoyance with a big workflow :-)

I don't mind adding log_context manually as in the example, but would be a great option to have as a flag in gwf.target().

[dansondergaard]

At the end of your workflow file, you can do this:

print(gwf.targets["MyTarget"].spec)

Then run your workflow file with python directly, so:

python workflow.py

[LudvigOlsen]

Will try that next time! I like having access to the slurm jobID in the log file though.

Anyway, it was just an idea for a possible improvement. I understand not wanting to clutter the interface. I'll use my context logger for the targets that make sense! :-)

[dansondergaard]

More introspection features are definitely a good idea, but there are many different needs and usability compromises. I'll give it some thought :-)