Diagnosing NODE_FAILURE when using NextFlow on Cheaha

[wwarriner]

What would you like to see added?

If the following conditions are true, then consider that one or more NextFlow tasks may have insufficient memory allocated. Assume $jobid is the Slurm Job ID for the relevant NextFlow task.

• sacct -j $jobid -X -o jobid,state shows NODE_FAILURE
• The file .exitcode does not exist in the NextFlow task's working directory. Working directory here refers to the NextFlow concept.
• No other helpful messages showing a more detailed description of the cause of the error, a more specific error name, or an exit code.
• Tasks fail intermittently (not necessary, but increases likelihood)

When we have encountered researcher workflows where the above are true, the cause of the error has invariably been due to an "Out of Memory" (OOM) event.