Add compact custom-made Nextflow error report #19

Cecilia-Sensalari · 2021-06-28T16:51:53Z

There is now a compact "error box" appearing at the end of the errorReport screen when the Nextflow pipeline crashes. It is implemented in workflow.onError in main.nf. It covers two cases:

In case the stopped process printed some ERROR lines* in its log file output, the error box tells the user 1) the name of the stopped process, 2) the error message lines and 3) the path to the log file where to find the complete output of the process
In case the stopped process didn't generate any log file or didn't print any ERROR lines* in its log file, the error is probably due to an external cause (e.g. comunication with the cluster) and the error box tells the user 1) the name of the stopped process, 2) the "Caused by:" line from Nextflow errorReport, 3) the errorMessage (if any) and 4) it points to the complete error report and to .nextflow.log

*are considered ERROR lines those that contain the "ERROR" word when parsed.

- There is now a customized error report when running the Nextflow pipeline and an error is encountered - After the default and long errorReport, it is also now shown a small error report through workflow.onError. - workflow.onError tells the user 1) the stopped process, 2) the error message lines and 3) the path to the log folder where to find the complete output of the process - Note: the shown error lines are those that contain the "ERROR" word when parsed.

- In case the log file of the stopped process doesn't exist or doesn't contain any ERROR line, catch the error from "Caused by:" errorReport lines and from errorMessage.

- If the log file exists AND if it contains ERROR lines, print the error lines - If any of the two conditions is not true, print the "Caused by:" line and the errorMessage

- The error box points to the process log file, which contains the complete output. However, when the process is "wgdOrthologs", workflow.onError doesn't know the name of the process log file because it depends on which two species were involved. - Therefore the code now obtaines the two names from parsing the errorMessage output lines and uses them to infer the filename, e.g. "wgd_orthologs_oryza_asparagus.log". - WIP: is it always possible to parse the names from the errorMessage lines?

Catches three cases: 1) process log file exists and contains ERROR lines: print lines and point to log file 2) process log file exists but no ERROR lines: the error was unexpected and just point to log file 3) process log file doesn't exist: print the errorReport "Caused by" line and the errorMessage, if any - To obtain the log filename of wgdOrtholog, parse the errorMessage, but only if it is not "null". If it is "null", the error cause is probably external to the pipeline itself (e.g. mis- communication with the cluster). In this case, the code follows case 3 above because the log filename is unknown.

lohausr

See comments and proposed changes.

main.nf

doc/source/faqs.rst

main.nf

Co-authored-by: lohausr <56632794+lohausr@users.noreply.github.com>

- log_names dictionary is moved at the beginning of the Nextflow file. - Every log file name throughout the code is replaced by calling the dictionary - The wgdOrtholog's filename depends on the current species pair, so only the fixed part is kept in the dictionary.

- The behaviour of "log.error" in workflow.onError() has changed and now it add extra newlines... - To have back the normal error box, all log.error lines are removed and the text is accumulated in a long string, that is printed at the end. - Some variable renaming

- For each process name (available in dictionary) check if it has a match in the errorReport; - if so, that's the process that caused the crash; assign the process to the "process" variable.

…rates into nextflow_error_report

- Each wgdOrtholog process produces a file storing the two species names in the work directory of the process. - The error logger knows the work directory and the code accesses the two names from the file therein. - Avoid parsing species names from unstable output

Cecilia-Sensalari · 2021-08-16T13:48:47Z

UPDATE:
About parsing in the error log the two species names in an interrupted wgdOrtholog process.
Each wgdOrthologs process now produces a file in its work directory storing the two names (work/xx/xxxxxxxx/species_names.tsv). Given that the errorReport shows the work directory of the interrupted process (see below), the code parses its path out of the errorReport and reads the species_names.tsv file herein.

The parsing of the path works this way: the code looks for the line containing "Work dir:" and then takes the successive line.

- First look for the line containing "Caused by:" in the errorReport - Get its index - Get the content of the successive line, which effectively contains the messsage

…rates into nextflow_error_report

- both in onError and onComplete

- both in onComplete and onError - onComplete will print that it can't delete possible leftovers in paralog_distrib. - onError will prints a message pointing to the log file without using the whole path

ksrates/fc_configfile.py

Co-authored-by: lohausr <56632794+lohausr@users.noreply.github.com>

main.nf

- If errorReport gives nothing useful, apply the same to errorMessage - If species names are still unknown, point to a generic wgd_ortholog_

main.nf

Co-authored-by: lohausr <56632794+lohausr@users.noreply.github.com>

…rates into nextflow_error_report

ksrates/plot_paralogs.py

ksrates/setup_correction.py

ksrates/wgd_paralogs.py

Co-authored-by: lohausr <56632794+lohausr@users.noreply.github.com>

lohausr

Seems all good to me now, thx!

cesen added 7 commits June 28, 2021 18:43

Small fix (separator length)

6a21210

Minor fix

ce71555

Error box catches also external error causes

0f57738

- In case the log file of the stopped process doesn't exist or doesn't contain any ERROR line, catch the error from "Caused by:" errorReport lines and from errorMessage.

Simplify error box implementation

598a21b

- If the log file exists AND if it contains ERROR lines, print the error lines - If any of the two conditions is not true, print the "Caused by:" line and the errorMessage

Nextflow error box catches "ctrl+C"

b75aa1c

Cecilia-Sensalari marked this pull request as draft July 6, 2021 10:12

cesen added 3 commits July 6, 2021 16:52

Update FAQs with error box description and figures

83e3764

Minor change (removed extra log line)

342e4a9

lohausr suggested changes Aug 9, 2021

View reviewed changes

Cecilia-Sensalari and others added 16 commits August 9, 2021 13:02

Apply code edits from review

622b52e

Co-authored-by: lohausr <56632794+lohausr@users.noreply.github.com>

Apply code edits from review

0221335

Co-authored-by: lohausr <56632794+lohausr@users.noreply.github.com>

FAQs: minor edit (removed paragraph indentation)

e5e02a9

main.nf: delete "if (LOG)" in workflow.onError

21eecd9

Error box drawn also for SIGINT error message

5a6c691

Make "if" and "for" syntax style consistent

7eec726

Improve directives for more details over error msg

429028b

main.nf onError parses "species_name" more safely

d8c34bd

main.nf: replace log filename with value from dict

ed32490

NF: safer parseing of process name from errReport

0f4c373

error box: process name matched through dictionary

6071b7b

- For each process name (available in dictionary) check if it has a match in the errorReport; - if so, that's the process that caused the crash; assign the process to the "process" variable.

Merge branch 'master' into nextflow_error_report

43847fa

Merge branch 'nextflow_error_report' of https://github.com/VIB-PSB/ks…

c9d6b90

…rates into nextflow_error_report

main.nf: more safely parse "Caused by:" line

8fed6c3

- First look for the line containing "Caused by:" in the errorReport - Get its index - Get the content of the successive line, which effectively contains the messsage

Cecilia-Sensalari added 6 commits August 20, 2021 11:14

Improve pattern to match species pair names

cc85c94

Merge branch 'nextflow_error_report' of https://github.com/VIB-PSB/ks…

c9be156

…rates into nextflow_error_report

Fix focal species parsing with strip()

54d2c96

- both in onError and onComplete

Code exits if focal species in config has spaces

51ac819

Update comment about species name parsing

5baec3a

If focal species not configured, print message

3c07cdc

- both in onComplete and onError - onComplete will print that it can't delete possible leftovers in paralog_distrib. - onError will prints a message pointing to the log file without using the whole path

lohausr reviewed Aug 20, 2021

View reviewed changes

ksrates/fc_configfile.py Outdated Show resolved Hide resolved

Make log error consistent with docs

688171c

Co-authored-by: lohausr <56632794+lohausr@users.noreply.github.com>

lohausr reviewed Aug 20, 2021

View reviewed changes

main.nf Outdated Show resolved Hide resolved

Cecilia-Sensalari added 5 commits August 20, 2021 14:25

onError: use focal_species = "" if not defined

5b9d5e7

Priority given to check if species name is empty

7a87c13

Bugfix: split("=") needed also within if-clause

32ea755

Use errorMessage as backup to get species names

3c5e41a

- If errorReport gives nothing useful, apply the same to errorMessage - If species names are still unknown, point to a generic wgd_ortholog_

Improve log.info about missing focal species name

881df42

lohausr reviewed Aug 20, 2021

View reviewed changes

main.nf Outdated Show resolved Hide resolved

Cecilia-Sensalari and others added 6 commits August 20, 2021 18:40

Minor (log line error box)

235f1b5

Co-authored-by: lohausr <56632794+lohausr@users.noreply.github.com>

Move parameter and colinearity checkpoint earlier

4beb9d5

Merge branch 'nextflow_error_report' of https://github.com/VIB-PSB/ks…

32fde4e

…rates into nextflow_error_report

Check if parsed species pair names are "null"

8d166d3

Error msg gives full path to wgd_ortholog_ logfile

0203a14

Python checks for paranome and collinearity params

4c1e98f