Skip to content

Show standard error for failing CWL jobs that capture standard error - #5555

Merged
adamnovak merged 7 commits into
masterfrom
issues/2994-show-standard-error
Jul 24, 2026
Merged

Show standard error for failing CWL jobs that capture standard error#5555
adamnovak merged 7 commits into
masterfrom
issues/2994-show-standard-error

Conversation

@adamnovak

Copy link
Copy Markdown
Member

This fixes #2994 by logging standard error from CWL tasks to the Toil log, when the tasks fail, even if standard error for the task is being captured to a file.

(When standard error is not captured, we already log it to the Toil log.)

The result looks just like if it wasn't being captured:

[2026-07-16T17:15:05-0400] [Thread-4 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] fail_with_log.cwl.hello._:640ebd69-f2a0-4ed6-a923-4b450915dd90.stderr follows:
=========>
	Message: This is a test
<=========

We don't do this when the job succeeds, because the user might be capturing the log because it's inconveniently large.

Changelog Entry

To be copied to the draft changelog by merger:

  • CWL job error logs are now reported for failing jobs even if the job's standard error is captured to a file.

Reviewer Checklist

  • Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
    • If it is coming from an external repo, make sure to pull it in for CI with:
      contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing
      
    • If there is no associated issue, create one.
  • Read through the code changes. Make sure that it doesn't have:
    • Addition of trailing whitespace.
    • New variable or member names in camelCase that want to be in snake_case.
    • New functions without type hints.
    • New functions or classes without informative docstrings.
    • Changes to semantics not reflected in the relevant docstrings.
    • New or changed command line options for Toil workflows that are not reflected in docs/running/{cliOptions,cwl,wdl}.rst
    • New features without tests.
  • Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.
  • Finish the review with an overall description of your opinion.

Merger Checklist

  • Make sure the PR passed tests, including the Gitlab tests, for the most recent commit in its branch.
  • Make sure the PR has been reviewed. If not, review it. If it has been reviewed and any requested changes seem to have been addressed, proceed.
  • Merge with the Github "Squash and merge" feature.
    • If there are multiple authors' commits, add Co-authored-by to give credit to all contributing authors.
  • Copy its recommended changelog entry to the Draft Changelog.
  • Append the issue number in parentheses to the changelog entry.

@mr-c mr-c left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Epic PR, nice re-org!

Comment thread src/toil/cwl/cwltoil.py Outdated
Comment thread src/toil/cwl/cwltoil.py Outdated
Comment thread src/toil/cwl/cwltoil.py Outdated
Comment thread src/toil/cwl/cwltoil.py Outdated
Comment thread src/toil/cwl/cwltoil.py Outdated
Comment thread src/toil/cwl/cwltoil.py Outdated

@mr-c mr-c left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some typos :-)

I should consider learning to spell

Co-authored-by: Michael R. Crusoe <1330696+mr-c@users.noreply.github.com>
@adamnovak
adamnovak merged commit c6c2b1e into master Jul 24, 2026
3 checks passed
@adamnovak
adamnovak deleted the issues/2994-show-standard-error branch July 24, 2026 15:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

error log file not visible with stderr redirection

2 participants