Skip to content

Print selective-check traceback on stdout rather than stderr#38441

Merged
potiuk merged 1 commit intoapache:mainfrom
potiuk:redirect-selective-checks-exceptions-to-stdout
Mar 24, 2024
Merged

Print selective-check traceback on stdout rather than stderr#38441
potiuk merged 1 commit intoapache:mainfrom
potiuk:redirect-selective-checks-exceptions-to-stdout

Conversation

@potiuk
Copy link
Copy Markdown
Member

@potiuk potiuk commented Mar 24, 2024

We are using stdout of selective-check to print diagnostic information and stderr is redirected to produce outputs for GIHUB_OUTPUTS special variable in GitHub Actions. However this means that when there is an error when running selective-checks, the traceback goes to stderr and we cannot see it, plus it makes Github Actions to fail with crypttic errors.

This PR catches uncaught exception and uses rich mechanism to print the traceback to the diagnostic (stdout) console instead so that we can see it - in colour as well as with local variables, which might become handy.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

We are using stdout of selective-check to print diagnostic information
and stderr is redirected to produce outputs for GIHUB_OUTPUTS
special variable in GitHub Actions. However this means that when
there is an error when running selective-checks, the traceback
goes to stderr and we cannot see it, plus it makes Github Actions
to fail with crypttic errors.

This PR catches uncaught exception and uses rich mechanism to print
the traceback to the diagnostic (stdout) console instead so that
we can see it - in colour as well as with local variables, which
might become handy.
@potiuk
Copy link
Copy Markdown
Member Author

potiuk commented Mar 24, 2024

Should help to diagnose issues like https://github.com/apache/airflow/actions/runs/8410545233/job/23029279435

Copy link
Copy Markdown
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An alternative would be use use 2>&1 in the shell call to redirect all stderr to stdout?

In total meaning: breeze ci selective-check 2>&1 | tee ${GITHUB_OUTPUT} ?

@potiuk
Copy link
Copy Markdown
Member Author

potiuk commented Mar 24, 2024

An alternative would be use use 2>&1 in the shell call to redirect all stderr to stdout?
In total meaning: breeze ci selective-check 2>&1 | tee ${GITHUB_OUTPUT} ?

We already do that actually

run: breeze ci selective-check 2>> ${GITHUB_OUTPUT}

We print diagnostics information to stdout so that it can be seen in colour - for example here https://github.com/apache/airflow/actions/runs/8411260687/job/23030711777?pr=38437#step:8:849

In the past I also tried to revert it - i.e. print all the diagnostics to stderr, and then print the outputs to stdout and do > ${GITHUB_OUTPUT} - but it was far too easy to overlook something and it's better to redirect stderr to GITHUB_OUTPUTS and leave stdout for diagnostics. Then we only have to catch all the exceptions like this and print traceback to stdout, it's far easier than trace down or pass "stderr" as the diagnostic output.

@potiuk potiuk merged commit 25e5d54 into apache:main Mar 24, 2024
@potiuk potiuk deleted the redirect-selective-checks-exceptions-to-stdout branch March 24, 2024 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants