Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Capture container logs and upload as artficats when failed #19755

Closed

Conversation

khalidmammadov
Copy link
Contributor

When CI containers crashed logs are not captured.
This will make default container output directory to files/container_logs that is uploaded as a next step in the CI when failed.
It will ensure that even containers are failed the latest progress from parallel runs are available for investigation.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think those are the same logs - unfortunately - we see in the output (last few lines). not much difference except that it it is the "whole" output file - of course we see also the other container output but I am afraid it won't help too much in this case.

The problem is somewhere in test_manager.py

@khalidmammadov
Copy link
Contributor Author

This is not just for that issue were we looking at but in general I though it would be good to have this data as it also provides date and timestamps of the message that could be useful as well. Also, I think since it's not dependent from monitoring frequency every single log would be captured and made ready in case of a failure.
But if you think we have got this data and this change is not necessary then I will close, at the end of the day you know this CI pipeline better than anyone! :)

@potiuk
Copy link
Member

potiuk commented Nov 23, 2021

Actually that might be a good one. But we need to make sure they are printed (further down) to the regular GitHub Actions log. The idea is that we should only upload artifacts when there is a failure and we should not duplicate what we manage to print to the log.

@khalidmammadov
Copy link
Contributor Author

It still prints into action logs (by monitoring command) and I was guided by below lines which is only activated when there is a failure. So, if it fails it will upload otherwise will work as normal.
- name: "Upload container logs"
uses: actions/upload-artifact@v2
if: failure()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants