UI: Lines 3094 - 7371 not shown in job output

[Jeffrey44113]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.
• I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

Running a playbook that has allot of lines / output it doesn't show any output for lines 3094 until 7371.

The job runs correctly.
When downloading the output to a file it is also seen that it skips everything between those lines.

AWX version

24.0.0

Select the relevant components

• UI
• UI (tech preview)
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

Firefox

Steps to reproduce

Run a job template that results in allot ( and I mean allot ) of lines.

Expected results

All output should be shown and output should not be skipped

Actual results

Lines 3094-7371 are skipped in job output

Additional information

No response

[fosterseth]

When downloading the output to a file it is also seen that it skips everything between those lines.

so you are missing stdout that you expected to be there?

In AWX UI, under settings > troubleshooting settings there is an option to disable cleaning up job pods

if you disable that the jobs pods will remain in your cluster indefinitely, allowing you to examine this post-run

it would be nice to know if you are missing data there as well (kubectl logs <job-pod-name>)

[Jeffrey44113]

Indeed the entire stdout of couple of tasks are missing since it isn't outputting couple of k of lines that is not in between that.

Will check this one next week since I will be off for a couple of days..

[Klaas-]

there are some issues with log rotation on k8s < 1.29 -- this could be an artifact of that.

[Jeffrey44113]

So doing some more testing. It seems that also the missing data is in the kubectl logs..

It just stops logging

[Jeffrey44113]

to add some more info:
k3s version:
v1.28.7+k3s1

[Klaas-]

did you read my message? Update to 1.29 and I am guessing this will disappear. ansible/receptor#446 #14158 Fixed by kubernetes/kubernetes#115702

[Jeffrey44113]

Yes I have seen your message but I had not seen any issue link or anything so as you might understand I won't be then running directly as crazy to update...
he only thing you had explained was "there are some issues with log rotation on k8s < 1.29 -- this could be an artifact of that."
Now yes you have added some links to some issues. But as you might understand I can't just update out of the blue the kubernetes version of a production system.
And also how exactly are you sure the issue will be solved and how is that in relation to this issue? Trying to understand it a bit instead of blindly updating and still not knowing what it caused of what the issue was..

[Jeffrey44113]

I will see if I can later today recreate the issue on a clone that I can then update also to a higher k3s version

[Jeffrey44113]

Was able to find some time and tested it out on a clone.
Upgraded to v1.29.3+k3s1
Also upgraded the operator.

It indeed now shows all the lines correctly.
Since this is not related to awx or the operator I will also close this issue with this comment.

For @Klaas- I didn't mean any disrespect or anything just wanted to learn and understand why. From your initial comment it didn't show any reason or why only that there is log issues somewhere in <1.29
You comment after that explained it a bit further so thank you for that.