AWX is incompatible with several docker log drivers including Graylog (GELF) due to Docker 16k log line limit. Results in incorrect job failures in AWX GUI.

[smullenrga]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I am not entitled to status updates or other assurances.

Summary

Docker implemented a 16k log line limit and splits log lines over 16k, adding log line metadata for partial log line handling. Currently only a few of the docker log drivers have been updated to handle these partial lines successfully. Jsonfilelog has been updated to reassemble the long lines and works successfully with AWX. Fluentd log driver has been updated to pass partial line metadata through to the recipient and will work with AWX as long as AWX has code for log line reassembly (I've been unable to verify this). The docker GELF driver has not been updated to deal with partial lines and will break the AWX GUI if any log lines over 16k are encountered (For me this happens with a windows gather facts). Anyone using the Docker GELF driver to send Docker logs to ELK stacks will see AWX GUI report failures for jobs that may actually succeed simply because AWX is expecting properly formatted JSON lines from the execution environments and will receive partial lines for any output lines over 16k.

Per the Docker issue opened regarding Graylog log driver updates, the Graylog spec does not support Docker's implementation of partial log line metadata and this issue may not be able to be corrected until/unless the graylog protocol is changed to support partial messages/chunking in a format that can work with Docker.

Docker Issue 43162

AWX version

19.4.0

Installation method

kubernetes

Modifications

no

Ansible version

3.X

Operating system

CentOS

Web browser

No response

Steps to reproduce

• Install AWX in a kubernetes environment
• Configure docker in the environment to use the gelf log driver (daemon.json)
• Run this playbook through AWX which generates a >16k line to STDOUT (adds 512 bytes to a string in a loop) test16kbug.txt

Expected results

Job should succeed with 35 iterations and generate output showing the resulting variable

Actual results

Resulting job will be failed and GUI output will quit after "Item=30". Examining the detailed lines of STDOUT received by AWX, as soon as the STDOUT lines exceed 16k, the STDOUT becomes broken with improperly concatenated lines that are no longer parsible as valid JSON.

Additional information

While I understand that this is really more of an issue with log drivers in Docker (I've opened an issue there), I feel it's critical that AWX users know/understand that AWX in Kubernetes is only functional using particular log drivers in Docker until the Docker side of the house is fixed. Further, AWX regression testing should apparently include testing with each docker log driver since the log drivers have a direct impact on the functionality of this product.

[gamuniz]

Due to the age of this issue and that isn't a wide spread closing this issue.