Job output being truncated

[netcho]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I am not entitled to status updates or other assurances.

Summary

Job output log gets truncated when reaching around 3300 lines without showing '...' symbol. There is no summary at the end

AWX version

19.4.0

Installation method

kubernetes

Modifications

no

Ansible version

2.11

Operating system

CentOS 8

Web browser

Firefox, Chrome

Steps to reproduce

Running a random playbook that generates more than 3500 lines of output.

Expected results

The full log should appear in the web UI.

Actual results

The log gets chopped off somewhere around 3300 lines.

Additional information

I've inspected the job via the API after it had finished. This is the last page of the jobs events:

`HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept
X-API-Node: awx-storpool-f444ccdf5-4vfld
X-API-Product-Name: AWX
X-API-Product-Version: 19.4.0
X-API-Time: 0.205s
X-UI-Max-Events: 8000

{
"count": 4900,
"next": null,
"previous": "/api/v2/jobs/26/job_events/?page=195",
"results": [
{
"id": 67765,
"type": "job_event",
"url": "/api/v2/job_events/67765/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67765/children/",
"host": "/api/v2/hosts/32/"
},
"summary_fields": {
"host": {
"id": 32,
"name": "c7influxdb12",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.240067Z",
"modified": "2021-10-11T16:55:23.453629Z",
"job": 26,
"event": "runner_on_skipped",
"counter": 4877,
"event_display": "Host Skipped",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb12",
"remote_addr": "c7influxdb12",
"start": "2021-10-11T16:55:23.161738",
"end": "2021-10-11T16:55:23.239227",
"duration": 0.077489,
"event_loop": null,
"uuid": "3f551a78-876a-49b7-b2f8-e0760c2b556a",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "3f551a78-876a-49b7-b2f8-e0760c2b556a",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 32,
"host_name": "c7influxdb12",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "\u001b[0;36mskipping: [c7influxdb12]\u001b[0m",
"start_line": 3298,
"end_line": 3299,
"verbosity": 0
},
{
"id": 67767,
"type": "job_event",
"url": "/api/v2/job_events/67767/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67767/children/",
"host": "/api/v2/hosts/34/"
},
"summary_fields": {
"host": {
"id": 34,
"name": "c7influxdb14",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.228073Z",
"modified": "2021-10-11T16:55:23.453188Z",
"job": 26,
"event": "runner_on_start",
"counter": 4876,
"event_display": "Host Started",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb14",
"uuid": "4ee32a7a-4768-42b8-ba82-4133f8f26b71",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "4ee32a7a-4768-42b8-ba82-4133f8f26b71",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 34,
"host_name": "c7influxdb14",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "",
"start_line": 3298,
"end_line": 3298,
"verbosity": 0
},
{
"id": 67768,
"type": "job_event",
"url": "/api/v2/job_events/67768/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67768/children/",
"host": "/api/v2/hosts/35/"
},
"summary_fields": {
"host": {
"id": 35,
"name": "c7influxdb15",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.263735Z",
"modified": "2021-10-11T16:55:23.454336Z",
"job": 26,
"event": "runner_on_start",
"counter": 4878,
"event_display": "Host Started",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb15",
"uuid": "a95caa4b-145d-4528-967d-2430d477ee07",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "a95caa4b-145d-4528-967d-2430d477ee07",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 35,
"host_name": "c7influxdb15",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "",
"start_line": 3299,
"end_line": 3299,
"verbosity": 0
},
{
"id": 67769,
"type": "job_event",
"url": "/api/v2/job_events/67769/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67769/children/",
"host": "/api/v2/hosts/33/"
},
"summary_fields": {
"host": {
"id": 33,
"name": "c7influxdb13",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.271034Z",
"modified": "2021-10-11T16:55:23.454521Z",
"job": 26,
"event": "runner_on_skipped",
"counter": 4879,
"event_display": "Host Skipped",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb13",
"remote_addr": "c7influxdb13",
"start": "2021-10-11T16:55:23.195306",
"end": "2021-10-11T16:55:23.270293",
"duration": 0.074987,
"event_loop": null,
"uuid": "a809ea21-89cb-48c7-abb3-11a894638f6d",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "a809ea21-89cb-48c7-abb3-11a894638f6d",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 33,
"host_name": "c7influxdb13",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "\u001b[0;36mskipping: [c7influxdb13]\u001b[0m",
"start_line": 3299,
"end_line": 3300,
"verbosity": 0
},
{
"id": 67759,
"type": "job_event",
"url": "/api/v2/job_events/67759/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67759/children/",
"host": "/api/v2/hosts/36/"
},
"summary_fields": {
"host": {
"id": 36,
"name": "c7influxdb2",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.295378Z",
"modified": "2021-10-11T16:55:23.454820Z",
"job": 26,
"event": "runner_on_start",
"counter": 4880,
"event_display": "Host Started",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb2",
"uuid": "b606d5ff-ee42-4897-8448-feb88b48d1a4",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "b606d5ff-ee42-4897-8448-feb88b48d1a4",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 36,
"host_name": "c7influxdb2",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "",
"start_line": 3300,
"end_line": 3300,
"verbosity": 0
},
{
"id": 67760,
"type": "job_event",
"url": "/api/v2/job_events/67760/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67760/children/",
"host": "/api/v2/hosts/34/"
},
"summary_fields": {
"host": {
"id": 34,
"name": "c7influxdb14",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.302579Z",
"modified": "2021-10-11T16:55:23.457255Z",
"job": 26,
"event": "runner_on_skipped",
"counter": 4881,
"event_display": "Host Skipped",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb14",
"remote_addr": "c7influxdb14",
"start": "2021-10-11T16:55:23.227583",
"end": "2021-10-11T16:55:23.301853",
"duration": 0.07427,
"event_loop": null,
"uuid": "3d2ccf1f-4679-4a82-befd-3ba2017abae6",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "3d2ccf1f-4679-4a82-befd-3ba2017abae6",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 34,
"host_name": "c7influxdb14",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "\u001b[0;36mskipping: [c7influxdb14]\u001b[0m",
"start_line": 3300,
"end_line": 3301,
"verbosity": 0
},
{
"id": 67770,
"type": "job_event",
"url": "/api/v2/job_events/67770/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67770/children/",
"host": "/api/v2/hosts/37/"
},
"summary_fields": {
"host": {
"id": 37,
"name": "c7influxdb3",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.322260Z",
"modified": "2021-10-11T16:55:23.457677Z",
"job": 26,
"event": "runner_on_start",
"counter": 4882,
"event_display": "Host Started",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb3",
"uuid": "07e10c03-caa4-4bba-a52b-35358a4f0384",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "07e10c03-caa4-4bba-a52b-35358a4f0384",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 37,
"host_name": "c7influxdb3",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "",
"start_line": 3301,
"end_line": 3301,
"verbosity": 0
},
{
"id": 67771,
"type": "job_event",
"url": "/api/v2/job_events/67771/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67771/children/",
"host": "/api/v2/hosts/35/"
},
"summary_fields": {
"host": {
"id": 35,
"name": "c7influxdb15",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.329101Z",
"modified": "2021-10-11T16:55:23.460450Z",
"job": 26,
"event": "runner_on_skipped",
"counter": 4883,
"event_display": "Host Skipped",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb15",
"remote_addr": "c7influxdb15",
"start": "2021-10-11T16:55:23.263506",
"end": "2021-10-11T16:55:23.328380",
"duration": 0.064874,
"event_loop": null,
"uuid": "b457b749-a76b-4c2f-a46e-4af9e8655e7a",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "b457b749-a76b-4c2f-a46e-4af9e8655e7a",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 35,
"host_name": "c7influxdb15",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "\u001b[0;36mskipping: [c7influxdb15]\u001b[0m",
"start_line": 3301,
"end_line": 3302,
"verbosity": 0
},
{
"id": 67777,
"type": "job_event",
"url": "/api/v2/job_events/67777/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67777/children/",
"host": "/api/v2/hosts/38/"
},
"summary_fields": {
"host": {
"id": 38,
"name": "c7influxdb4",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.348512Z",
"modified": "2021-10-11T16:55:23.705379Z",
"job": 26,
"event": "runner_on_start",
"counter": 4884,
"event_display": "Host Started",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb4",
"uuid": "0ea999f3-aece-4c4a-a64c-7f9a2ec22e46",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "0ea999f3-aece-4c4a-a64c-7f9a2ec22e46",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 38,
"host_name": "c7influxdb4",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "",
"start_line": 3302,
"end_line": 3302,
"verbosity": 0
},
{
"id": 67778,
"type": "job_event",
"url": "/api/v2/job_events/67778/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67778/children/",
"host": "/api/v2/hosts/36/"
},
"summary_fields": {
"host": {
"id": 36,
"name": "c7influxdb2",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.356107Z",
"modified": "2021-10-11T16:55:23.705379Z",
"job": 26,
"event": "runner_on_skipped",
"counter": 4885,
"event_display": "Host Skipped",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb2",
"remote_addr": "c7influxdb2",
"start": "2021-10-11T16:55:23.295209",
"end": "2021-10-11T16:55:23.355423",
"duration": 0.060214,
"event_loop": null,
"uuid": "dc8fe9a4-b0ef-4d22-a4fa-1875891478c6",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "dc8fe9a4-b0ef-4d22-a4fa-1875891478c6",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 36,
"host_name": "c7influxdb2",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "\u001b[0;36mskipping: [c7influxdb2]\u001b[0m",
"start_line": 3302,
"end_line": 3303,
"verbosity": 0
},
{
"id": 67779,
"type": "job_event",
"url": "/api/v2/job_events/67779/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67779/children/",
"host": "/api/v2/hosts/39/"
},
"summary_fields": {
"host": {
"id": 39,
"name": "c7influxdb5",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.378308Z",
"modified": "2021-10-11T16:55:23.705379Z",
"job": 26,
"event": "runner_on_start",
"counter": 4886,
"event_display": "Host Started",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb5",
"uuid": "9335f89b-4d0f-4723-ad96-bcaead3ffe5f",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "9335f89b-4d0f-4723-ad96-bcaead3ffe5f",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 39,
"host_name": "c7influxdb5",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "",
"start_line": 3303,
"end_line": 3303,
"verbosity": 0
},
{
"id": 67780,
"type": "job_event",
"url": "/api/v2/job_events/67780/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67780/children/",
"host": "/api/v2/hosts/37/"
},
"summary_fields": {
"host": {
"id": 37,
"name": "c7influxdb3",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.385507Z",
"modified": "2021-10-11T16:55:23.705379Z",
"job": 26,
"event": "runner_on_skipped",
"counter": 4887,
"event_display": "Host Skipped",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb3",
"remote_addr": "c7influxdb3",
"start": "2021-10-11T16:55:23.322106",
"end": "2021-10-11T16:55:23.384787",
"duration": 0.062681,
"event_loop": null,
"uuid": "30d829b6-6b0e-474f-8ac1-09e45c6111c4",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "30d829b6-6b0e-474f-8ac1-09e45c6111c4",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 37,
"host_name": "c7influxdb3",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "\u001b[0;36mskipping: [c7influxdb3]\u001b[0m",
"start_line": 3303,
"end_line": 3304,
"verbosity": 0
},
{
"id": 67781,
"type": "job_event",
"url": "/api/v2/job_events/67781/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67781/children/",
"host": "/api/v2/hosts/40/"
},
"summary_fields": {
"host": {
"id": 40,
"name": "c7influxdb6",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.404919Z",
"modified": "2021-10-11T16:55:23.705379Z",
"job": 26,
"event": "runner_on_start",
"counter": 4888,
"event_display": "Host Started",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb6",
"uuid": "ce1074b8-236b-4093-90a9-e1b9bc9f5d2b",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "ce1074b8-236b-4093-90a9-e1b9bc9f5d2b",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 40,
"host_name": "c7influxdb6",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "",
"start_line": 3304,
"end_line": 3304,
"verbosity": 0
},
{
"id": 67772,
"type": "job_event",
"url": "/api/v2/job_events/67772/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67772/children/",
"host": "/api/v2/hosts/38/"
},
"summary_fields": {
"host": {
"id": 38,
"name": "c7influxdb4",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.411188Z",
"modified": "2021-10-11T16:55:23.704079Z",
"job": 26,
"event": "runner_on_skipped",
"counter": 4889,
"event_display": "Host Skipped",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb4",
"remote_addr": "c7influxdb4",
"start": "2021-10-11T16:55:23.348322",
"end": "2021-10-11T16:55:23.410414",
"duration": 0.062092,
"event_loop": null,
"uuid": "d30c57f2-55ab-4cb7-ad2a-92a4ab30b29f",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "d30c57f2-55ab-4cb7-ad2a-92a4ab30b29f",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 38,
"host_name": "c7influxdb4",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "\u001b[0;36mskipping: [c7influxdb4]\u001b[0m",
"start_line": 3304,
"end_line": 3305,
"verbosity": 0
},
{
"id": 67773,
"type": "job_event",
"url": "/api/v2/job_events/67773/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67773/children/",
"host": "/api/v2/hosts/39/"
},
"summary_fields": {
"host": {
"id": 39,
"name": "c7influxdb5",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.437816Z",
"modified": "2021-10-11T16:55:23.704079Z",
"job": 26,
"event": "runner_on_skipped",
"counter": 4891,
"event_display": "Host Skipped",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb5",
"remote_addr": "c7influxdb5",
"start": "2021-10-11T16:55:23.378143",
"end": "2021-10-11T16:55:23.436901",
"duration": 0.058758,
"event_loop": null,
"uuid": "11aa684d-8be4-4db5-a9e7-efd0a6ac6052",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "11aa684d-8be4-4db5-a9e7-efd0a6ac6052",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 39,
"host_name": "c7influxdb5",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "\u001b[0;36mskipping: [c7influxdb5]\u001b[0m",
"start_line": 3305,
"end_line": 3306,
"verbosity": 0
},
{
"id": 67782,
"type": "job_event",
"url": "/api/v2/job_events/67782/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67782/children/",
"host": "/api/v2/hosts/41/"
},
"summary_fields": {
"host": {
"id": 41,
"name": "c7influxdb7",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.430634Z",
"modified": "2021-10-11T16:55:23.705379Z",
"job": 26,
"event": "runner_on_start",
"counter": 4890,
"event_display": "Host Started",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb7",
"uuid": "bbec48dd-eb34-4573-8e91-4a13a123524c",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "bbec48dd-eb34-4573-8e91-4a13a123524c",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 41,
"host_name": "c7influxdb7",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "",
"start_line": 3305,
"end_line": 3305,
"verbosity": 0
},
{
"id": 67783,
"type": "job_event",
"url": "/api/v2/job_events/67783/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67783/children/",
"host": "/api/v2/hosts/42/"
},
"summary_fields": {
"host": {
"id": 42,
"name": "c7influxdb8",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.457544Z",
"modified": "2021-10-11T16:55:23.705379Z",
"job": 26,
"event": "runner_on_start",
"counter": 4892,
"event_display": "Host Started",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb8",
"uuid": "3807ef9c-8dae-4c7c-acdb-bf674085f4c2",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "3807ef9c-8dae-4c7c-acdb-bf674085f4c2",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 42,
"host_name": "c7influxdb8",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "",
"start_line": 3306,
"end_line": 3306,
"verbosity": 0
},
{
"id": 67774,
"type": "job_event",
"url": "/api/v2/job_events/67774/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67774/children/",
"host": "/api/v2/hosts/40/"
},
"summary_fields": {
"host": {
"id": 40,
"name": "c7influxdb6",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.465963Z",
"modified": "2021-10-11T16:55:23.704079Z",
"job": 26,
"event": "runner_on_skipped",
"counter": 4893,
"event_display": "Host Skipped",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb6",
"remote_addr": "c7influxdb6",
"start": "2021-10-11T16:55:23.404765",
"end": "2021-10-11T16:55:23.465187",
"duration": 0.060422,
"event_loop": null,
"uuid": "f4e688ee-8813-464e-a414-0a412205d2dc",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "f4e688ee-8813-464e-a414-0a412205d2dc",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 40,
"host_name": "c7influxdb6",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "\u001b[0;36mskipping: [c7influxdb6]\u001b[0m",
"start_line": 3306,
"end_line": 3307,
"verbosity": 0
},
{
"id": 67785,
"type": "job_event",
"url": "/api/v2/job_events/67785/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67785/children/",
"host": "/api/v2/hosts/43/"
},
"summary_fields": {
"host": {
"id": 43,
"name": "c7influxdb9",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.492535Z",
"modified": "2021-10-11T16:55:23.705494Z",
"job": 26,
"event": "runner_on_start",
"counter": 4894,
"event_display": "Host Started",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb9",
"uuid": "7fe98c20-b7f7-4550-a972-c1f8a7ef3cbc",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "7fe98c20-b7f7-4550-a972-c1f8a7ef3cbc",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 43,
"host_name": "c7influxdb9",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "",
"start_line": 3307,
"end_line": 3307,
"verbosity": 0
},
{
"id": 67776,
"type": "job_event",
"url": "/api/v2/job_events/67776/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67776/children/",
"host": "/api/v2/hosts/41/"
},
"summary_fields": {
"host": {
"id": 41,
"name": "c7influxdb7",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.499914Z",
"modified": "2021-10-11T16:55:23.705942Z",
"job": 26,
"event": "runner_on_skipped",
"counter": 4895,
"event_display": "Host Skipped",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb7",
"remote_addr": "c7influxdb7",
"start": "2021-10-11T16:55:23.430458",
"end": "2021-10-11T16:55:23.499157",
"duration": 0.068699,
"event_loop": null,
"uuid": "e473b46f-417c-4d39-920b-a91921eccde2",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "e473b46f-417c-4d39-920b-a91921eccde2",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 41,
"host_name": "c7influxdb7",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "\u001b[0;36mskipping: [c7influxdb7]\u001b[0m",
"start_line": 3307,
"end_line": 3308,
"verbosity": 0
},
{
"id": 67775,
"type": "job_event",
"url": "/api/v2/job_events/67775/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67775/children/",
"host": "/api/v2/hosts/45/"
},
"summary_fields": {
"host": {
"id": 45,
"name": "c7jenkins1",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.517696Z",
"modified": "2021-10-11T16:55:23.708275Z",
"job": 26,
"event": "runner_on_start",
"counter": 4896,
"event_display": "Host Started",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7jenkins1",
"uuid": "b3f3be2b-a49a-4be8-a7ec-624079889ef3",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "b3f3be2b-a49a-4be8-a7ec-624079889ef3",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 45,
"host_name": "c7jenkins1",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "",
"start_line": 3308,
"end_line": 3308,
"verbosity": 0
},
{
"id": 67784,
"type": "job_event",
"url": "/api/v2/job_events/67784/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67784/children/",
"host": "/api/v2/hosts/42/"
},
"summary_fields": {
"host": {
"id": 42,
"name": "c7influxdb8",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.526142Z",
"modified": "2021-10-11T16:55:23.710225Z",
"job": 26,
"event": "runner_on_skipped",
"counter": 4897,
"event_display": "Host Skipped",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb8",
"remote_addr": "c7influxdb8",
"start": "2021-10-11T16:55:23.457362",
"end": "2021-10-11T16:55:23.525455",
"duration": 0.068093,
"event_loop": null,
"uuid": "2b2a2a11-188c-4215-b5aa-13fcb93e8d88",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "2b2a2a11-188c-4215-b5aa-13fcb93e8d88",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 42,
"host_name": "c7influxdb8",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "\u001b[0;36mskipping: [c7influxdb8]\u001b[0m",
"start_line": 3308,
"end_line": 3309,
"verbosity": 0
},
{
"id": 67787,
"type": "job_event",
"url": "/api/v2/job_events/67787/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67787/children/",
"host": "/api/v2/hosts/43/"
},
"summary_fields": {
"host": {
"id": 43,
"name": "c7influxdb9",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.550679Z",
"modified": "2021-10-11T16:55:24.708397Z",
"job": 26,
"event": "runner_on_skipped",
"counter": 4899,
"event_display": "Host Skipped",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7influxdb9",
"remote_addr": "c7influxdb9",
"start": "2021-10-11T16:55:23.492366",
"end": "2021-10-11T16:55:23.549987",
"duration": 0.057621,
"event_loop": null,
"uuid": "c3ff65d9-d8a1-4acc-b9a6-f6fd564122a9",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "c3ff65d9-d8a1-4acc-b9a6-f6fd564122a9",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 43,
"host_name": "c7influxdb9",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "\u001b[0;36mskipping: [c7influxdb9]\u001b[0m",
"start_line": 3309,
"end_line": 3310,
"verbosity": 0
},
{
"id": 67788,
"type": "job_event",
"url": "/api/v2/job_events/67788/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67788/children/",
"host": "/api/v2/hosts/46/"
},
"summary_fields": {
"host": {
"id": 46,
"name": "c7jenkins2",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.543531Z",
"modified": "2021-10-11T16:55:24.707504Z",
"job": 26,
"event": "runner_on_start",
"counter": 4898,
"event_display": "Host Started",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7jenkins2",
"uuid": "47ac212d-dcde-4c22-93b8-554413ea2806",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "47ac212d-dcde-4c22-93b8-554413ea2806",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 46,
"host_name": "c7jenkins2",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "",
"start_line": 3309,
"end_line": 3309,
"verbosity": 0
},
{
"id": 67786,
"type": "job_event",
"url": "/api/v2/job_events/67786/",
"related": {
"job": "/api/v2/jobs/26/",
"children": "/api/v2/job_events/67786/children/",
"host": "/api/v2/hosts/51/"
},
"summary_fields": {
"host": {
"id": 51,
"name": "c7rssh",
"description": "imported"
},
"job": {
"id": 26,
"name": "Deploy Node Exporter",
"description": "Deploys node-exporter on all the nodes and update Prometheus job configuration",
"status": "error",
"failed": true,
"elapsed": 294.667,
"type": "job",
"job_template_id": 11,
"job_template_name": "Deploy Node Exporter"
},
"role": {}
},
"created": "2021-10-11T16:55:23.568361Z",
"modified": "2021-10-11T16:55:24.707234Z",
"job": 26,
"event": "runner_on_start",
"counter": 4900,
"event_display": "Host Started",
"event_data": {
"playbook": "deploy-node-exporter.yaml",
"playbook_uuid": "558b35ef-40f7-4037-89bb-b6990a64904e",
"play": "Installing node-exporter",
"play_uuid": "425223bd-b287-5849-8161-0000000000ab",
"play_pattern": "node_exporter_autodeploy",
"task": "Updating apt local cache",
"task_uuid": "425223bd-b287-5849-8161-0000000000ef",
"task_action": "apt",
"task_args": "",
"task_path": "/runner/project/tasks/install-debug-tools.yaml:1",
"host": "c7rssh",
"uuid": "5029dae3-b8d2-4650-92db-347d1597e2f8",
"guid": "139adefc07e245419668433b9dbac5ab"
},
"event_level": 3,
"failed": false,
"changed": false,
"uuid": "5029dae3-b8d2-4650-92db-347d1597e2f8",
"parent_uuid": "425223bd-b287-5849-8161-0000000000ef",
"host": 51,
"host_name": "c7rssh",
"playbook": "deploy-node-exporter.yaml",
"play": "Installing node-exporter",
"task": "Updating apt local cache",
"role": "",
"stdout": "",
"start_line": 3310,
"end_line": 3310,
"verbosity": 0
}
]
}`

I've inspected the task runner and found out that it had stopped flushing events. Here are its log:
2021-10-11 16:55:24,707 DEBUG [-] awx.main.commands.run_callback_receiver JobEvent.objects.bulk_create(1) 2021-10-11 16:55:24,708 DEBUG [-] awx.main.commands.run_callback_receiver JobEvent.objects.bulk_create(1) 2021-10-11 16:55:24,708 DEBUG [-] awx.main.commands.run_callback_receiver JobEvent.objects.bulk_create(1) 2021-10-11 16:55:39,386 WARNING [eba2d012bf404395b817bd78d5614f99] awx.main.dispatch scaling down worker pid:410 2021-10-11 16:55:39,389 WARNING [eba2d012bf404395b817bd78d5614f99] awx.main.dispatch scaling down worker pid:416 2021-10-11 16:55:39,389 DEBUG [eba2d012bf404395b817bd78d5614f99] awx.main.dispatch task d29e80b0-8f47-466f-a898-c0e501543a09 starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-10-11 16:55:39,389 WARNING [015a97f4d1fd46aa82431abc833b6e03] awx.main.dispatch worker exiting gracefully pid:410 2021-10-11 16:55:39,391 DEBUG [eba2d012bf404395b817bd78d5614f99] awx.main.scheduler Running task manager. 2021-10-11 16:55:39,393 WARNING [015a97f4d1fd46aa82431abc833b6e03] awx.main.dispatch worker exiting gracefully pid:416 2021-10-11 16:55:39,398 DEBUG [eba2d012bf404395b817bd78d5614f99] awx.main.scheduler Starting Scheduler 2021-10-11 16:55:39,449 DEBUG [eba2d012bf404395b817bd78d5614f99] awx.main.dispatch task 5cb01822-f604-4aff-a0c4-0676ad7b5fa4 starting awx.main.analytics.analytics_tasks.send_subsystem_metrics(*[]) 2021-10-11 16:55:39,461 DEBUG [eba2d012bf404395b817bd78d5614f99] awx.main.scheduler Finishing Scheduler 2021-10-11 16:55:49,406 DEBUG [1e3b68bf7516407c82dcf61f596f1640] awx.main.dispatch task 0e4bde40-94a4-46a5-bfea-6bcee4fe2738 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-10-11 16:55:49,416 DEBUG [1e3b68bf7516407c82dcf61f596f1640] awx.main.tasks Starting periodic scheduler 2021-10-11 16:55:49,420 DEBUG [1e3b68bf7516407c82dcf61f596f1640] awx.main.tasks Last scheduler run was: 2021-10-11 16:55:19.489728+00:00 2021-10-11 16:55:59,417 DEBUG [16290ba3dbda4592b20ead9a9906c2d5] awx.main.dispatch task 08f2e1ba-75f0-44d2-a980-0dc25934dc0e starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-10-11 16:55:59,421 DEBUG [16290ba3dbda4592b20ead9a9906c2d5] awx.main.scheduler Running task manager. 2021-10-11 16:55:59,429 DEBUG [16290ba3dbda4592b20ead9a9906c2d5] awx.main.scheduler Starting Scheduler 2021-10-11 16:55:59,455 DEBUG [16290ba3dbda4592b20ead9a9906c2d5] awx.main.dispatch task cc338ad4-0b3d-4a86-be19-d857ae29b087 starting awx.main.analytics.analytics_tasks.send_subsystem_metrics(*[]) 2021-10-11 16:55:59,522 DEBUG [16290ba3dbda4592b20ead9a9906c2d5] awx.main.scheduler Finishing Scheduler RESULT 2 OK2021-10-11T19:56:01.529136816+03:00 READY 2021-10-11 16:56:19,547 DEBUG [9f219064c0d64f198195e8ae96db6203] awx.main.dispatch task caab817f-d036-4871-8a4d-7ee8a6e62375 starting awx.main.tasks.cluster_node_heartbeat(*[]) 2021-10-11 16:56:19,550 DEBUG [9f219064c0d64f198195e8ae96db6203] awx.main.tasks Cluster node heartbeat task. 2021-10-11 16:56:19,579 DEBUG [9f219064c0d64f198195e8ae96db6203] awx.main.dispatch task b2153d8a-d990-4fd8-97ff-e4d5c16ca127 starting awx.main.tasks.awx_receptor_workunit_reaper(*[]) 2021-10-11 16:56:19,579 DEBUG [9f219064c0d64f198195e8ae96db6203] awx.main.dispatch task dc69b016-ae0c-49af-958d-9078add35c4a starting awx.main.tasks.awx_k8s_reaper(*[]) 2021-10-11 16:56:19,581 DEBUG [9f219064c0d64f198195e8ae96db6203] awx.main.tasks Checking for unreleased receptor work units Instance Group already registered controlplane Instance Group already registered default (changed: True) Instance Group already registered controlplane Instance Group already registered default (changed: True) Instance Group already registered controlplane Instance Group already registered default (changed: True) Instance Group already registered controlplane Instance Group already registered default (changed: True) Instance Group already registered controlplane Instance Group already registered default (changed: True) 2021-10-11 16:56:19,589 WARNING [9f219064c0d64f198195e8ae96db6203] awx.main.dispatch scaling up worker pid:428 2021-10-11 16:56:19,596 DEBUG [9f219064c0d64f198195e8ae96db6203] awx.main.tasks Checking for orphaned k8s pods for default-2. 2021-10-11 16:56:19,597 DEBUG [9f219064c0d64f198195e8ae96db6203] awx.main.dispatch task 6b7af523-fc8e-4ae0-bc3f-147f96e3cce4 starting awx.main.tasks.awx_periodic_scheduler(*[]) 2021-10-11 16:56:19,604 DEBUG [9f219064c0d64f198195e8ae96db6203] awx.main.tasks Starting periodic scheduler 2021-10-11 16:56:19,606 DEBUG [9f219064c0d64f198195e8ae96db6203] awx.main.tasks Last scheduler run was: 2021-10-11 16:55:49.416716+00:00 2021-10-11 16:56:19,605 WARNING [9f219064c0d64f198195e8ae96db6203] awx.main.dispatch scaling up worker pid:430 2021-10-11 16:56:19,611 DEBUG [9f219064c0d64f198195e8ae96db6203] awx.main.dispatch task a25370b6-280e-404d-aa91-5aff02796bef starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-10-11 16:56:19,614 DEBUG [9f219064c0d64f198195e8ae96db6203] awx.main.scheduler Running task manager. 2021-10-11 16:56:19,615 DEBUG [9f219064c0d64f198195e8ae96db6203] awx.main.dispatch task 7bf46a9b-1bd5-472f-b198-c3afa78bc1c8 starting awx.main.analytics.analytics_tasks.send_subsystem_metrics(*[]) 2021-10-11 16:56:19,623 DEBUG [9f219064c0d64f198195e8ae96db6203] awx.main.scheduler Starting Scheduler 2021-10-11 16:56:19,710 DEBUG [9f219064c0d64f198195e8ae96db6203] awx.main.scheduler Finishing Scheduler 2021-10-11 16:56:22,909 DEBUG [-] awx.main.commands.run_callback_receiver 68 is alive 2021-10-11 16:56:22,909 DEBUG [-] awx.main.commands.run_callback_receiver 68 is alive 2021-10-11 16:56:39,487 WARNING [92b6d5a8bcd74445aa6739e7fa46cfb2] awx.main.dispatch scaling down worker pid:418 2021-10-11 16:56:39,488 DEBUG [92b6d5a8bcd74445aa6739e7fa46cfb2] awx.main.dispatch task c3258452-ba2d-4728-b976-eff833a7fe7a starting awx.main.scheduler.tasks.run_task_manager(*[]) 2021-10-11 16:56:39,489 WARNING [92b6d5a8bcd74445aa6739e7fa46cfb2] awx.main.dispatch scaling down worker pid:422 2021-10-11 16:56:39,490 DEBUG [92b6d5a8bcd74445aa6739e7fa46cfb2] awx.main.scheduler Running task manager. 2021-10-11 16:56:39,489 WARNING [9f219064c0d64f198195e8ae96db6203] awx.main.dispatch worker exiting gracefully pid:418 2021-10-11 16:56:39,491 WARNING [9f219064c0d64f198195e8ae96db6203] awx.main.dispatch worker exiting gracefully pid:422 2021-10-11 16:56:39,498 DEBUG [92b6d5a8bcd74445aa6739e7fa46cfb2] awx.main.scheduler Starting Scheduler 2021-10-11 16:56:39,549 DEBUG [92b6d5a8bcd74445aa6739e7fa46cfb2] awx.main.dispatch task 8e88a778-2405-4517-a798-50c079cc07ba starting awx.main.analytics.analytics_tasks.send_subsystem_metrics(*[]) 2021-10-11 16:56:39,592 DEBUG [92b6d5a8bcd74445aa6739e7fa46cfb2] awx.main.scheduler Finishing Scheduler 2021-10-11 16:56:42,825 INFO [139adefc07e245419668433b9dbac5ab] awx.main.commands.run_callback_receiver Event processing is finished for Job 26, sending notifications 2021-10-11 16:56:42,825 INFO [139adefc07e245419668433b9dbac5ab] awx.main.commands.run_callback_receiver Event processing is finished for Job 26, sending notifications 2021-10-11 16:56:43,676 DEBUG [139adefc07e245419668433b9dbac5ab] awx.main.tasks job 26 (running) finished running, producing 4900 events.

I also monitored the Pod that was running the playbook and it did print in its stdout all the events that had occured during the execution of the playbook. I've added the following additional settings to settings.py:

USE_X_FORWARDED_HOST = True STDOUT_MAX_BYTES_DISPLAY = 20971520 MAX_UI_JOB_EVENTS = 8000 EVENT_STDOUT_MAX_BYTES_DISPLAY = 65536 MAX_EVENT_RES_DATA = 7000000 JOB_EVENT_WORKERS = 8

[AlexSCorey]

There is a lot of work being done on the job output that is up for PR right now #11193. If you have some time, check out that branch and see if it resolves your issues.

[netcho]

I doubt that this PR will fix it. The problem lies in the fact that at a certain point, the awx callback_receiver stops scraping events from the Pod where ansible-runner is executing and storing them in the DB. I came to this conclusion after querying the API directly so I can isolate the UI. The last line numbers stored in the DB and returned by the API weren't the last that ansible-runner produced in the Pod.

[chrismeyersfsu]

@netcho can you get the job events for the job in question, ordered by counter? Your conclusion that lines past =~ 3,300 may not be valid, but if you order by -end_line then we can better tell if you are missing the tail end of the output.

/api/v2/jobs/2/job_events/?order_by=-end_line

[oliverf1]

You might be hitting this issue #10366
In a nutshell, when the log files at the container level are rotated, the kubernetes API is not able to get all the logs, so the receptor daemon is not able to get all the logs so it does not send them to the AWX DB and in turn they are not present in AWX.
As a workaround you can increase the size of the log file so all the logs of job/pod are in the same file, but it's just pushing the issue forward

[netcho]

@oliverf1 @chrismeyersfsu I can confirm that kubectl logs -f <pod-name> stops at exactly the same place as the web GUI, so I tihnk it's best to mark this issue as a duplicate of #10366

[chrismeyersfsu]

Closing as this seems to be a duplicate. Thanks for the updates people!