Missing output of executed job

[marekvesely-direct]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I am not entitled to status updates or other assurances.

Summary

When i execute more templates at the same time one of job is missing output.

AWX version

19.2.2

Installation method

kubernetes

Modifications

no

Ansible version

4.1.0

Operating system

Windows 10, Ubuntu 20.04.2 LTS

Web browser

Firefox

Steps to reproduce

Create simple configuration of AWX and execute more jobs at the same time.

Expected results

Actual results

Additional information

I don't know how to specify this more, because there are no outputs, nothing about this. :-(

[nixocio]

@marekvesely-direct , thanks for filing this issue.

If you can add more information that will help to sort out this issue.

• Network requests being made on the browser/ Responses
• Can you see the details of job output once the job is finished?

[marekvesely-direct]

There is api response of this job event.

GET /api/v2/jobs/303/job_events/?order_by=counter&page=1&page_size=50

HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept
X-API-Node: awx-prod-844b9647f6-xnxkr
X-API-Product-Name: AWX
X-API-Product-Version: 19.2.2
X-API-Time: 0.010s
X-UI-Max-Events: 4000

{
    "count": 0,
    "next": null,
    "previous": null,
    "results": []
}

There is HAR exported from Firefox: https://directpojistovnaas-my.sharepoint.com/:u:/g/personal/marek_vesely_direct_cz/EYnEosghQUFDlC0qh42Gr3UBX9EXCqWjS49OvfWeVic4hg?e=tbBIHS
Either when job is running i dont see output in AWX.

[marekvesely-direct]

So when i executed 4 templates at the same time via Scheduler, one of them was without output.

[marekvesely-direct]

@nixocio is this enough? I dont know what else i should find about this, there is no error ...

[shanemcd]

This might be another instance of #9961

[marekvesely-direct]

There is some error output but in my case nothing. I checked also logs of workers and there is everything but probably it wasn't sent to AWX otchestrating machine.

[marekvesely-direct]

Ok, I figured out this issue. This was because of resource limits i added to AWX. After i removed limits it works as expected.

[Hroman9n]

@marekvesely-direct Hi, at the moment I am facing exactly the same problem (no output, empty API response). Can you specify which resource limits you removed?

[marekvesely-direct]

@Hroman9n I configured this because in history I had 1 node cluster of Kube, but now I have 3 node so I completely removed this and everything works fine now.

  web_resource_requirements:
    requests:
      cpu: 100m
      memory: 500m
    limits:
      cpu: 2000m
      memory: 4Gi
  task_resource_requirements:
    requests:
      cpu: 50m
      memory: 200m
    limits:
      cpu: 1000m
      memory: 1Gi
  ee_resource_requirements:
    requests:
      cpu: 200m
      memory: 200m
    limits:
      cpu: 1000m
      memory: 2Gi
  postgres_resource_requirements:
    requests:
      cpu: 200m
      memory: 1Gi
    limits:
      cpu: 1000m
      memory: 4Gi

[marekvesely-direct]

I suppose error is there because it reaches limits of automation_job pod and it is killed by Kubernetes. So there is no output anything in controller machine.