Jobs with ID > 9999 are stuck in pending state, but nothing is happening #10489
Comments
|
Maybe this is related: https://groups.google.com/g/awx-project/c/GfDZ3iXxjzc |
|
Just tested with job ids > 10k and I can't reproduce this. Is there anything useful on pod logs when you hit this? |
|
No, the task container logs are filled only showing blocked jobs now, because job 10000 - 100010 are running but not doing anything. If you want I can cancel all jobs with ID 10000+ and start a new job, so I have more recent job logs. Do you need logs from web and awx-ee container as well? |
shanemcd
changed the title
|
Hello. Did you migrate from a local docker install or did you start fresh with the operator? |
|
Can you provide the json from |
|
Hi @shanemcd Sorry for my late response, I took a long weekend off.
We did a fresh start with the operatorHello. Did you migrate from a local docker install or did you start fresh with the operator?
Sure ! (I had to replace some hostnames with ***** because of company policy, I hope this is ok?) |
|
ENVIRONMENT I hit the issue after job id 1600. |
|
@shanemcd Today we have hit the 10000 jobs again, and same behaviour. All jobs with JobID 10000+ are stuck. What do you need from us to debug and fix this issue? |
|
any one found a solution for this one ? :-) |
My workaround is to cancel the pending jobs and redeploy the awx. It will work for a while, but it will reoccur. |
|
We are experiencing the same thing. We went from a 17.0.1 local docker install to 19.2.2 on kubernetes (k3s) and already had well over 10,000 historical jobs. When we try to run jobs, we receive zero output, even after setting verbosity to level 4. We have tried 19.2.0 and 19.2.1 to no avail. I have noticed that by default we have two instance groups (controlplane and tower) and one container group (default). Our jobs defaulted to trying to use the container group and that is when we see zero output. If I force the job to run with controlplane or tower instead of default, we are stuck in a pending state and the job never runs. |
|
Following up on my previous post. After changing the instance group to tower or controlpane, below is the message provided while the job is stuck in the pending state: It is also worth noting that when the default container group is associated with a normal job template there is no output and I never see a new pod spin up, but I can successfully run an inventory sync and see a new pod temporarily spin up with "automation-job" in the name. |
|
After having this occur on k3s, I stood up a new server with RKE2 running and ran into the exact same issue. There does appear to be something wrong as it pertains to the instance groups and how they function on certain clusters. |
|
For project_updates and jobs when they are told to run on the tower instance group (one of the default instance groups) instead of running using the default container group, they show the following errors in the awx-task container while stuck in the pending state (the log was grepped for scheduler):
It appears as though we are being caught by the following conditional statements: awx/awx/main/scheduler/task_manager.py Lines 489 to 498 in a6383e7 It seems as though our tasks' can_run_containerized attribute is somehow set to false and this is preventing our tasks from running. I don't know how this attribute is declared on a per task basis, is there a way for us to manually change it for a task and test? |
|
The class for the unified jobs appear to set the can_run_containerized and can_run_on_control_plane properties to false: ( awx/awx/main/models/unified_jobs.py Lines 742 to 751 in c09cad3 However, for just jobs, it is set to true ( Lines 1239 to 1241 in c09cad3 Lines 746 to 748 in c09cad3 Also, for inventories can_run_containerized is set to true (why inventory sync would still work): awx/awx/main/models/inventory.py Lines 1217 to 1219 in c09cad3 I looked into our unified job templates (https://yourawxserver.yourdomain.com/api/v2/unified_job_templates) and our job templates (https://yourawxserver.yourdomain.com/api/v2/job_templates). I determined that many of our templates and all of our projects were classified as unified job templates. It appears as though any launched template will hang; because it is either a unified job template itself, or it is associated with a project which is classified as a unified job template. If anyone else who has experienced this issue can corroborate what I have mentioned above, that could be very helpful. Does anyone know the intended distinction between "unified job templates" and just "job templates"? Also, does anyone know if all projects should be classified as "unified job templates" and why it would be desirable to have their can_run_containerized and can_run_on_control_plane properties set to false? I am wondering if we have a case of objects being erroneously identified as "unified job templates," or if there is something else going on. |
|
All templates (job, workflow, project updates, etc) are unified job templates - that's the base class they all inherit from. |
|
@wenottingham Is it the intention to only allow jobs to run on container groups and no longer allow jobs to run on instance groups? There seems to be two issues:
|
|
I think that I may have tracked down the issue, although I am wondering if this is truly exclusive to job IDs > 9999 or not. If it is, then there may be something additional. The below code snippet, where the can_run_containerized property is set to true is present in inventory.py and jobs.py, but not projects.py. All of the issues that have been presented in this issue are occurring when a project update is run or a job is run and it automatically attempts to run a project update before executing the playbook for the job. @Property Was there any reason to exclude setting this property to true in awx/awx/main/models/projects.py? If not, I believe this could be the root of the issue. |
|
FYI, I have encountered an issue #11051 which I found to be connected to this one. From what I found the issue occurs for jobs with IDs > 9999 that use ssh keys with passphrases only in our case. It is happening because the jobs are indefinitely waiting for the passphrase. When I manually inject the passphrases, the jobs proceed. Maybe this helps further troubleshoot the issue. |
|
FYI, this issue appears to be fixed for me with v19.4.0. It appears as though the changes made to instance and container group logic seems to have resolved this issue. |
|
Yeah! I've updated our test environment to 9.4.0 this evening. This one contained the test database which had jobs with job id > 9999 in a pending state without doing anything. And they seem to run now. I will update production soon, the current job ID for production is in the 5k now (had to reset it last week because of this bug), so we will find out soon enough if our production environment keeps working after the upgrade to 19.4.0. If so I will close this issue. |
|
I posted a question on stackoverflow which led me here -- https://stackoverflow.com/questions/70320452/awx-5-0-0-all-jobs-stop-processing-and-hang-indefinitely-why There's a bit of analysis in there -- I don't see a reason why the job ID count hitting 10k would trigger this condition. However, the symptoms are identical to other commenters here -- it's the injection of credentials via the file pipe causing the issue. Maybe the file pipe name becomes invalid somehow after the job number hits 10k? |
As an update to this comment, it turns out in our case the root cause was Crowdstrike -- it had recently deployed script-based execution monitoring and this feature was blocking the FIFO file pipe used by AWX to start up the jobs with the needed SSH key. I opened a new issue for the AWX team to look at improving the FIFO pipe write so as to give users a better indication as to what is happening in case a hang were to happen. |
|
Nice find. Closing this in favor of your new issue. |
|
I think #11453 is the new issue. |
|
I've posted a monkey-patch (please read the final ATTENTION) to by-pass that limitation if you don't want to disable Crowdstrike or tune configuration at enterprise level. |
|
the crowdstrike issue seems fixed with the update 6.32.12905 |
ISSUE TYPE
SUMMARY
Last week we encountered a strange problem that our AWX jobs wouldn't run anymore. They were stuck in the running (or pending) state, but nothing seems to happen. No output in the job logs, no awx-ee pods being started in our kubernetes cluster, no strange things in the logs. Cancelling and restarting the jobs didn't work either. I noticed that it was job ID 10000 that was de first job which was stuck in running state.
I created a new database for AWX, connected our deployement to this new database and now our jobs could complete again. Changing it back to the old database stuck the jobs, so it must had something to do with AWX or the database. Yesterday I created a test setup with 10 git based projects and a script which would sync those projects periodically via the api. This morning it reached job ID 10000 again, same result. Jobs with id greater then 9999 are started but nothing seems to happen.
ENVIRONMENT
STEPS TO REPRODUCE
Run 9999 jobs in AWX (project sync's, inventory syncs or job_templates, it doesn't matter)
Run an extra job and you would see that it is starting according to the UI, but it will never finish the job.
EXPECTED RESULTS
Job's with ID > 9999 finishing succesfully
ACTUAL RESULTS
Job's with ID > 9999 are stuck
ADDITIONAL INFORMATION
I could provide a database, but then I need to run the test again, with non production credentials. I'm not allowed to post a database dump containing our production credentials of course. Let me know if you need this dump.
The text was updated successfully, but these errors were encountered: