Worker container on Docker causes high cpu load #3125
Comments
|
I see the same behavior since updating to |
|
Please post the logs, none of the tasks are scheduled to run that often so the only way this could happen is if there are a lot of tasks in the queue, and in that case the CPU should eventually go down after they are all done. (I was also not able to reproduce this on compose or k8s) |
|
These are the logs after recreating the worker container, CPU stays at 100% without any updates to the logs. |
|
Try clearing the redis cache (if the container has no storage mounted, you can just re-create it) and see if it still happens |
|
Recreated both redis and the worker container, but nothing changed. |
|
Same issue here. Running in a compose stack. No change after restarting redis (no persistent data). |
Signed-off-by: Jens Langhammer <jens.langhammer@beryju.org> #3125
|
The next beta build in ~20 minutes will include the commit above, which adds the Create a file called ---
version: '3.4'
services:
flower:
image: ${AUTHENTIK_IMAGE:-ghcr.io/goauthentik/server}:${AUTHENTIK_TAG:-2022.8.1}
restart: unless-stopped
command: worker-status
environment:
AUTHENTIK_REDIS__HOST: redis
AUTHENTIK_POSTGRESQL__HOST: postgresql
AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
# AUTHENTIK_ERROR_REPORTING__ENABLED: "true"
# WORKERS: 2
env_file:
- .env
ports:
- "0.0.0.0:${AUTHENTIK_PORT_HTTP:-9001}:9000"Afterwards, you can access port 9001 and see which tasks are running |
|
Thank you, but where do I find beta images? Do I have to use a special image tag different from Obviously I'm running on an "old" But it looks like there's something wrong with the path to the GeoIP database. I'll chek the path. |
|
Even using a new and clean volume will produce the error |
|
See https://goauthentik.io/docs/installation/beta for beta images; also the GeoIP error should be unrelated to this |
|
That did the trick. After about 3 minutes Flower has shown over 1,100 successful tasks; almost any task is named Processed number of completed tasks
Again about 2 minutes later there are only a few tasks per minute and the cpu usage looks quite good again. |
|
That is very interesting, I assume you upgraded to 8.1 and havent run a worker since the upgrade so a lot of those might have been queued? Does the pattern of CPU load repeat afterwards? |
|
Almost right. I've upgraded from 7.3 to 8.1 and haven't run a worker. But I had to switch to the beta image and after bringing the stack up again there still were about 1,100 tasks in queue. It looks like all the tasks have completed successfully. The CPU load return to known load after the "boot storm". ;-) The heavy and continuing CPU load pattern does not repeat anymore. Worker and Flower are still running and everything looks fine. Shall I switch back to stable release / image or do you need further information for investigation? |
|
Feel free to run whichever version, I'm curious to see if the problem happens again on 2022.8.1 since that was still causing issues from what I understood? |
|
No change on my side running the beta image. CPU still sits at 100%, without Flower showing any active tasks, only 11 succeeded.
|
|
Switching to beta and recreating the redis contianer did solve the issue. However, it does appear that over a hundred blueprint tasks are being spanwed and completed every container start. Should this be happening every time? Follow up: CPU spiked up to 300% (100% on 3 cores) when I came back to it today. Flower was already disabled so I couldn't look at it, so I've re-enabled it and recreated the containers to investigate later if it happens again. Results after: 300% again, with these tasks (unchanging for a while): |
|
Same issue here. Running 2022.8.2 stable. Deleting Redis cache did not change anything. |
|
Upgrading to 2022.09 solved the issue for me. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
This problem is happening for me still, I'm getting an endless loop of in strace. Not sure what's going on there. |
|
What version are you running? Can you restart the container with Flower enabled as described in this thread and see if there are many events running? You can also try to disable all blueprints and restart the container to see if the issue is still present. If it is still happening, the problem is likely unrelated to this issue (which seemed to be mostly caused by blueprint application tasks). |
|
I've opened a new issue for this at #4707 . Flower reports no tasks running. It appears to be an issue specifically with celerybeat. |
|
Well, reverting to version
|
|
Still an issue with 2023.8.3 |
Describe the bug
Right after starting up my docker-compose setup based on the given
docker-compose.ymlfile the worker-container causes high cpu load. It looks like the system tasks will be fired continuously every second.To Reproduce
Steps to reproduce the behavior:
docker-compose updocker-compose topdocker-compose stop workerExpected behavior
I would expect the system tasks would not be fired every second or continuously and not consuming so much cpu.
Screenshots
You can see the high cpu load in
btop.Logs
Version and Deployment (please complete the following information):
Additional context
Right now I cannot find any errors in the logs. I use JumpCloud as a LDAP-backend.
Any help or assistance is appreciated.
KR,
MiGoller
The text was updated successfully, but these errors were encountered: