Starting Redash with ECS Fargate causes [CRITICAL] WORKER TIMEOUT #5977

Watson-Sei · 2023-04-08T08:13:10Z

Watson-Sei
Apr 8, 2023

I'm trying to build Redash on AWS ECS. Set up PostgreSQL and Redis in advance. Then I ran create_db to initialize the database and deploy workers and servers. But when I checked the worker's logs, I found the following logs: I have no idea where these are coming from. help. Also, could this Worker issue be causing the server to timeout? I'm building an infrastructure with aws cdk go. Please help me find the cause from the following repository.

worker log

2023/4/8 14:45:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:45:00,704 ERRO pool worker_healthcheck event buffer overflowed, discarding event 24	redash-worker
2023/4/8 14:44:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:44:00,641 ERRO pool worker_healthcheck event buffer overflowed, discarding event 23	redash-worker
2023/4/8 14:43:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:43:00,576 ERRO pool worker_healthcheck event buffer overflowed, discarding event 22	redash-worker
2023/4/8 14:42:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:42:00,511 ERRO pool worker_healthcheck event buffer overflowed, discarding event 21	redash-worker
2023/4/8 14:41:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:41:00,449 ERRO pool worker_healthcheck event buffer overflowed, discarding event 20	redash-worker
2023/4/8 14:40:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:40:00,385 ERRO pool worker_healthcheck event buffer overflowed, discarding event 19	redash-worker
2023/4/8 14:39:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:39:00,323 ERRO pool worker_healthcheck event buffer overflowed, discarding event 18	redash-worker
2023/4/8 14:38:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:38:00,259 ERRO pool worker_healthcheck event buffer overflowed, discarding event 17	redash-worker
2023/4/8 14:37:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:37:00,194 ERRO pool worker_healthcheck event buffer overflowed, discarding event 16	redash-worker
2023/4/8 14:36:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:36:00,131 ERRO pool worker_healthcheck event buffer overflowed, discarding event 15	redash-worker
2023/4/8 14:35:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:35:00,067 ERRO pool worker_healthcheck event buffer overflowed, discarding event 14	redash-worker
2023/4/8 14:34:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:34:00,003 ERRO pool worker_healthcheck event buffer overflowed, discarding event 13	redash-worker
2023/4/8 14:33:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:33:00,939 ERRO pool worker_healthcheck event buffer overflowed, discarding event 12	redash-worker
2023/4/8 14:32:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:32:00,876 ERRO pool worker_healthcheck event buffer overflowed, discarding event 11	redash-worker
2023/4/8 14:31:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:31:00,809 ERRO pool worker_healthcheck event buffer overflowed, discarding event 10	redash-worker
2023/4/8 14:30:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:30:00,742 ERRO pool worker_healthcheck event buffer overflowed, discarding event 9	redash-worker
2023/4/8 14:29:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:29:00,678 ERRO pool worker_healthcheck event buffer overflowed, discarding event 8	redash-worker
2023/4/8 14:28:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:28:00,615 ERRO pool worker_healthcheck event buffer overflowed, discarding event 7	redash-worker
2023/4/8 14:27:00	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:27:00,552 ERRO pool worker_healthcheck event buffer overflowed, discarding event 6	redash-worker
2023/4/8 14:16:01	79fa72ca83e1406d97e21faf3c790512 	2023/04/08 05:16:01 [worker_healthcheck] Performing `RQ Worker Healthcheck` check for process name worker-0	redash-worker
2023/4/8 14:16:01	79fa72ca83e1406d97e21faf3c790512 	2023/04/08 05:16:01 [worker_healthcheck] Performing `RQ Worker Healthcheck` check for process name worker-1	redash-worker
2023/4/8 14:16:01	79fa72ca83e1406d97e21faf3c790512 	2023/04/08 05:16:01 [worker_healthcheck] Received TICK_60 event from supervisor	redash-worker
2023/4/8 14:15:46	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:15:46,835 INFO success: worker-0 entered RUNNING state, process has stayed up for > than 300 seconds (startsecs)	redash-worker
2023/4/8 14:15:46	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:15:46,835 INFO success: worker-1 entered RUNNING state, process has stayed up for > than 300 seconds (startsecs)	redash-worker
2023/4/8 14:15:01	79fa72ca83e1406d97e21faf3c790512 	2023/04/08 05:15:01 [worker_healthcheck] No processes in state RUNNING found for process worker	redash-worker
2023/4/8 14:15:01	79fa72ca83e1406d97e21faf3c790512 	RESULT 2	redash-worker
2023/4/8 14:15:01	79fa72ca83e1406d97e21faf3c790512 	OKREADY	redash-worker
2023/4/8 14:15:01	79fa72ca83e1406d97e21faf3c790512 	2023/04/08 05:15:01 [worker_healthcheck] Received TICK_60 event from supervisor	redash-worker
2023/4/8 14:14:01	79fa72ca83e1406d97e21faf3c790512 	2023/04/08 05:14:01 [worker_healthcheck] No processes in state RUNNING found for process worker	redash-worker
2023/4/8 14:14:01	79fa72ca83e1406d97e21faf3c790512 	RESULT 2	redash-worker
2023/4/8 14:14:01	79fa72ca83e1406d97e21faf3c790512 	OKREADY	redash-worker
2023/4/8 14:14:01	79fa72ca83e1406d97e21faf3c790512 	2023/04/08 05:14:01 [worker_healthcheck] Received TICK_60 event from supervisor	redash-worker
2023/4/8 14:13:01	79fa72ca83e1406d97e21faf3c790512 	RESULT 2	redash-worker
2023/4/8 14:13:01	79fa72ca83e1406d97e21faf3c790512 	OKREADY	redash-worker
2023/4/8 14:13:01	79fa72ca83e1406d97e21faf3c790512 	2023/04/08 05:13:01 [worker_healthcheck] No processes in state RUNNING found for process worker	redash-worker
2023/4/8 14:13:01	79fa72ca83e1406d97e21faf3c790512 	2023/04/08 05:13:01 [worker_healthcheck] Received TICK_60 event from supervisor	redash-worker
2023/4/8 14:12:01	79fa72ca83e1406d97e21faf3c790512 	2023/04/08 05:12:01 [worker_healthcheck] No processes in state RUNNING found for process worker	redash-worker
2023/4/8 14:12:01	79fa72ca83e1406d97e21faf3c790512 	RESULT 2	redash-worker
2023/4/8 14:12:01	79fa72ca83e1406d97e21faf3c790512 	OKREADY	redash-worker
2023/4/8 14:12:01	79fa72ca83e1406d97e21faf3c790512 	2023/04/08 05:12:01 [worker_healthcheck] Received TICK_60 event from supervisor	redash-worker
2023/4/8 14:11:00	79fa72ca83e1406d97e21faf3c790512 	2023/04/08 05:11:00 [worker_healthcheck] No processes in state RUNNING found for process worker	redash-worker
2023/4/8 14:11:00	79fa72ca83e1406d97e21faf3c790512 	RESULT 2	redash-worker
2023/4/8 14:11:00	79fa72ca83e1406d97e21faf3c790512 	OKREADY	redash-worker
2023/4/8 14:11:00	79fa72ca83e1406d97e21faf3c790512 	READY	redash-worker
2023/4/8 14:11:00	79fa72ca83e1406d97e21faf3c790512 	2023/04/08 05:11:00 [worker_healthcheck] Received TICK_60 event from supervisor	redash-worker
2023/4/8 14:11:00	79fa72ca83e1406d97e21faf3c790512 	2023/04/08 05:11:00 [worker_healthcheck] Starting the health check for worker process Checks config: [(<class 'redash.cli.rq.WorkerHealthcheck'>, {})]	redash-worker
2023/4/8 14:11:00	79fa72ca83e1406d97e21faf3c790512 	2023/04/08 05:11:00 [worker_healthcheck] Installing signal handlers.	redash-worker
2023/4/8 14:10:47	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:10:47,590 INFO success: worker_healthcheck entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)	redash-worker
2023/4/8 14:10:46	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:10:46,588 INFO spawned: 'worker-1' with pid 24	redash-worker
2023/4/8 14:10:46	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:10:46,585 INFO spawned: 'worker_healthcheck' with pid 22	redash-worker
2023/4/8 14:10:46	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:10:46,587 INFO spawned: 'worker-0' with pid 23	redash-worker
2023/4/8 14:10:45	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:10:45,582 INFO supervisord started with pid 1	redash-worker
2023/4/8 14:10:45	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:10:45,582 INFO RPC interface 'supervisor' initialized	redash-worker
2023/4/8 14:10:45	79fa72ca83e1406d97e21faf3c790512 	2023-04-08 05:10:45,582 CRIT Server 'unix_http_server' running without any HTTP authentication checking	redash-worker
2023/4/8 14:10:45	79fa72ca83e1406d97e21faf3c790512 	Starting RQ worker...

server log

2023/4/8 14:15:02    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:15:02 +0000] [62] [INFO] Booting worker with pid: 62    redash-server
2023/4/8 14:15:01    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:15:01 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:50)    redash-server
2023/4/8 14:15:01    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:15:01 +0000] [50] [INFO] Worker exiting (pid: 50)    redash-server
2023/4/8 14:14:37    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:14:37 +0000] [58] [INFO] Booting worker with pid: 58    redash-server
2023/4/8 14:14:37    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:14:37 +0000] [13] [INFO] Worker exiting (pid: 13)    redash-server
2023/4/8 14:14:37    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:14:37 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:13)    redash-server
2023/4/8 14:14:32    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:14:32 +0000] [54] [INFO] Booting worker with pid: 54    redash-server
2023/4/8 14:14:31    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:14:31 +0000] [46] [INFO] Worker exiting (pid: 46)    redash-server
2023/4/8 14:14:31    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:14:31 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:46)    redash-server
2023/4/8 14:14:07    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:14:07 +0000] [50] [INFO] Booting worker with pid: 50    redash-server
2023/4/8 14:14:07    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:14:07 +0000] [16] [INFO] Worker exiting (pid: 16)    redash-server
2023/4/8 14:14:07    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:14:07 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:16)    redash-server
2023/4/8 14:13:40    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:13:40 +0000] [46] [INFO] Booting worker with pid: 46    redash-server
2023/4/8 14:13:39    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:13:39 +0000] [14] [INFO] Worker exiting (pid: 14)    redash-server
2023/4/8 14:13:39    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:13:39 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:14)    redash-server
2023/4/8 14:12:51    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:12:51 +0000] [16] [INFO] Booting worker with pid: 16    redash-server
2023/4/8 14:12:51    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:12:51 +0000] [15] [INFO] Booting worker with pid: 15    redash-server
2023/4/8 14:12:51    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:12:51 +0000] [14] [INFO] Booting worker with pid: 14    redash-server
2023/4/8 14:12:51    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:12:51 +0000] [13] [INFO] Booting worker with pid: 13    redash-server
2023/4/8 14:12:51    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:12:51 +0000] [1] [INFO] Using worker: sync    redash-server
2023/4/8 14:12:51    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:12:51 +0000] [1] [INFO] Starting gunicorn 20.0.4    redash-server
2023/4/8 14:12:51    f214c38b7a6b49fab211fe1c688b5389     [2023-04-08 05:12:51 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000/ (1)

cdk repository describing the ECS configuration in question:
https://github.com/Watson-Sei/aws-cdk-provider-redash

justinclift · 2023-04-09T09:23:22Z

justinclift
Apr 9, 2023
Maintainer

Looking at this, it's unclear if this is using a specific release of Redash, or if it's building from the latest master branch of the Redash GitHub repository.

If it's building from the master branch, then I wonder if that is some unforeseen consequence of this recent commit? 5b9fd40

8 replies

Watson-Sei Apr 10, 2023
Author

@justinclift

I tried to launch the container again using the preview, but the worker timeout on the server side was not resolved. I also set it to 600 seconds. I don't understand why I can't start the server side so much.

justinclift Apr 10, 2023
Maintainer

Yeah, that's weird.

Sorry I haven't been of more help. I'm far more used to doing stuff with Redash on non-AWS infrastructure. 😦

Watson-Sei Apr 11, 2023
Author

I am using the cluster type of aws elasticache for redis, does the process that handles redis in redash support clusters?

justinclift Apr 11, 2023
Maintainer

No idea personally. Redash is the only thing that I use that has anything to do with Redis, and none of the hosted Newdash customers are anywhere near the size of needing clustered Redash.

Far before any kind of need to cluster Redis, the unreasonable memory footprint of Python workers becomes problematic for higher usage installations. 😉 Am working on fixing that particular problem, but it's a months-long endeavour. 😄

Watson-Sei Apr 11, 2023
Author

It's hard to wait for worker optimization, so I tried increasing cpu and memory. worker timeout no longer occurs, but the server side still never succeeds in target Group health check😢 Ignore if not relevant

change cpu and memory value:

Cpu:            jsii.Number(4096),
MemoryLimitMiB: jsii.Number(8192),

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Starting Redash with ECS Fargate causes [CRITICAL] WORKER TIMEOUT #5977

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Starting Redash with ECS Fargate causes [CRITICAL] WORKER TIMEOUT #5977

Watson-Sei Apr 8, 2023

Replies: 1 comment · 8 replies

justinclift Apr 9, 2023 Maintainer

Watson-Sei Apr 10, 2023 Author

justinclift Apr 10, 2023 Maintainer

Watson-Sei Apr 11, 2023 Author

justinclift Apr 11, 2023 Maintainer

Watson-Sei Apr 11, 2023 Author

Watson-Sei
Apr 8, 2023

Replies: 1 comment 8 replies

justinclift
Apr 9, 2023
Maintainer

Watson-Sei Apr 10, 2023
Author

justinclift Apr 10, 2023
Maintainer

Watson-Sei Apr 11, 2023
Author

justinclift Apr 11, 2023
Maintainer

Watson-Sei Apr 11, 2023
Author