Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] The task cannot be resumed after the worker is shut down while the task is running #525

Open
zijiwork opened this issue Aug 3, 2022 · 2 comments
Assignees
Labels
bug Something isn't working

Comments

@zijiwork
Copy link

zijiwork commented Aug 3, 2022

Describe the bug
When the task is running, I try to shut down the worker, but restarting the worker task cannot recover normally, and the status of the Asynq Monitoring task is active. and cannot be closed

To Reproduce
Steps to reproduce the behavior (Code snippets if applicable):

go tasks.TaskWorker()

TaskWorker

	// start server
	if err := srv.Start(mux); err != nil {
		common.LOG.Error(fmt.Sprintf("could not start server: %v", err))
	}

	quit := make(chan os.Signal, 1)
	signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)
	<-quit // wait for termination signal
	log.Println("[asynq] Receive exit signal task worker exit")
	srv.Shutdown()

try restarting the asynq worker

2022/08/03 - 14:29:31.445       INFO    /Users/micheng/code/luban/controller/application/deploy.go:297  2022-08-03 14:29:31 - [task-rpc] - [updated_replicas:1, replicas:2, available_replicas:1, observed_generation:189] waiting...
2022/08/03 - 14:29:34.577       INFO    /Users/micheng/code/luban/controller/application/deploy.go:297  2022-08-03 14:29:34 - [task-rpc] - [updated_replicas:1, replicas:2, available_replicas:1, observed_generation:189] waiting...
2022/08/03 - 14:29:37.699       INFO    /Users/micheng/code/luban/controller/application/deploy.go:297  2022-08-03 14:29:37 - [task-rpc] - [updated_replicas:1, replicas:2, available_replicas:1, observed_generation:189] waiting...
2022/08/03 - 14:29:40.803       INFO    /Users/micheng/code/luban/controller/application/deploy.go:297  2022-08-03 14:29:40 - [task-rpc] - [updated_replicas:1, replicas:2, available_replicas:1, observed_generation:189] waiting...
[GIN] 2022/08/03 - 14:29:41 | 200 |   26.725875ms |             ::1 | GET      "/task/monitor/api/queues/critical/completed_tasks?page=1&size=100"
Receive SIGTERM ,exiting gracefully....
2022/08/03 14:29:43 [asynq] Receive exit signal task scheduler exit
2022/08/03 14:29:43 [asynq] Receive exit signal task worker exit
asynq: pid=49878 2022/08/03 06:29:43.051790 INFO: Starting graceful shutdown
asynq: pid=49878 2022/08/03 06:29:43.051778 INFO: Scheduler shutting down
2022/08/03 14:29:43 [agent] Receive exit signal web server exit
2022/08/03 14:29:43 Shutting down server...
2022/08/03 14:29:43 [agent] Receive exit signal rpc server exit

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
image

Environment (please complete the following information):

  • OS: MacOS
  • Version asynq v0.19.0 asynqmon v0.4.0
@zijiwork zijiwork added the bug Something isn't working label Aug 3, 2022
@dxl3811051
Copy link

Hello, I want to ask you how to stop a running task. I click the stop button, but the task is still running. My task is a simple loop print

@liran
Copy link

liran commented Jun 27, 2023

I've also had situations where tasks get stuck in active state and can't be resumed.
asynq version 0.24.1, redis 7.0.7 cluster
image

These 3 tasks are in the active list
image

But the task state is still pending
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants