multi-queue deal with different tasks

[big91987]

dear all,
here're my requirements: I have different hardware types of workers and want to match different tasks .
Different worker are used to handle tasks of different queues( eg worker with tag 'cpu' 'gpu' 'npu' deals separately with tasks such as 'calculate softmax' form 3 queue of {'cup_queue', 'gpu_queue', 'npu_queue'} )
how can I impl this process？ thx a lot 😃
i found a example of celery， is there any Similar usage in asynq？ https://stackoverflow.com/questions/51631455/how-to-route-tasks-to-different-queues-with-celery-and-django

[hibiken]

Hi @big91987 👋 Thank you for opening a discussion!

Let me make sure if I got your question correctly:

• You have a worker (aka asynq.Server) which needs to handle tasks from three queues: ('cup_queue', 'gpu_queue', 'npu_queue')
• Task of type 'calculate softmax' can be enqueued to any of these three queues

Is that correct?

[big91987]

Yeah, in fact, it is planned that my service will have three types of workers（Parts CPU，parts GPU and parts NPU ）to handle separately tasks in the three queues

[hibiken]

Ok, then here's what the configuration would look like:

srv := asynq.NewServer(redisConn, asynq.Config{
     Queues: mapa[string]{
         "cpu_queue": 1,
         "gpu_queue": 1,
         "npu_queue": 1,
     },
     // ... other config fields
})


mux := asynq.NewServeMux()
mux.HandleFunc("calculate_softmax", handleCalculateSoftmax)

if err := srv.Run(mux); err != nil {
     log.Fatalf("Failed to run a server: %v", err)
}

and if you need to know which queue the task came from, you can get that from the context:

func handleCalculateSoftmax(ctx context.Context, t *asynq.Task) error {
     qname, ok := asynq.GetQueueName(ctx)
     // ...rest of the handler code
}

Let me know if you have questions :)

[big91987]

ok thx a lot ~ I will try it 🤝