Repository navigation
Is it good to have a lot of sensors to monitor jobs run? #34401
|
Hello, I am working on data engineering project which ingest and transform a lot of data from various source to the datalake. But the jobs scheduled didn't run everyday, by the way sensors are run interval every 30 sec. My question is the increasing number of sensors will effect to the resource of the container or not? And How should I do to enhance it? |
Replies: 1 comment 2 replies
|
Hi @praepanwa-ted ! A single sensor is light, but every sensor is checked by the dagster-daemon on a loop (every 30 sec in your case), and each check calls your code location. So with many chained sensors, there is always some background work running, even on days when no job runs. Also, each step in the chain can wait up to one interval before the next job starts. Some options I would suggest, from simple to more "Dagster way": 1. Keep steps that always run together in the same job (or make them dependent assets). Then you don't need a sensor between them at all. 2. If you still need sensors, use one run status sensor for many jobs instead of one sensor per job. Since your jobs don't run every day, you can also increase the interval: 3. Use assets with Declarative Automation, so Dagster starts the downstream work for you, with no chain of sensors:
4. If you want to keep many sensors, you can let the daemon run them in parallel in But in my experience, reducing the number of sensors (options 1 to 3) helps much more than adding workers. Hope this helps! |
Hi @praepanwa-ted ! A single sensor is light, but every sensor is checked by the dagster-daemon on a loop (every 30 sec in your case), and each check calls your code location. So with many chained sensors, there is always some background work running, even on days when no job runs. Also, each step in the chain can wait up to one interval before the next job starts.
Some options I would suggest, from simple to more "Dagster way":
1. Keep steps that always run together in the same job (or make them dependent assets). Then you don't need a sensor between them at all.
2. If you still need sensors, use one run status sensor for many jobs instead of one sensor per job. Since your jobs don't run…