Why do all sensors poll? #12712
agon-shabi
started this conversation in
Ideas
Replies: 1 comment 1 reply
-
Thanks for bringing this up @agon-shabi - we've had discussions about this as well, and this is the direction that we'd ultimately like to go in. |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I understand for some sensors which are designed to periodically check some external state, but for run status sensors, or the asset reconciliation sensors - which you could say are logically "driven" by new events in the dagster-owned database - doesn't it seem slightly wasteful, and introduce unnecessary delays in orchestration?
I certainly get this feeling when looking at a page full of 30-secondly sensor ticks, where only one of them will yield a
RunRequest
on any given day (all of my jobs are daily-partitioned). Same when I see a job complete and have to wait between 0 and 30 seconds for the next sensor-driven job to start.This may already have occurred to you, and I might be talking out of my backside as I have next to no hands-on experience with postgres, but shouldn't it be possible for each sensor-running process to create a "view" on the database, subscribe to updates, and respond to updates as they happen?
This "view" would be equivalent to the query that the current sensors already periodically execute, so I guess you wouldn't even be adding extra load onto the db server - this might even end up reducing it, not to mention you won't have to worry about storing details of every sensor tick.
I'm sure there are some challenges to solve here around idempotency and sensor resiliency, but I can't imagine any show-stoppers.
Anyway, love the product and the ideas coming out of this camp, thanks!
Beta Was this translation helpful? Give feedback.
All reactions