Use monotonic clock for OpenAIBatchTrigger polling timeout#69534
Open
YAshhh29 wants to merge 1 commit into
Open
Use monotonic clock for OpenAIBatchTrigger polling timeout#69534YAshhh29 wants to merge 1 commit into
YAshhh29 wants to merge 1 commit into
Conversation
3e9bc0f to
94aeeb2
Compare
Author
|
Fixed the CI failures:
Verified the trigger under a real event loop locally. Drafted-by: GitHub Copilot (Claude Opus 4.6); reviewed by @YAshhh29 before posting |
94aeeb2 to
82da305
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OpenAIBatchTriggermeasures its polling timeout withtime.time()(wallclock) in two places, and
OpenAITriggerBatchOperatorsets the deadlinewith
time.time() + self.timeout. Because wall-clock time can jump — NTPcorrections, DST, container clock skew, VM pause/resume — a deferred batch
task can either time out early or run far past its intended deadline.
This isn't tied to an existing issue — I spotted it while auditing
time.time()/time.monotonic()usage across the AI/ML providers, in thesame spirit as the dep-audit that produced #69408.
Airflow's own coding standards flag this exact pattern:
Why the fix isn't a one-line swap
time.monotonic()values are only meaningful within a single process. Theoperator (in the worker) and the trigger (in the Triggerer) run in
different processes, so the operator cannot pre-compute a monotonic
deadline and hand it to the trigger. The trigger must call
time.monotonic()itself.To make that possible the trigger's preferred constructor argument
changed from
end_time(absolute wall-clock deadline) totimeout(aduration in seconds). The trigger now records
time.monotonic()at thestart of
run()and reports the elapsed monotonic duration on timeout.Backward compatibility
OpenAIBatchTrigger.__init__still accepts the legacyend_timeargument so that triggers serialized by the previous version of the
operator continue to run after an upgrade. When a legacy
end_timeispresent the trigger derives a best-effort remaining duration from the
wall clock once, then tracks the rest with the monotonic clock — so even
the legacy path is no longer fully at the mercy of wall-clock jumps
inside the polling loop.
serialize()preserves whichever argument thetrigger was constructed with, so a rolling upgrade never rewrites an
in-flight trigger's schema.
What changes
providers/openai/src/.../triggers/openai.pytimeout: float | Noneconstructor arg and validation(exactly one of
timeout/end_timemust be given).run()measures elapsed time viatime.monotonic().serialize()emits whichever field the trigger was constructed with.it reported
time.time() - self.end_time, which is seconds pastthe deadline — misleading and sometimes negative).
providers/openai/src/.../operators/openai.pytimeout=self.timeoutinstead of computingend_time.import time.providers/openai/tests/.../test_openai.pytimeout=.test_serialization_with_legacy_end_timeproves thebackward-compat serialization path.
test_timeout_uses_monotonic_not_wall_clock— the regressiontest — patches
time.monotonicwith an ever-increasing counter andasserts the timeout fires while
time.timeis never consulted. Itfails against the pre-fix code, which decided the timeout from the
wall clock. (
time.monotonicis mocked withitertools.countrather than a fixed list because the asyncio event loop also calls
time.monotonicinternally and would exhaust a finite side_effect.)providers/openai/docs/changelog.rst: bug-fix note explaining thebehavior change and the
end_time→timeoutmigration.How to reproduce the original bug
OpenAITriggerBatchOperator(deferrable=True, timeout=600).backward wall-clock adjustment on the Triggerer host (e.g.
sudo date -s '10 minutes ago', or a large NTP step).self.end_time < time.time()is no longer true.The same pattern also fires early on forward clock jumps.
Related
.github/instructions/code-review.instructions.mdWas generative AI tooling used to co-author this PR?
Generated-by: GitHub Copilot (Claude Opus 4.6) following the guidelines