Skip to content

Use monotonic clock for OpenAIBatchTrigger polling timeout#69534

Open
YAshhh29 wants to merge 1 commit into
apache:mainfrom
YAshhh29:fix-openai-batch-trigger-monotonic-clock
Open

Use monotonic clock for OpenAIBatchTrigger polling timeout#69534
YAshhh29 wants to merge 1 commit into
apache:mainfrom
YAshhh29:fix-openai-batch-trigger-monotonic-clock

Conversation

@YAshhh29

@YAshhh29 YAshhh29 commented Jul 7, 2026

Copy link
Copy Markdown

OpenAIBatchTrigger measures its polling timeout with time.time() (wall
clock) in two places, and OpenAITriggerBatchOperator sets the deadline
with time.time() + self.timeout. Because wall-clock time can jump — NTP
corrections, DST, container clock skew, VM pause/resume — a deferred batch
task can either time out early or run far past its intended deadline.

This isn't tied to an existing issue — I spotted it while auditing
time.time()/time.monotonic() usage across the AI/ML providers, in the
same spirit as the dep-audit that produced #69408.

Airflow's own coding standards flag this exact pattern:

time.monotonic() for durations, not time.time().

.github/instructions/code-review.instructions.md and AGENTS.md

Why the fix isn't a one-line swap

time.monotonic() values are only meaningful within a single process. The
operator (in the worker) and the trigger (in the Triggerer) run in
different processes, so the operator cannot pre-compute a monotonic
deadline and hand it to the trigger. The trigger must call
time.monotonic() itself.

To make that possible the trigger's preferred constructor argument
changed from end_time (absolute wall-clock deadline) to timeout (a
duration in seconds). The trigger now records time.monotonic() at the
start of run() and reports the elapsed monotonic duration on timeout.

Backward compatibility

OpenAIBatchTrigger.__init__ still accepts the legacy end_time
argument so that triggers serialized by the previous version of the
operator continue to run after an upgrade. When a legacy end_time is
present the trigger derives a best-effort remaining duration from the
wall clock once, then tracks the rest with the monotonic clock — so even
the legacy path is no longer fully at the mercy of wall-clock jumps
inside the polling loop. serialize() preserves whichever argument the
trigger was constructed with, so a rolling upgrade never rewrites an
in-flight trigger's schema.

What changes

  • providers/openai/src/.../triggers/openai.py
    • Add timeout: float | None constructor arg and validation
      (exactly one of timeout/end_time must be given).
    • run() measures elapsed time via time.monotonic().
    • serialize() emits whichever field the trigger was constructed with.
    • Timeout error message now reports actual elapsed seconds (previously
      it reported time.time() - self.end_time, which is seconds past
      the deadline — misleading and sometimes negative).
  • providers/openai/src/.../operators/openai.py
    • Passes timeout=self.timeout instead of computing end_time.
    • Removes the now-unused import time.
  • providers/openai/tests/.../test_openai.py
    • Existing tests now use timeout=.
    • New test_serialization_with_legacy_end_time proves the
      backward-compat serialization path.
    • New test_timeout_uses_monotonic_not_wall_clock — the regression
      test — patches time.monotonic with an ever-increasing counter and
      asserts the timeout fires while time.time is never consulted. It
      fails against the pre-fix code, which decided the timeout from the
      wall clock. (time.monotonic is mocked with itertools.count
      rather than a fixed list because the asyncio event loop also calls
      time.monotonic internally and would exhaust a finite side_effect.)
    • Constructor validation tests for the "neither" and "both" error paths.
  • providers/openai/docs/changelog.rst: bug-fix note explaining the
    behavior change and the end_timetimeout migration.

How to reproduce the original bug

  1. Deploy a DAG with OpenAITriggerBatchOperator(deferrable=True, timeout=600).
  2. While the batch is running and the trigger is deferred, force a
    backward wall-clock adjustment on the Triggerer host (e.g.
    sudo date -s '10 minutes ago', or a large NTP step).
  3. Observe that the trigger keeps polling well past 600 seconds because
    self.end_time < time.time() is no longer true.

The same pattern also fires early on forward clock jumps.

Related


Was generative AI tooling used to co-author this PR?
  • Yes — GitHub Copilot (Claude Opus 4.6)

Generated-by: GitHub Copilot (Claude Opus 4.6) following the guidelines

@YAshhh29

YAshhh29 commented Jul 8, 2026

Copy link
Copy Markdown
Author

Fixed the CI failures:

  1. The trigger tests mocked time.monotonic with a fixed 2-element list, but the asyncio event loop calls time.monotonic internally — it exhausted the mock and raised StopIteration (which is why only the non-DB / lowest-deps / compat jobs failed; the trigger tests are deselected in the DB backends). Switched to an ever-increasing itertools.count side_effect that the loop can call freely; the trigger's two clock reads are consecutive so measured elapsed is deterministic.
  2. Fixed British serialised → American serialized spellings that failed the docs spellcheck.

Verified the trigger under a real event loop locally.


Drafted-by: GitHub Copilot (Claude Opus 4.6); reviewed by @YAshhh29 before posting

@YAshhh29 YAshhh29 force-pushed the fix-openai-batch-trigger-monotonic-clock branch from 94aeeb2 to 82da305 Compare July 8, 2026 17:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant