Skip to content

Validate trigger events in Anthropic deferrable tasks#69379

Merged
kaxil merged 2 commits into
apache:mainfrom
takayoshi-makabe:add-execute-complete-methods-anthoropic-provider
Jul 6, 2026
Merged

Validate trigger events in Anthropic deferrable tasks#69379
kaxil merged 2 commits into
apache:mainfrom
takayoshi-makabe:add-execute-complete-methods-anthoropic-provider

Conversation

@takayoshi-makabe

Copy link
Copy Markdown
Contributor

The Anthropic provider's three deferrable resume paths — AnthropicBatchOperator.execute_complete, AnthropicBatchSensor.execute_complete and AnthropicAgentSessionOperator.execute_complete — used the trigger event without validating it:

  • event=None: happens when a deferred task resumes without a payload — a custom or subclassed trigger that yields TriggerEvent(None), or an event payload lost between the triggerer and the worker. Crashed with an opaque
    TypeError: 'NoneType' object is not subscriptable.
  • Unrecognised status: happens on triggerer/worker version skew (a task defers under one provider version and resumes under another whose trigger emits a different status vocabulary) or with a custom trigger. These methods branch deny-list style (timeout/error raise, everything else succeeds), so an unknown status fell through into the success path — the task was marked SUCCESS even though the Anthropic batch/session outcome was unknown.

This adds a validate_execute_complete_event() helper applied at the top of all three methods. The None check mirrors the amazon provider's helper of the same name (https://github.com/apache/airflow/blob/main/providers/amazon/src/airflow/providers/amazon/aws/utils/__init__.py#L70-L75).
The status check plays the role amazon's call sites cover with their if validated_event["status"] != "success": raise allow-list branching (https://github.com/apache/airflow/blob/main/providers/amazon/src/airflow/providers/amazon/aws/operators/emr.py#L207-L209)

Note: event validation across other providers is inconsistent — databricks has a required-keys helper (validate_trigger_event), while e.g. google and dbt.cloud access the event unvalidated with the same deny-list branching. This PR only fixes the Anthropic provider.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Claude Code


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@Vamsi-klu

Copy link
Copy Markdown
Contributor

Could the helper also validate the keys each resume path needs, not just status? An event like {"status": "success"} passes validate_execute_complete_event() but then the operator/sensor hits KeyError on session_id, batch_id, or request_counts. It would be cleaner for AnthropicTriggerEventError to cover malformed allowed-status events too, with tests for missing required keys per execute_complete path.


Drafted-by: Codex (GPT-5); reviewed by @Vamsi-klu before posting

@eladkal eladkal requested a review from kaxil July 6, 2026 16:23
Comment thread providers/anthropic/src/airflow/providers/anthropic/hooks/anthropic.py Outdated
Comment thread providers/anthropic/src/airflow/providers/anthropic/hooks/anthropic.py Outdated
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
@kaxil kaxil requested a review from gopidesupavan as a code owner July 6, 2026 18:00
@kaxil kaxil merged commit 1597221 into apache:main Jul 6, 2026
79 checks passed
@takayoshi-makabe takayoshi-makabe deleted the add-execute-complete-methods-anthoropic-provider branch July 6, 2026 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants