Add system tests for Dataproc trigger on_kill cancel behavior by srchilukoori · Pull Request #65982 · apache/airflow

srchilukoori · 2026-04-27T22:39:15Z

Follow-up to #65742 as requested by @shahar1. Adds a system test DAG that exercises the cancel_on_kill behavior introduced when DataprocSubmitTrigger and DataprocSubmitJobDirectTrigger migrated to BaseTrigger.on_kill().

Changes:

New file: providers/google/tests/system/google/cloud/dataproc/example_dataproc_cancel_on_kill.py

The DAG contains two tests sharing a single Dataproc cluster:

Test	What it does	What it proves
A (happy path)	Deferrable submit with `cancel_on_kill=True` runs SparkPi to completion	Trigger plumbing with `cancel_on_kill=True` does not break normal deferrable execution
B (cancel path)	Submits a long-running SparkPi (1M slices) asynchronously, cancels it via `DataprocHook.cancel_job()`, asserts `CANCELLED` state	The `cancel_job()` call that `on_kill()` delegates to actually cancels running Dataproc jobs

Testing:

System tests require GCP credentials and a project with Dataproc API enabled. I do not have access to a GCP environment to run this test end-to-end. The test was verified via:

python3 -m py_compile — syntax OK
ruff check / ruff format — all checks passed
All pre-commit hooks passed
Pattern consistency verified against existing Dataproc system tests (example_dataproc_spark_deferrable.py, example_dataproc_spark_async.py, example_dataproc_start_from_trigger.py)

To run manually:

SYSTEM_TESTS_ENV_ID=<id> SYSTEM_TESTS_GCP_PROJECT=<project> \
  pytest providers/google/tests/system/google/cloud/dataproc/example_dataproc_cancel_on_kill.py -s

related: #65742

Was generative AI tooling used to co-author this PR?

Yes — Claude

shahar1

Great, thank you!

@VladaZakharova @MaksYermak - is it compatible for running on your environment? I'll be happy for your feedback.

VladaZakharova · 2026-04-28T07:14:49Z

hi
thank you for submitting PR.
In google provider we don't actually create a system test for every parameter we have, because then google provider will blow because of it's own size :)
I can see that you created this PR to test changes from another PR. I think in this case only good unit tests will be enough.

shahar1 · 2026-04-28T08:45:59Z

hi
thank you for submitting PR.
In google provider we don't actually create a system test for every parameter we have, because then google provider will blow because of it's own size :)
I can see that you created this PR to test changes from another PR. I think in this case only good unit tests will be enough.

Is it possible to retain it, but exclude from the Google automated tests?

Per reviewer feedback (apache#65982), exclude the cancel_on_kill system test from automated Google CI runs. Test now skips at collection time unless RUN_MANUAL_DATAPROC_CANCEL_ON_KILL_TEST=1 is set, allowing manual end-to-end runs without burdening automated infrastructure.

srchilukoori · 2026-04-29T14:06:36Z

Thanks @shahar1, made the change.

Added an env-var gate at module level in example_dataproc_cancel_on_kill.py:

if not os.environ.get("RUN_MANUAL_DATAPROC_CANCEL_ON_KILL_TEST"):
    pytest.skip("Manual-only system test: set RUN_MANUAL_DATAPROC_CANCEL_ON_KILL_TEST=1 to run.", allow_module_level=True)

How this works: by default the test is skipped at collection time, so automated runs (which don't set the env var) won't pick it up. To run manually, set RUN_MANUAL_DATAPROC_CANCEL_ON_KILL_TEST=1. Same pytest.skip(..., allow_module_level=True) pattern already used for "service not available" gates in example_bigquery_to_mysql.py etc.

Verified locally with pytest --collect-only --system:

without the env var: 0 tests collected (SKIPPED with the explanation message)
with RUN_MANUAL_DATAPROC_CANCEL_ON_KILL_TEST=1: 1 test collected

shahar1 · 2026-04-29T15:14:26Z

Thanks @shahar1, made the change.

Added an env-var gate at module level in example_dataproc_cancel_on_kill.py:
if not os.environ.get("RUN_MANUAL_DATAPROC_CANCEL_ON_KILL_TEST"):
    pytest.skip("Manual-only system test: set RUN_MANUAL_DATAPROC_CANCEL_ON_KILL_TEST=1 to run.", allow_module_level=True)
How this works: by default the test is skipped at collection time, so automated runs (which don't set the env var) won't pick it up. To run manually, set RUN_MANUAL_DATAPROC_CANCEL_ON_KILL_TEST=1. Same pytest.skip(..., allow_module_level=True) pattern already used for "service not available" gates in example_bigquery_to_mysql.py etc.

Verified locally with pytest --collect-only --system:

without the env var: 0 tests collected (SKIPPED with the explanation message)

with RUN_MANUAL_DATAPROC_CANCEL_ON_KILL_TEST=1: 1 test collected

Nice idea, but currently seems incompatible with other tests, as they fail - try to figure out how to work around that.

srchilukoori · 2026-04-29T21:24:17Z

Fixed in 678492e. Replaced pytest.skip(allow_module_level=True) with pytestmark = pytest.mark.skipif(...) — file imports cleanly now, test still skipped unless RUN_MANUAL_DATAPROC_CANCEL_ON_KILL_TEST=1 is set.

cc: @VladaZakharova

@shahar1

Follow-up to apache#65742 (requested by @shahar1). Adds a system test DAG that exercises the cancel_on_kill plumbing introduced when DataprocSubmitTrigger and DataprocSubmitJobDirectTrigger migrated to BaseTrigger.on_kill(). Test A: deferrable submit with cancel_on_kill=True completes normally. Test B: submits a long-running job, cancels it via DataprocHook.cancel_job() (the same call on_kill delegates to), and asserts CANCELLED state.

The bare `from airflow.decorators import task` import fails MyPy in the providers check because `airflow.decorators` does not export `task` in Airflow 3.x. Switch to the standard `AIRFLOW_V_3_0_PLUS` conditional import pattern used by all other system tests.

Per reviewer feedback (apache#65982), exclude the cancel_on_kill system test from automated Google CI runs. Test now skips at collection time unless RUN_MANUAL_DATAPROC_CANCEL_ON_KILL_TEST=1 is set, allowing manual end-to-end runs without burdening automated infrastructure.

pytest.skip(allow_module_level=True) raises a Skipped exception at import time that DagBag records as an import error, breaking the test_should_be_importable parametrized test in DB-core CI jobs. pytestmark = pytest.mark.skipif(...) lets the file import cleanly (DagBag happy, importability test passes) while still skipping test_run when RUN_MANUAL_DATAPROC_CANCEL_ON_KILL_TEST is unset.

potiuk · 2026-05-05T16:13:07Z

@srchilukoori — Your unresolved review thread(s) from @shahar1 appear to have been addressed (post-review commits and/or in-thread replies on every thread, with the latest commit pushed after the most recent thread). I've added the ready for maintainer review label so the PR re-enters the maintainer review queue.

@shahar1 — could you take another look when you have a chance? If you agree the feedback was addressed, please mark the threads as resolved so the queue signal stays accurate. If a thread still needs work, please reply in-line — @srchilukoori will follow up.

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

shahar1 · 2026-05-05T16:39:17Z

Merged from main, good to merge after CI is green

srchilukoori · 2026-05-06T12:38:44Z

Merged from main, good to merge after CI is green

@shahar1 all checks are green. Please merge when you get a minute.

@shahar1

…#65982) * Add system tests for Dataproc trigger on_kill cancel behavior Follow-up to apache#65742 (requested by @shahar1). Adds a system test DAG that exercises the cancel_on_kill plumbing introduced when DataprocSubmitTrigger and DataprocSubmitJobDirectTrigger migrated to BaseTrigger.on_kill(). Test A: deferrable submit with cancel_on_kill=True completes normally. Test B: submits a long-running job, cancels it via DataprocHook.cancel_job() (the same call on_kill delegates to), and asserts CANCELLED state. * Fix MyPy failure: use version-compat import pattern for task decorator The bare `from airflow.decorators import task` import fails MyPy in the providers check because `airflow.decorators` does not export `task` in Airflow 3.x. Switch to the standard `AIRFLOW_V_3_0_PLUS` conditional import pattern used by all other system tests. * Gate cancel_on_kill system test behind opt-in env var Per reviewer feedback (apache#65982), exclude the cancel_on_kill system test from automated Google CI runs. Test now skips at collection time unless RUN_MANUAL_DATAPROC_CANCEL_ON_KILL_TEST=1 is set, allowing manual end-to-end runs without burdening automated infrastructure. * Replace module-level pytest.skip with pytestmark skipif pytest.skip(allow_module_level=True) raises a Skipped exception at import time that DagBag records as an import error, breaking the test_should_be_importable parametrized test in DB-core CI jobs. pytestmark = pytest.mark.skipif(...) lets the file import cleanly (DagBag happy, importability test passes) while still skipping test_run when RUN_MANUAL_DATAPROC_CANCEL_ON_KILL_TEST is unset. * Rename env flag to RUN_MANUAL_GOOGLE_SYSTEM_TESTS for generalization --------- Co-authored-by: Sadha Chilukoori <ssreddy.8555@gmail.com> Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>

srchilukoori requested a review from shahar1 as a code owner April 27, 2026 22:39

boring-cyborg Bot added area:providers provider:google Google (including GCP) related issues labels Apr 27, 2026

srchilukoori force-pushed the test/dataproc-trigger-on-kill-system-tests branch from a99a3c6 to f13986d Compare April 28, 2026 00:51

shahar1 approved these changes Apr 28, 2026

View reviewed changes

Sadha Chilukoori added 4 commits May 2, 2026 11:56

srchilukoori force-pushed the test/dataproc-trigger-on-kill-system-tests branch from 678492e to 649d397 Compare May 2, 2026 18:56

shahar1 reviewed May 3, 2026

View reviewed changes

Comment thread providers/google/tests/system/google/cloud/dataproc/example_dataproc_cancel_on_kill.py Outdated

Rename env flag to RUN_MANUAL_GOOGLE_SYSTEM_TESTS for generalization

d9f467c

potiuk added the ready for maintainer review Set after triaging when all criteria pass. label May 5, 2026

Merge branch 'main' into test/dataproc-trigger-on-kill-system-tests

7a34364

Merge branch 'main' into test/dataproc-trigger-on-kill-system-tests

55d3431

potiuk merged commit f38982c into apache:main May 10, 2026
94 checks passed

srchilukoori deleted the test/dataproc-trigger-on-kill-system-tests branch May 11, 2026 13:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add system tests for Dataproc trigger on_kill cancel behavior#65982

Add system tests for Dataproc trigger on_kill cancel behavior#65982
potiuk merged 7 commits into
apache:mainfrom
srchilukoori:test/dataproc-trigger-on-kill-system-tests

srchilukoori commented Apr 27, 2026 •

edited

Loading

Uh oh!

shahar1 left a comment

Uh oh!

VladaZakharova commented Apr 28, 2026

Uh oh!

shahar1 commented Apr 28, 2026

Uh oh!

srchilukoori commented Apr 29, 2026

Uh oh!

shahar1 commented Apr 29, 2026

Uh oh!

srchilukoori commented Apr 29, 2026 •

edited

Loading

Uh oh!

Uh oh!

potiuk commented May 5, 2026

Uh oh!

shahar1 commented May 5, 2026

Uh oh!

srchilukoori commented May 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

srchilukoori commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Was generative AI tooling used to co-author this PR?

Uh oh!

shahar1 left a comment

Choose a reason for hiding this comment

Uh oh!

VladaZakharova commented Apr 28, 2026

Uh oh!

shahar1 commented Apr 28, 2026

Uh oh!

srchilukoori commented Apr 29, 2026

Uh oh!

shahar1 commented Apr 29, 2026

Uh oh!

srchilukoori commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

potiuk commented May 5, 2026

Uh oh!

shahar1 commented May 5, 2026

Uh oh!

srchilukoori commented May 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

srchilukoori commented Apr 27, 2026 •

edited

Loading

srchilukoori commented Apr 29, 2026 •

edited

Loading