Skip to content

Pass verify and botocore_config to BatchJobTrigger in BatchSensor (#72278) - #72449

Open
nickgarza-google-tse wants to merge 1 commit into
apache:mainfrom
nickgarza-google-tse:fix-batch-sensor-defer-args-issue-72278
Open

Pass verify and botocore_config to BatchJobTrigger in BatchSensor (#72278)#72449
nickgarza-google-tse wants to merge 1 commit into
apache:mainfrom
nickgarza-google-tse:fix-batch-sensor-defer-args-issue-72278

Conversation

@nickgarza-google-tse

Copy link
Copy Markdown

Closes: #72278
Related-to: #72144

Description

In BatchSensor, when running in deferrable mode, BatchJobTrigger was initialized passing only region_name, omitting verify and botocore_config.

Because BatchSensor inherits from AwsBaseSensor, it already captures these configurations during initialization. Without forwarding them to BatchJobTrigger, the triggerer process reconstructs the AWS Batch client hook using default connection settings, causing custom SSL certificates (verify) and custom botocore_config (such as timeouts, retries, and user agents) to be lost upon deferral.

This change:

  1. Passes verify=self.verify and botocore_config=self.botocore_config to BatchJobTrigger in BatchSensor.execute() upon deferral (aligning with BatchOperator).
  2. Adds unit test coverage asserting that region_name, verify, and botocore_config survive and persist into the serialized trigger payload.

Use Case / Motivation

Preserves custom AWS configuration (e.g., custom CA bundles/SSL settings and botocore connection parameters) across task deferral boundaries when polling AWS Batch jobs asynchronously.


Related Issues


Code Changes Summary

File Change
providers/amazon/src/airflow/providers/amazon/aws/sensors/batch.py Passed verify=self.verify and botocore_config=self.botocore_config to BatchJobTrigger.
providers/amazon/tests/unit/amazon/aws/sensors/test_batch.py Added test_execute_in_deferrable_mode_passes_aws_configs verifying parameters in serialized trigger dictionary.

Checklists

  • Are you creating a PR in a personal branch instead of main?
  • Tests are added/updated and pass locally using Breeze:
    • breeze testing providers-tests providers/amazon/tests/unit/amazon/aws/sensors/test_batch.py
    • breeze testing providers-tests providers/amazon/tests/unit/amazon/aws/triggers/test_batch.py
  • Static checks pass:
    • breeze static-checks --last-commit

@boring-cyborg

boring-cyborg Bot commented Sep 2, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@SameerMesiah97 SameerMesiah97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have left my comments. It would great if a maintainer could trigger CI to ensure any breaking changes are caught.

aws_conn_id=self.aws_conn_id,
region_name=self.region_name,
verify=self.verify,
botocore_config=self.botocore_config,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would advise you to run the prek hook locally to catch formatting errors like this.

_, kwargs = trigger.serialize()
assert kwargs.get("region_name") == AWS_REGION
assert kwargs.get("verify") == "/custom/ca_bundle.pem"
assert kwargs.get("botocore_config") == {"read_timeout": 45}

@SameerMesiah97 SameerMesiah97 Sep 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think it is necessary to cover region_name as my understanding is that the intention of this PR is to forward the 2 additional arguments i.e. verify and botocore_config.

@vincbeck

vincbeck commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Forward hook config to BatchJobTrigger from the BatchSensor

3 participants