Status of testing Providers that were prepared on June 07, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.24.0rc2

• ECS Overrides for AWS Batch submit_job (#39903): @yehoshuadimarsky
• feat: implement amazon s3 to dynamodb transfer operator (#39654): @dondaum
  Linked issues:
  • Linked Issue #39377: @eladkal
• Adding Glue Data Quality Rule Recommendation Run (#40014): @gopidesupavan
• Allow user-specified object attributes to be used in check_fn for S3KeySensor (#39950): @ellisms
  Linked issues:
  • Linked Issue #39898: @eladkal
• Adding Amazon Glue Data Quality Service (#39923): @gopidesupavan
• Deduplicate model name in SageMakerTransformOperator (#39956): @mateuslatrova
  Linked issues:
  • Linked Issue #39502: @mxuan0
• fix: DynamoDBToS3Operator - remove process_func from templated_fields (#39948): @raphaelauv
• Fix aws assume role session creation when deferrable (#40051): @gopidesupavan
  Linked issues:
  • Linked Issue #39970: @arollet-decathlon
• Resolving ECS fargate deprecated warnings (#39834): @dirrao
• Resolving EMR notebook deprecated warnings (#39829): @dirrao
• Bump boto min versions (#40052): @ferruzzi
• Implement per-provider tests with lowest-direct dependency resolution (#39946): @potiuk
• Resolve aws emr deprecations in tests (#40020): @dirrao

Provider celery: 3.7.2rc1

• Fixing exception types to include TypeError, which is what is raised in (#40012): @leakec
  Linked issues:
  • Linked Issue #40011: @leakec
• catch sentry flush if exception happens in _execute_in_fork finally block (#40060): @Lee-W
• add PID and return code to _execute_in_fork logging (#40058): @Lee-W

Provider cncf.kubernetes: 8.3.1rc1

• Fixes K8s trigger failing running pods with timeout (#40019): @Usiel
• Refresh properties on KubernetesPodOperator on token expiration also when logging (#39789): @paolo-moriello
• Fix Pod Number Exception in the sync mode if reattach_on_restart parameter is False (#39329): @e-galan
• Avoid resetting adopted task instances when retrying for kubernetes executor (#39406): @tanvn
• Include fatal reason for pod pending events (#39924): @wolfier

All users involved in the PRs:
@potiuk @mateuslatrova @ferruzzi @gopidesupavan @dirrao @leakec @tanvn @paolo-moriello @wolfier @raphaelauv @ellisms @Usiel @yehoshuadimarsky @e-galan @dondaum @Lee-W

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[potiuk]

My changes were just in tests but I tested that indeed the new celery provider fixes the problem with celery executor reported in #40011 (Impacting also previous versions of airflow). WDYT @eladkal, maybe we should yank the 3.7.1 of celery provider already - to avoid people to install it accidentally ? It's pretty bad when you install 3.7.1 for all previous Airlfow version that use celery provider.

[eladkal]

WDYT @eladkal, maybe we should yank the 3.7.1 of celery provider already - to avoid people to install it accidentally ?

Agreed

[potiuk]

Yenked 3.7.1

[dondaum]

Tested #39654, works as expected.

[fdemiane]

No issues from my side

[tanvn]

Tested #39406 by installing cncf.kubernetes: 8.3.1rc1 upon my Airflow 2.8.4, confirmed it is working as expected.

[arollet-decathlon]

Tested #40051 it fixed #39970 ! Huge thanks @gopidesupavan

[yehoshuadimarsky]

Tested Amazon provider, works for me

[leakec]

Tested celery: 3.7.2rc1 and verified it implements #40012 and that it fixes #40011.

[gopidesupavan]

Tested both #40014 and #39923 working fine.

[gopidesupavan]

Thanks @arollet-decathlon for testing #40051

[dirrao]

Tested #39834, #39829 and #40020. They looks good to me.

[Usiel]

Tested #40019 with Airflow 2.9.1, all good 👍

[eladkal]

Thank you everyone. Providers are released.

I invite everyone to help improve providers for the next release, a list of open issues can be found here.