Status of testing Providers that were prepared on March 15, 2022

[potiuk]

Body

Provider alibaba: 1.1.0rc1

• Add oss_task_handler into alibaba-provider and enable remote logging to OSS (#21785): @EricGao888

Provider amazon: 3.1.1rc1

• Added AWS RDS sensors (#21231): @kazanzhy
• Added AWS RDS operators (#20907): @kazanzhy
• Add RedshiftDataHook (#19137): @john-jac
• Feature: Add invoke lambda function operator (#21686): @schirag1993
• Add JSON output on SqlToS3Operator (#21779): @mariotaddeucci
• Implement a Sagemaker DeleteModelOperator and Delete model hook. (#21673): @hsrocks
• Added Hook for Amazon RDS (#20642): @kazanzhy
• Added SNS example DAG and rst (#21475): @ferruzzi
• retry on very specific eni provision failures (#22002): @zachliu
• Configurable AWS Session Factory (#21778): @ac1997
• S3KeySensor to use S3Hook url parser (#21500): @dstandish
• ECSOperator: Get log events after sleep to get all logs (#21574): @kanga333
• Use temporary file in GCSToS3Operator (#21295): @rafalh
• AWS RDS integration fixes (#22125): @kazanzhy
• Fix the Type Hints in RedshiftSQLOperator (#21885): @kaxil
• Bug Fix - S3DeleteObjectsOperator will try and delete all keys (#21458): @njrs92
• Fix Amazon SES emailer signature (#21681): @potiuk
• Fix EcsOperatorError, so it can be loaded from a picklefile (#21441): @ngwallace
• Fix RedshiftDataOperator and update doc (#22157): @vincbeck
• bugfix (#22137): @zachliu
• If uploading task logs to S3 fails, retry once (#21981): @steveyz-astro
• Bug-fix GCSToS3Operator (#22071): @rsg17
• Refactor query status polling logic in EMRContainerHook (#21423): @victorphoenix3
• use different logger to avoid duplicate log entry (#22256): @zachliu
• [doc] Improve s3 operator example by adding task upload_keys (#21422): @zhongjiajie
• Rename 'S3' hook name to 'Amazon S3' (#21988): @treyyi
• Add template fields to DynamoDBToS3Operator (#22080): @eladkal

Provider databricks: 2.4.0rc1

• Add new options to DatabricksCopyIntoOperator (#22076): @alexott
• Databricks hook - retry on HTTP Status 429 as well (#21852): @alexott
• Skip some tests for Databricks from running on Python 3.10 (#22221): @potiuk

Provider docker: 2.5.1rc1

• Avoid trying to kill container when it did not succeed for Docker (#22145): @potiuk

Provider google: 6.6.0rc1

• Support Uploading Bigger Files to Google Drive (#22179): @ulsc
• Make chunk_size in GoogleDriveHook standard (#22222): @ulsc
• Add guide for DataprocInstantiateInlineWorkflowTemplateOperator (#22062): @NiloFreitas
• Allow GCS Metadata to be included in GCS Upload (#22058): @patricker
• Dataplex operators (#20377): @wojsamjan
• Add support for ARM platform (#22127): @potiuk
• Use yaml safe load (#22091): @lwyszomi

Provider snowflake: 2.5.2rc1

• Remove Snowflake limits (#22181): @potiuk

The guidelines on how to test providers can be found in

Verify providers by contributors

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[potiuk]

Hello everyone. this is the out-of-bands release of all providers - with Trove Classifiers from PyPI added (Once we release them - they will all appear in https://pypi.org/search/?q=&o=&c=Framework+%3A%3A+Apache+Airflow&c=Framework+%3A%3A+Apache+Airflow+%3A%3A+Provider)

In Amazon, I marked all the changes that were tested in the previous candidate).

[kazanzhy]

#21231, #20907, #20642 - tested in previous RC

#22125 - OK

[potiuk]

#22181 -> OK. limits removed.
#22127 -> OK. (running it on M1 and it's blazing fast)
#22145 -> OK: [2022-03-15, 01:06:54 UTC] {docker.py:411} INFO - Not attempting to kill container as it was not created
#22221 -> OK: Tests only (main is green)
#21681 -> OK: Signature is changed.

[zhongjiajie]

#21422 -> OK: Improve docs with more example tasks

[zachliu]

#22002
#22137
#22256
tested in my production environment
EcsOperator now has auto-retries 💪

[alexott]

Tested #22076, #21852 - looks good

[potiuk]

We are getting fast a serious coverage this time :) .

[ulsc]

#22179 -> OK: Supports uploading bigger files
#22222 -> OK: Chunk size tells more about the default value

[lwyszomi]

#20377 👍
#22091 👍

[vincbeck]

#22157 OK

[vincbeck]

#19137 OK

[EricGao888]

#21785 -> OK: Enable remote logging into Alibaba Cloud OSS

[repl-mike-roest]

#21681: OK
#21475: OK
#21779: OK

Edit: Typo 21861 -> 21681

[potiuk]

Thanks @repl-mike-roest for out-of-list tests! Great open-source-spirit!

[josh-fell]

@potiuk FWIW I know they aren't in this list but since they made it in the RC, I tested the fixes for dbt Cloud in v1.0.1rc1 (#22136, #22126) and they look great!

[potiuk]

Yeah. New providers aren't here :)

[potiuk]

Thanks for all the testing. We will have to release a new wave, as there was a general issue found in #22380 (but there will be just a few new changes to vote).

[zachliu]

dumb question: will these new providers show up in the constraints-2.2.4/constraints-3.8.txt? or be pushed to Airflow 2.2.5?

[potiuk]

dumb question: will these new providers show up in the constraints-2.2.4/constraints-3.8.txt? or be pushed to Airflow 2.2.5?

The latter.

The constraints for each version contain the "snapshot" of providers that are available during release.

Constraints for released version (so constraints-2.2.4 for example) are very rarely updated unless we want to bugfix some dependency issue.

You are free to upgrade the new providers individually as described in https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#installation-and-upgrading-of-airflow-providers-separately (just mind that this wave will be re-run as they have mistakenly two extra packages as dependencies so I think about yanking all of them after we release a new wave). But for the "base" installation of airflow using constraints, stability of the installation is more important than using newer providers.

[zachliu]

gotcha! thanks! i must have missed my orientation or suffered from foggy brain on Mondays. what does "yanking them" mean? 😅

[potiuk]

https://blog.piwheels.org/new-features-deletion-yanking-and-more/

[zachliu]

aha! to save face, i'll say "no wonder it sounds so familiar"