Status of testing Providers that were prepared on June 07, 2022

[potiuk]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider alibaba: 2.0.0rc2

• SSL Bucket, Light Logic Refactor and Docstring Update for Alibaba Provider (#23891): @koconder
• Apply per-run log templates to log handlers (#24153): @uranusjr

Provider amazon: 4.0.0rc2

• Create get_partition and create_partition methods in GlueCatalogHook (#23857): @gmcrocetti
• Add support for associating custom tags to job runs submitted via EmrContainerOperator (#23769): @skadyan
• Fix RedshiftCreateClusterOperator for single-node cluster type (#23839): @pankajastro
• fix: StepFunctionHook ignores explicit set region_name (#23976): @Taragolis
• Fix Amazon EKS example DAG raises warning during Imports (#23849): @Dark-Knight11
• Move string arg evals to execute() in EksCreateClusterOperator (#23877): @josh-fell
• Fix: AwsGlueJobOperator - patches #24215 (#24308): @gmcrocetti
• Light Refactor and Clean-up AWS Provider (#23907): @koconder
• Update sample dag and doc for RDS (#23651): @vincbeck
• Reformat the whole AWS documentation (#23810): @vincbeck
• Replace "absolute()" with "resolve()" in pathlib objects (#23675): @potiuk
• Apply per-run log templates to log handlers (#24153): @uranusjr
• Refactor GlueJobHook get_or_create_glue_job method. (#24215): @gmcrocetti
• Update the DMS Sample DAG and Docs (#23681): @ferruzzi
• Update doc and sample dag for Quicksight (#23653): @vincbeck
• Update doc and sample dag for EMR Containers (#24087): @vincbeck
• Add AWS project structure tests (re: AIP-47) (#23630): @o-nikolas
• Add doc and sample dag for GCSToS3Operator (#23730): @vincbeck
• Remove old Athena Sample DAG (#24170): @ferruzzi
• Clean up f-strings in logging calls (#23597): @josh-fell

Provider apache.beam: 4.0.0rc2

• Added missing project_id to the wait_for_job (#24020): @lwyszomi
• Support impersonation service account parameter for Dataflow runner (#23961): @lwyszomi
• chore: Refactoring and Cleaning Apache Providers (#24219): @koconder

Provider apache.cassandra: 3.0.0rc2

• chore: Refactoring and Cleaning Apache Providers (#24219): @koconder

Provider apache.druid: 3.0.0rc2

• chore: Refactoring and Cleaning Apache Providers (#24219): @koconder

Provider apache.hdfs: 3.0.0rc2

• chore: Refactoring and Cleaning Apache Providers (#24219): @koconder

Provider apache.hive: 3.0.0rc2

• chore: Refactoring and Cleaning Apache Providers (#24219): @koconder
• AIP-47 - Migrate hive DAGs to new design #22439 (#24204): @chethanuk-plutoflume

Provider apache.kylin: 3.0.0rc2

• AIP-47 - Migrate kylin DAGs to new design #22439 (#24205): @chethanuk-plutoflume
• chore: Refactoring and Cleaning Apache Providers (#24219): @koconder

Provider apache.pinot: 3.0.0rc2

• chore: Refactoring and Cleaning Apache Providers (#24219): @koconder

Provider apache.spark: 3.0.0rc2

• Add typing for airflow/configuration.py (#23716): @potiuk
• Fix backwards-compatibility introduced by fixing mypy problems (#24230): @potiuk
• AIP-47 - Migrate spark DAGs to new design #22439 (#24210): @chethanuk-plutoflume
• chore: Refactoring and Cleaning Apache Providers (#24219): @koconder

Provider cncf.kubernetes: 4.1.0rc2

• Use KubernetesHook to create api client in KubernetesPodOperator (#20578): @dstandish
• [FEATURE] KPO add optional K8S hook (#22086): @raphaelauv
• Add param docs to KubernetesHook and KubernetesPodOperator (#23955) (#24054): @mariasumedre-3pg
• Use "remote" pod when patching KPO pod as "checked" (#23676): @dstandish
• Don't use the root logger in KPO _suppress function (#23835): @dstandish
• Fix KubernetesPodOperator await_container_completion condition (#23883): @akakakakakaa
• Migrate Cncf.Kubernetes example DAGs to new design #22441 (#24132): @chethanuk-plutoflume
• Clean up f-strings in logging calls (#23597): @josh-fell

Provider databricks: 3.0.0rc2

• Add Databricks Deferrable Operators (#19736): @eskarimov
• Add git_source to DatabricksSubmitRunOperator (#23620): @akolar-db
• fix: DatabricksSubmitRunOperator and DatabricksRunNowOperator cannot define .json as template_ext (#23622) (#23641): @jonathansp
• Fix UnboundLocalError when sql is empty list in DatabricksSqlHook (#23815): @kazanzhy

Provider dbt.cloud: 2.0.0rc2

• Enable dbt Cloud provider to interact with single tenant instances (#24264): @epapineau
• Fix typo in dbt Cloud provider description (#23179): @josh-fell
• Fix new MyPy errors in main (#22884): @potiuk

Provider docker: 3.0.0rc2

• Remove xcom_push from DockerOperator (#23981): @eladkal
• Migrate Docker example DAGs to new design (#23167): @Bowrna

Provider elasticsearch: 4.0.0rc2

• Apply per-run log templates to log handlers (#24153): @uranusjr

Provider exasol: 3.0.0rc2

• Fix UnboundLocalError when sql is empty list in ExasolHook (#23812): @kazanzhy

Provider github: 2.0.0rc2

• Remove GithubOperator use in GithubSensor.__init__() (#24214): @josh-fell

Provider google: 8.0.0rc2

• Add key_secret_project_id parameter which specifies a project with KeyFile (#23930): @MaksYermak
• Added impersonation_chain for dataflow operators (#24046): @lwyszomi
• Add fields to CLOUD_SQL_EXPORT_VALIDATION. (#23724): @ishiis
• Update credentials when using ADC in Compute Engine (#23773): @MaksYermak
• set color to operators in cloud_sql.py (#24000): @ishiis
• Sql to gcs with exclude columns (#23695): @jaegwonseo
• [Issue#22846] allow option to encode or not encode UUID when uploading from Cassandra to GCS (#23766): @fuxiao224
• Workflows assets & system tests migration (AIP-47) (#24105): @wojsamjan
• Spanner assets & system tests migration (AIP-47) (#23957): @wojsamjan
• Speech To Text assets & system tests migration (AIP-47) (#23643): @wojsamjan
• Cloud SQL assets & system tests migration (AIP-47) (#23583): @wojsamjan
• Cloud Storage assets & StorageLink update (#23865): @wojsamjan
• fix BigQueryInsertJobOperator (#24165): @raphaelauv
• Fix the link to google workplace (#24080): @francois2metz
• Fix DataprocJobBaseOperator not compatible with TaskGroups (#23791): @gmcrocetti
• Remove hack from BigQuery DTS hook (#23887): @henriqueribeiro
• Fix GCSToGCSOperator cannot copy a single file/folder without copying other files/folders with that prefix (#24039): @gitstart-airflow
• Workaround job race bug on biguery to gcs transfer (#24330): @potiuk
• Fix BigQuery system tests (#24013): @bhirsz
• Ensure @contextmanager decorates generator func (#23103): @uranusjr
• Migrate Dataproc to new system tests design (#22777): @bhirsz
• AIP-47 - Migrate google leveldb DAGs to new design ##22447 (#24233): @chethanuk-plutoflume
• Apply per-run log templates to log handlers (#24153): @uranusjr

Provider hashicorp: 3.0.0rc2

• Add explanatory note for contributors about updating Changelog (#24229): @potiuk
• Prepare provider documentation 2022.05.11 (#23631): @potiuk
• pydocstyle D202 added (#24221): @Bowrna
• Prepare docs for May 2022 provider's release (#24231): @potiuk

Provider jdbc: 3.0.0rc2

• Handler parameter from JdbcOperator to JdbcHook.run (#23817): @kazanzhy

Provider jenkins: 3.0.0rc2

• Bug Fix for apache-airflow-providers-jenkins JenkinsJobTriggerOperator (#22802): @SasanAhmadi

Provider microsoft.azure: 4.0.0rc2

• Pass connection extra parameters to wasb BlobServiceClient (#24154): @tanelk
• Apply per-run log templates to log handlers (#24153): @uranusjr
• Migrate Microsoft example DAGs to new design #22452 - azure (#24141): @chethanuk-plutoflume
• Add typing to Azure Cosmos Client Hook (#23941): @potiuk

Provider oracle: 3.0.0rc2

• Add parameters to templated fields in OracleOperator (#22857): @malthe
• Make numpy effectively an optional dependency for Oracle provider (#24272): @potiuk

Provider papermill: 3.0.0rc2

• Add support to specify language name in PapermillOperator (#23916): @nhojjohn
• Fix langauge override in papermill operator (#24301): @nhojjohn

Provider presto: 3.0.0rc2

• Add PrestoToSlackOperator (#23979): @eladkal

Provider qubole: 3.0.0rc2

• Add typing for airflow/configuration.py (#23716): @potiuk
• Fix backwards-compatibility introduced by fixing mypy problems (#24230): @potiuk

Provider sftp: 3.0.0rc2

• Adding fnmatch type regex to SFTPSensor (#24084): @alexkruc

Provider snowflake: 3.0.0rc2

• Fix error when SnowflakeHook take empty list in sql param (#23767): @kazanzhy

Provider sqlite: 3.0.0rc2

• Fix SqliteHook compatibility with SQLAlchemy engine (#23790): @kaxil

Provider ssh: 3.0.0rc2

• Add disabled_algorithms as an extra parameter for SSH connections (#24090): @pauldalewilliams
• fixing SSHHook bug when using allow_host_key_change param (#24116): @alexkruc

The guidelines on how to test providers can be found in

Verify providers by contributors

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[vincbeck]

All my changes are related to doc and/or sample dag. No testing needed

[potiuk]

Cool. Bulk-updated them as "tested" :D

[raphaelauv]

#23981 work

[potiuk]

I also tested and updated all "my" changes

[raphaelauv]

#20578 I have no regression ( not using a new connection or using args of the KPO ) if still using core airlfow kubernetes settings 👍

[potiuk]

One comment. I am releasing rc2 in a moment (@josh-fell found that our RELEASE notes (part of packages) contained Airflow 2.1+ so I am re-relesing all providers with that one change included (but feel free to continue the testing - the only difference will be those descriptions.

[ishiis]

#23724 #24000 is tested.

[Dark-Knight11]

#23849 contains only refactoring of code. No testing needed

[pauldalewilliams]

I've tested #24090 and can confirm it's working as expected.

[pankajastro]

Tested #23839 working as expected.

[dstandish]

@potiuk do you know why k8s is a major?

[jonathansp]

I've tested #23622 and it works as expected.

[gilangardya]

I've tested #24165 (google, BigQueryInsertJobOperator) with composer-2.0.15-airflow-2.2.5, it works as expected.
These are the tests I did:

Job type	Statement	Details	7.0.0	2022.5.18+composer	8.0.0rc1
COPY	N/A	Copy BigQuery Table	❌	❌	✅
EXTRACT	N/A	To CSV	❌	❌	✅
LOAD	N/A	From CSV	❌	❌	✅
QUERY	N/A	SELECT query	✅	✅	✅
QUERY	N/A	WRITE_TRUNCATE with partitioned table	✅	✅	✅
QUERY	DDL	CREATE TABLE AS SELECT	✅	✅	✅
QUERY	DML	INSERT	✅	✅	✅
QUERY	DML	MERGE	✅	✅	✅
QUERY	DCL	GRANT	❌	❌	✅
QUERY	Procedural	DECLARE	❌	❌	✅
QUERY	Debugging	ASSERT	❌	❌	✅
QUERY	Other	EXPORT DATA	❌	❌	✅
QUERY	Multi-statement	DECLARE + INSERT + MERGE	❌	❌	✅

[nhojjohn]

I checked #23916 and noticed an problem with it. Feel free to revert it or consider the following PR which fixes the issue #24301

[SasanAhmadi]

I've tested #22802 and it is working as expected.

[potiuk]

I checked #23916 and noticed an problem with it. Feel free to revert it or consider the following PR which fixes the issue #24301

Cool. I will look to it - I am going release rc2 so I will merge it before.

[potiuk]

@potiuk do you know why k8s is a major?

As discussed on slack - mistake :). Will release minor as rc2.

[gmcrocetti]

1. Create get_partition and create_partition methods in GlueCatalogHook  #23857 works;
2. Refactor GlueJobHook get_or_create_glue_job method. #24215 is broken but there's a patch fixing it: Fix: AwsGlueJobOperator - patches #24215 #24308;
3. Fix DataprocJobBaseOperator not compatible with TaskGroups #23791 I'm currently unable to test it. @eladkal, as the author of the issue would you be willing to test it ?

[alexkruc]

Verified the installation and that everything exists properly for my fixes:

• Adding fnmatch type regex to SFTPSensor #24084
• fixing SSHHook bug when using allow_host_key_change param #24116:

[potiuk]

Thanks @gmcrocetti - as I am releasing rc2 soon, I merged your fix :)

[kazanzhy]

#23767 working as expected

[eskarimov]

Tested #19736 by executing DatabricksRunNowDeferrableOperator and DatabricksSubmitRunDeferrableOperator, both work as expected.

[Taragolis]

Tested #23976 working fine

[o-nikolas]

My changes are testing related and not part of provider code shipped to customers. Though I verified those tests are still passing 👍

[potiuk]

I re-released the RC2 after few fixes (thanks to those who noticed them) and also corrected a number of documentation issues (that turned to be a bit more complex due to AIP-47 induced changes) - thanks @josh-fell and @deeptansh for raising the issues.

Announcement is about to be sent to the devlist. There is no real need to re-test the already tested changes (I left them checked) as the changes were minimal when it comes to code.

But the not-checked issues also need some ❤️ :).

Summary of changes in RC2:

• @dstandish - the "cncf.kubernetes' is now 4.1.0 not 5.0.0 (my mistake)
• @nhojjohn - your fix Fix langauge override in papermill operator #24301 should be included in RC2 - worth double checking
• @deeptansh - the links to example dags in the docs should now be stable and correct
• A number of "system tests" (google and dbt) were moved while automating doc links - they were both missing "cloud" subdir (wich is needed for consistency
• Amazon and Google which are in progress with AIP-47 have two links (examples + legacy)
• I workarounded (proper fix will come) a Google provider BigQuery extract job failure

Have fun with testing (we release on Monday if all is good).

[josh-fell]

All of mine are 👍

[nhojjohn]

tested papermill operator with #24301 in rc2 and works now. Thanks!

[potiuk]

Looking forward for more testing till Monday :).

[koconder]

Tests looking good

[potiuk]

Thanks everyone!. Release is accepted. Proceeding with release.