Status of testing Providers that were prepared on August 30, 2021

[potiuk]

I have a kind request for all the contributors to the latest provider packages release.
Could you help us to test the RC versions of the providers and let us know in the comment,
if the issue is addressed there.

Providers that need testing

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 2.2.0rc1

• Add an Amazon EMR on EKS provider package (#16766): @dacort
• redshift to S3: add optional SQL parameters (#17640): @Xrampino
• Add new LocalFilesystemToS3Operator under Amazon provider (#17168) (#17382): @denimalpaca
• Add Mongo projections to hook and transfer (#17379): @JavierLopezT
• make platform version as independent parameter of ECSOperator (#17281): @codenamestif
• Improve AWS SQS Sensor (#16880) (#16904): @baolsen
• Implemented Basic EKS Integration (#16571): @ferruzzi
• Fixing ParamValidationError when executing load_file in Glue hooks/operators (#16012): @rahulraina7
• Fixes #16972 - Slugify role session name in AWS base hook (#17210): @mcreenan
• Fix broken XCOM in EKSPodOperator (#17918): @mik-laj @ferruzzi
• Fix provider.yaml errors due to exit(0) in test (#17858): @potiuk
• Adds secret backend and logging information to provider yaml (#17625): @potiuk

Provider apache.hdfs: 2.1.0rc1

• hdfs provider: allow SSL webhdfs connections (#17637): @Aakcht

Provider apache.hive: 2.0.2rc1

• HiveHook fix get_pandas_df() failure when it tries to read an empty table (#17777): @iblaine

Provider apache.livy: 2.1.0rc1

• Fetching and logging Livy session logs for LivyOperator (#17393): @sreenath-kamath

Provider asana: 1.1.0rc1

• Use built-in cached_property on Python 3.8 for Asana provider (#17597): @kaxil

Provider docker: 2.1.1rc1

• Add support for configs, secrets, networks and replicas for DockerSwarmOperator (#17474): @enima2684

Provider elasticsearch: 2.0.3rc1

• Fix Invalid log order in ElasticsearchTaskHandler (#17551): @suhanovv
• Adds secret backend and logging information to provider yaml (#17625): @potiuk

Provider exasol: 2.0.1rc1

• ExasolHook get_pandas_df does not return pandas dataframe but None (#17850): @Goodkat

Provider google: 5.1.0rc1

• Add error check for config_file parameter in GKEStartPodOperator (#17700): @leahecole
• Gcp ai hyperparameter tuning (#17790): @subkanthi
• Allow omission of initial_node_count if node_pools is specified (#17820): @leahecole
• [Airflow 13779] use provided parameters in the wait_for_pipeline_state hook (#17137): @lwyszomi
• Enable specifying dictionary paths in template_fields_renderers (#17321): @nathadfield
• Don't cache Google Secret Manager client (#17539): @jedcunningham
• [AIRFLOW-9300] Add DatafusionPipelineStateSensor and aync option to the CloudDataFusionStartPipelineOperator (#17787): @lwyszomi
• GCP Secret Manager Error Handling for Missings Credentials (#17264): @fhoda
• Adds secret backend and logging information to provider yaml (#17625): @potiuk

---

Provider hashicorp: 2.1.0rc1 : DROPPED FROM THE WAVE

• Enable Connection creation from Vault parameters (#15013): @davlum
• Adds secret backend and logging information to provider yaml (#17625): @potiuk
  UPDATE: There is a regression in Hashicorp. The secret backend does not work. Will be released separately.

---

Provider jdbc: 2.0.1rc1

• Fix type annotations in OracleOperator, JdbcOperator, SqliteOperator (#17406): @shadrus

Provider microsoft.azure: 3.1.1rc1

• Adds secret backend and logging information to provider yaml (#17625): @potiuk

Provider mongo: 2.1.0rc1

• Add Mongo projections to hook and transfer (#17379): @JavierLopezT
• Import Hooks lazily individually in providers manager (#17682): @potiuk

Provider neo4j: 2.0.1rc1

• doc: fixed docstring for sql param in Neo4jOperator (#17407): @shadrus

Provider postgres: 2.2.0rc1

• Make schema in DBApiHook private (#17423): @potiuk

Provider salesforce: 3.2.0rc1

• Adding JWT, IP filtering, and direct session login support for SalesforceHook (#17399): @josh-fell

Provider samba: 3.0.0rc1

• Switch to 'smbprotocol' library (#17273): @malthe

Provider slack: 4.0.1rc1

• Example dag slackfile (#17400): @subkanthi
• Fixed SlackAPIFileOperator to upload file and file content (#17247): @subkanthi

Provider sqlite: 2.0.1rc1

• Fix sqlite hook - insert and replace functions (#17695): @subkanthi

Provider yandex: 2.1.0rc1

• Add autoscaling subcluster support and remove defaults (#17033): @peter-volkov

New providers

• Provider alibaba: 1.0.0rc1: @Gabriel39
• Provider microsoft.psrp: 1.0.0rc1: @malthe

Providers that do not need testing

Those are providers that were either doc-only or only got the meta-data updated including preparing for connection import optimisation that will be available in Airflow 2.2.0. Those provider do not contain changes that need additional testing.

• Provider airbyte: 2.1.1rc1
• Provider apache.beam: 3.0.1rc1
• Provider apache.cassandra: 2.0.1rc1
• Provider apache.drill: 1.0.1rc1
• Provider apache.druid: 2.0.2rc1
• Provider apache.kylin: 2.0.1rc1
• Provider apache.pig: 2.0.1rc1
• Provider apache.pinot: 2.0.1rc1
• Provider apache.spark: 2.0.1rc1
• Provider apache.sqoop: 2.0.2rc1
• Provider cloudant: 2.0.1rc1
• Provider databricks: 2.0.1rc1
• Provider datadog: 2.0.1rc1
• Provider dingding: 2.0.1rc1
• Provider discord: 2.0.1rc1
• Provider facebook: 2.0.1rc1
• Provider ftp: 2.0.1rc1
• Provider grpc: 2.0.1rc1
• Provider http: 2.0.1rc1
• Provider imap: 2.0.1rc1
• Provider jenkins: 2.0.2rc1
• Provider jira: 2.0.1rc1
• Provider microsoft.mssql: 2.0.1rc1
• Provider microsoft.winrm: 2.0.1rc1
• Provider mysql: 2.1.1rc1
• Provider odbc: 2.0.1rc1
• Provider opsgenie: 2.0.1rc1
• Provider oracle: 2.0.1rc1
• Provider pagerduty: 2.0.1rc1
• Provider papermill: 2.0.1rc1
• Provider plexus: 2.0.1rc1
• Provider presto: 2.0.1rc1
• Provider qubole: 2.0.1rc1
• Provider redis: 2.0.1rc1
• Provider segment: 2.0.1rc1
• Provider sendgrid: 2.0.1rc1
• Provider sftp: 2.1.1rc1
• Provider singularity: 2.0.1rc1
• Provider snowflake: 2.1.1rc1
• Provider ssh: 2.1.1rc1
• Provider tableau: 2.1.1rc1
• Provider telegram: 2.0.1rc1
• Provider trino: 2.0.1rc1
• Provider vertica: 2.0.1rc1
• Provider zendesk: 2.0.1rc1

[nathadfield]

#17321 tested and looks good to me!

[Aakcht]

Tested #17637 - looks good

[iblaine]

#17777 tested and looks good

[nathadfield]

I've just tried to start a version of our Astronomer based Airflow (v2.1.3) using the v2.1.0rc1 Hashicorp provider but I'm seeing the following on startup.

File "/usr/local/lib/python3.7/site-packages/airflow/configuration.py", line 470, in getimport
2021-08-31T07:58:31.887552500Z     f'The object could not be loaded. Please check "{key}" key in "{section}" section. '
2021-08-31T07:58:31.887570500Z airflow.exceptions.AirflowConfigException: The object could not be loaded. Please check "backend" key in "secrets" section. Current value: "airflow.providers.hashicorp.secrets.vault.VaultBackend".

[pavelhlushchanka]

#17281 looks good

[enima2684]

#17474 <#17474> tested and looks good

…

On Tue, Aug 31, 2021 at 10:22 AM Pavel Hlushchanka ***@***.***> wrote: #17281 <#17281> looks good — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17922 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF6ZXBPXLR4W23HCIICVIMLT7SGNTANCNFSM5DCZC7YQ> .

[potiuk]

I've just tried to start a version of our Astronomer based Airflow (v2.1.3) using the v2.1.0rc1 Hashicorp provider but I'm seeing the following on startup.

Did it work with previous version of the package (2.0.0) ?

[nathadfield]

I've just tried to start a version of our Astronomer based Airflow (v2.1.3) using the v2.1.0rc1 Hashicorp provider but I'm seeing the following on startup.

Did it work with previous version of the package (2.0.0) ?

Yep!

[potiuk]

Yep!

All right. I think I know. We'll drop Hashicorp from this wave and release it separately. Good spot.

[Gabriel39]

alibaba 1.0.0rc1: tested and working :)

[suhanovv]

#17551 Looks good

[peter-volkov]

checked Provider yandex: 2.1.0rc1
with apache-airflow==2.1.3
works well.

Is it expected that apache-airflow-backport-providers-yandex for Airflow 1.* does not have new rc version?
Will it be released?

[potiuk]

Is it excepted that apache-airflow-backport-providers-yandex for Airflow 1.* does not have new rc version?
Will it be released?

According to the earlier announcements, we stopped releasing backports in March. Airflow 1.10 reached end of life in June and we encourage everyone to migrate to Airflow 2. There will be no more releases (not even critical security fixes) for Airflow 1.10 nor backports.

[josh-fell]

#17399 Tested and verified. Thanks for putting this together @potiuk!

[mcreenan]

Tested amazon: 2.2.0rc1 for #17210 and it looks good.

I was able to run my test dag and validate it fails with amazon provider 1.2.0 and succeeded with 2.2.0rc1.

I love that the providers are separate and we can simply update the version in requirements.txt instead of having to update to a newer version of Airflow. Great work on that!

[potiuk]

I love that the providers are separate and we can simply update the version in requirements.txt instead of having to update to a newer version of Airflow. Great work on that!

Glad you like it! That was quite an effort of quite many people but I do agree it's been worth!

[Goodkat]

#17850 tested and verified. Now it returns the pandas dataframe as expected.

[potiuk]

Cool! . We are at 1/3 of the fixes confirmed after 1/3 of the time. Great progress :) . Thanks everyone who are helping!

[rahulraina7]

Tested #16012 . Loading the file in s3 as expected

[dacort]

Tested #16766 with apache-airflow==2.1.3 - used the example DAG to trigger an EMR on EKS job to completion! 🎉

[potiuk]

FYI. I also marked some of the providers (jdbc, neo4j) that had only "type annotation" and doc changes on top of the "2.2.0" optimisation. There are no substantial changes and they are installing, importing and they are discovered properly, So no need to test them separately either.

BTW. For everyone watching - all the providers are now "Warning free" as well. There were a few providers in which just importing operators generated unnecessary warnings and some examples used deprecated operators. After merging #17900 w have no warnings (except those generated in 3rd-party libraries) and all examples are also using non-deprecated operators.

[potiuk]

Tested #17423 - works!

[potiuk]

We have majority of providers tested and working now- I would approciate some test on the ones that have not been touched yet,

[Xrampino]

Tested #17640. Works as expected!

[subkanthi]

Tested #17695 . Thanks

[JavierLopezT]

Tested #17379. It's working fine

[potiuk]

Last chance to test remaining providers :D

[subkanthi]

@potiuk tested both
Example dag slackfile (#17400):
Fixed SlackAPIFileOperator to upload file and file content (#17247):

it works, thanks

[potiuk]

Thanks everyone for testing! You rock!