Status of testing Providers that were prepared on April 10, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider airbyte: 3.7.0rc1

• Ensure Airbyte Provider is Compatible with Cloud and Config APIs (#37943): @chrishronek
  Linked issues:
  • Linked Issue #35011: @vytautasnord
• fix: try002 for provider airbyte (#38786): @dondaum
  Linked issues:
  • Linked Issue #38761: @Taragolis
• add "return" statement to "yield" within a while loop in airbyte triggers (#38390): @Lee-W

Provider alibaba: 2.7.3rc1

• Avoid use of assert outside of the tests (#37718): @Taragolis

Provider amazon: 8.20.0rc1

• AWS Batch Executor (#37618): @syedahsn
• Add fallback region_name value to AWS Executors (#38704): @Taragolis
• Introduce Amazon Bedrock service (#38602): @ferruzzi
• Amazon Bedrock - Model Customization Jobs (#38693): @ferruzzi
• ECS Executor - add support to adopt orphaned tasks. (#37786): @ferruzzi
  Linked issues:
  • Linked Issue #35491: @o-nikolas
• Update AWS auth manager CLI command to not disable AVP schema validation (#38301): @vincbeck
• Reduce s3hook memory usage (#37886): @ellisms
  Linked issues:
  • Linked Issue #35449: @Taragolis
• Add check in AWS auth manager to check if the Amazon Verified Permissions schema is up to date (#38333): @vincbeck
• fix: EmrServerlessStartJobOperator not serializing DAGs correctly whe… (#38022): @jliu0812
  Linked issues:
  • Linked Issue #38005: @jliu0812
• add "return" statement to "yield" within a while loop in amazon triggers (#38396): @Lee-W
• Fix set deprecated amazon operators arguments in MappedOperator (#38346): @Taragolis
• ECSExecutor API Retry bug fix (#38118): @syedahsn
• Fix region argument in MappedOperator based on AwsBaseOperator / AwsBaseSensor (#38178): @Taragolis
• ECS Executor retry task bug fix (#37979): @syedahsn
• Fix init checks for aws redshift to s3 operator (#37861): @okirialbert
  Linked issues:
  • Linked Issue #36484: @shahar1
• Make the method BaseAuthManager.is_authorized_custom_view abstract (#37915): @vincbeck
• Replace "Role" by "Group" in AWS auth manager (#38078): @vincbeck
• Avoid use of assert outside of the tests (#37718): @Taragolis
• Use AwsLogsHook when fetching Glue job logs (#38010): @VincentChantreau
• Implement filter_permitted_dag_ids in AWS auth manager (#37666): @vincbeck
• AWS auth manager CLI: persist the policy store description when doing updates (#37946): @vincbeck
• Change f-string to formatting into the logging messages for Batch Executor (#37929): @Taragolis

Provider apache.beam: 5.6.3rc1

• Fix skip Apache Beam pipeline options if value is set to false (#38496): @dondaum
  Linked issues:
  • Linked Issue #38457: @gblazq
• Fix side-effect of default options in Beam Operators (#37916): @potiuk
• Avoid to use subprocess in asyncio loop (#38292): @Taragolis
• Avoid change attributes into the constructor in Apache Beam operators (#37934): @Taragolis

Provider apache.cassandra: 3.4.2rc1

• Re-enable cassandra for Python 3.12 after the driver got libev support (#38314): @potiuk

Provider apache.hive: 8.0.0rc1

• Setting use_beeline by default for hive cli connection (#38763): @amoghrajesh
• Removing deprecated code in hive provider (#38859): @amoghrajesh
• Adding support to hive hook for high availability Hive installations (#38651): @amoghrajesh
• Fix TRY002 for apache hive provider (#38781): @amoghrajesh
• Fix deprecated apache.hive operators arguments in MappedOperator (#38351): @Taragolis

Provider apache.spark: 4.7.2rc1

• Rename SparkSubmitOperator/SparkJDBCOperator fields' names to comply with templated fields validation (#38051): @shahar1
  Linked issues:
  • Linked Issue #36484: @shahar1
• Rename SparkSqlOperator's field to comply with templated fields' validation (#38045): @shahar1
  Linked issues:
  • Linked Issue #36484: @shahar1

Provider celery: 3.6.2rc1

• Ensure exit is called in decorator context managers (#38383): @dstandish
  Linked issues:
  • Linked Issue #36800: @joaopamaral
• Don't dispose sqlalchemy engine when using internal api (std task runner) (#38562): @dstandish
• Use celery worker CLI from Airflow package for Airflow < 2.8.0 (#38879): @potiuk
  Linked issues:
  • Linked Issue #34945: @Bisk1
  • Linked Issue #36794: @pavansharma36
• Allow to use redis>=5 (#38385): @Taragolis
• Re-raise of AirflowOptionalProviderFeatureException should be direct (#38555): @dstandish

Provider cncf.kubernetes: 8.1.0rc1

• KPO Add follow log in termination step (#38081): @pankajastro
• Add GKECreateCustomResourceOperator and GKEDeleteCustomResourceOperator operators (#37616): @MaksYermak
• Implement deferrable mode for KubernetesJobOperator (#38251): @moiseenkov
• Create KubernetesPatchJobOperator operator (#38146): @MaksYermak
• Implement delete_on_status parameter for KubernetesDeleteJobOperator (#38458): @moiseenkov
• Implement deferrable mode for GKEStartJobOperator (#38454): @moiseenkov
• Use startup_check_interval_seconds instead of poll_interval to check pod while startup (#38075): @pankajastro
• Implement wait_until_job_complete parameter for KubernetesJobOperator (#37998): @moiseenkov
• Use SIGINT signal number instead of signal name (#37905): @pdebelak
  Linked issues:
  • Linked Issue #37903: @pdebelak
• Fix spark operator log retrieval from driver (#38106): @sudohainguyen
  Linked issues:
  • Linked Issue #37681: @thispejo
• Fix dynamic allocation specs handling for custom launcher (#38223): @sudohainguyen
• Fix case if SparkKubernetesOperator.application_file is templated file (#38035): @Taragolis
  Linked issues:
  • Linked Issue #38017: @gbloisi-openaire
• fix: reduce irrelevant error logs for pod events. (#37944): @sudiptob2
  Linked issues:
  • Linked Issue #36077: @ketozhang
• Add GKEListJobsOperator and GKEDescribeJobOperator (#37598): @VladaZakharova
• removed usage of deprecated function for naming the pod in provider k8s pod.py (#38638): @idantepper
• Create DeleteKubernetesJobOperator and GKEDeleteJobOperator operators (#37793): @MaksYermak
• Refactor GKE hooks (#38404): @moiseenkov

Provider cohere: 1.1.3rc1

• Limit cohere to <5 (#38348): @Taragolis

Provider common.io: 1.3.1rc1

• Improve XComObjectStoreBackend implementation (#38608): @uranusjr
• Rename to XComObjectStorageBackend (#38607): @uranusjr
  Linked issues:
  • Linked Issue #38142: @pankajastro
• Turn common.io xcom exception into OptionalProviderFeatureException (#38543): @potiuk
• Update ObjectStoragePath for universal_pathlib>=v0.2.2 (#37930): @bolkedebruin

Provider common.sql: 1.12.0rc1

• Add hook_params to template_fields for BaseSQLOperator-related Operators (#38724): @odaneau-astro
• Make placeholder of DbApiHook configurable in UI (#38528): @dabla
  Linked issues:
  • Linked Issue #36000: @jarnedemunter1
• Undeprecating DBApiHookForTests._make_common_data_structure (#38573): @Taragolis

Provider databricks: 6.3.0rc1

• Add cancel_previous_run to DatabricksRunNowOperator (#38702): @SubhamSinghal
• add repair_run support to DatabricksRunNowOperator in deferrable mode (#38619): @Lee-W
  Linked issues:
  • Linked Issue #38619: @Lee-W
• Update ACL during job reset (#38741): @SubhamSinghal
• remove redundant else block in DatabricksExecutionTrigger (#38397): @Lee-W
• Rename DatabricksSqlOperator's fields' names to comply with templated fields validation (#38052): @shahar1
  Linked issues:
  • Linked Issue #36484: @shahar1

Provider dbt.cloud: 3.7.1rc1

• [OpenLineage] fix: disabled_for_operators now stops whole event emission (#38033): @kacpermuda
• add "return" statement to "yield" within a while loop in dbt triggers (#38395): @Lee-W

Provider docker: 3.10.0rc2

• Support multiple Docker hosts in docker_url attribute. (#38466): @oboki
• Fix deprecated DockerOperator operator arguments in MappedOperator (#38379): @Taragolis
• Remove redundant usage of importlib_metadata for distribution versions (#38368): @Taragolis
• DockerOperator: use DOCKER_HOST as default for docker_url (#38387): @maresb

Provider elasticsearch: 5.3.4rc1

• Add missing attrs / correct bad attrs on TaskInstancePydantic (#37854): @dstandish

Provider fab: 1.0.3rc1

• Rename allowed_filter_attrs to allowed_sort_attrs (#38626): @jedcunningham
• Fix azure authentication when no email is set (#38872): @gschuurman

Provider ftp: 3.8.0rc2

• Add ssl context for verification of certs in FTPS hook (#38266): @amoghrajesh

Provider google: 10.17.0rc1

• Add impersonation_scopes to BigQuery (#38169): @ying-w
• Deferrable mode for RunPipelineJobOperator (#37969): @e-galan
• Add GKECreateCustomResourceOperator and GKEDeleteCustomResourceOperator operators (#37616): @MaksYermak
• Add VertexAI Language Model and Multimodal Model Operators for Google Cloud Generative AI use (#37721): @CYarros10
• Add GKEListJobsOperator and GKEDescribeJobOperator (#37598): @VladaZakharova
• Create GKEStartKueueJobOperator operator (#37477): @MaksYermak
• Create DeleteKubernetesJobOperator and GKEDeleteJobOperator operators (#37793): @MaksYermak
• Update GCS hook to get crc32c hash for CMEK-protected objects (#38191): @shahar1
  Linked issues:
  • Linked Issue #34980: @dmedora
  • Linked Issue #34982: @dmedora
• Set job labels for traceability in BigQuery jobs (#37736): @stiak
• Deferrable mode for CreateBatchPredictionJobOperator (#37818): @moiseenkov
• Fix BigQuery connection and add docs (#38430): @shahar1
  Linked issues:
  • Linked Issue #28790: @Taragolis
• fix(google,log): Avoid log name overriding (#38071): @AlexisBRENON
• Fix credentials error for S3ToGCSOperator trigger (#37518): @korolkevich
  Linked issues:
  • Linked Issue #37331: @aviramst
• Fix parent_model parameter in GCP Vertex AI AutoML and Custom Job operators (#38417): @shahar1
  Linked issues:
  • Linked Issue #37400: @devinmnorris
• add "return" statement to "yield" within a while loop in google triggers (#38394): @Lee-W
• Fix cursor unique name surpasses Postgres identifier limit in PostgresToGCSOperator (#38040): @dybolo
  Linked issues:
  • Linked Issue #37654: @obarisk
• Fix GCS Anonymous user issue because none token (#38102): @pankajastro
• Fix BigQueryTablePartitionExistenceTrigger partition query (#37655): @Silviu-Surcica
• Add google-cloud-bigquery as explicit google-provider dependency (#38753): @potiuk
• Avoid to use functools.lru_cache in class methods in google provider (#38652): @Taragolis
• Refactor GKE hooks (#38404): @moiseenkov
• Remove unused loop variable from airflow package (#38308): @hussein-awala
• templated fields logic checks for cloud_storage_transfer_service (#37519): @okirialbert
• Rename MLEngine operators' fields' names to comply with templated fields validation (#38053): @shahar1
  Linked issues:
  • Linked Issue #36484: @shahar1
• Rename Vertex AI AutoML operators fields' names to comply with templated fields validation (#38049): @shahar1
  Linked issues:
  • Linked Issue #36484: @shahar1
• Rename DeleteCustomTrainingJobOperator's fields' names to comply with templated fields validation (#38048): @shahar1
  Linked issues:
  • Linked Issue #36484: @shahar1
• Restore delegate_to for Google Transfer Operators retrieving from Google Cloud. (#37925): @emery-finkelstein
  Linked issues:
  • Linked Issue #29088: @shahar1
  • Linked Issue #30748: @shahar1
• Deprecate sync parameter for CreateHyperparameterTuningJobOperator (#37938): @moiseenkov
• Upgrade google-ads version (#37787): @molcay

Provider http: 4.10.1rc1

• add "return" statement to "yield" within a while loop in http triggers (#38392): @Lee-W
• Consolidate HttpOperator http request between sync and async mode (#37293): @hussein-awala
  Linked issues:
  • Linked Issue #37291: @dlesco

Provider microsoft.azure: 10.0.0rc1

• Feature/refactor azure synapse pipeline class (#38723): @romsharon98
• Add ADLSCreateObjectOperator (#37821): @dabla
• add "return" statement to "yield" within a while loop in azure triggers (#38393): @Lee-W
• fix cosmos hook static checks by making providing partition_key mandatory (#38199): @hussein-awala
  Linked issues:
  • Linked Issue #38196: @Taragolis
• refactor: Refactored new magic method of BaseOperatorMeta to avoid bad mixing classic and decorated operators (#37937): @dabla
• update to latest service bus library (#38384): @swathipil
  Linked issues:
  • Linked Issue #31168: @kashifkhan
• Limit azure-cosmos<4.6.0 (#38175): @eladkal

Provider microsoft.psrp: 2.6.1rc1

• Avoid use of assert outside of the tests (#37718): @Taragolis

Provider odbc: 4.5.0rc1

• Make placeholder of DbApiHook configurable in UI (#38528): @dabla
  Linked issues:
  • Linked Issue #36000: @jarnedemunter1

Provider openlineage: 1.7.0rc1

• Add lineage_job_namespace and lineage_job_name OpenLineage macros (#38829): @dolfinus
• openlineage: add opt-in option (#37725): @JDarDagran
• [OpenLineage] Remove redundant operator information from facets (#38264): @kacpermuda
• [OpenLineage] fix: disabled_for_operators now stops whole event emission (#38033): @kacpermuda
• [OpenLineage] fix: Add fallbacks when retrieving Airflow configuration to avoid errors being raised (#37994): @kacpermuda
• [OpenLineage] fix: Fix parent id macro and remove unused utils (#37877): @kacpermuda
• Avoid use of assert outside of the tests (#37718): @Taragolis
• Add missing attrs / correct bad attrs on TaskInstancePydantic (#37854): @dstandish

Provider papermill: 3.6.2rc1

• Turn Pydantic into an optional dependency (#37320): @potiuk

Provider redis: 3.6.1rc1

• Allow to use redis>=5 (#38385): @Taragolis

Provider samba: 4.6.0rc2

• Allow specification of buffer length for GCS to Samba (#38373): @collinmcnulty

Provider sftp: 4.9.1rc1

• add "return" statement to "yield" within a while loop in sftp triggers (#38391): @Lee-W
• Close open connections for deferrable SFTPSensor (#38881): @pankajkoti

Provider slack: 8.6.2rc1

• Fix set deprecated slack operators arguments in MappedOperator (#38345): @Taragolis
• Update SqlToSlackApiFileOperator with new param to check empty output (#38079): @andyguwc

Provider snowflake: 5.4.0rc1

• feat: update SnowflakeSqlApiHook to support OAuth (#37922): @andyguwc
• Remove reference to execution_info in snowflake hook docstring (#37804): @remisalmon

Provider weaviate: 1.3.4rc1

• Fix WeaviateIngestOperator/WeaviateDocumentIngestOperator arguments in MappedOperator (#38402): @Taragolis

Provider yandex: 3.9.1rc1

• [yandexcloud/yq] support iam token from metadata, simplify code (#38411): @uzhastik
• Avoid use of assert outside of the tests (#37718): @Taragolis

All users involved in the PRs:
@ferruzzi @dondaum @potiuk @sudiptob2 @eladkal @korolkevich @bolkedebruin @dolfinus @stiak @Lee-W @AlexisBRENON @molcay @dabla @dybolo @shahar1 @e-galan @CYarros10 @chrishronek @amoghrajesh @romsharon98

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[pdebelak]

I can confirm that my change to cncf.kubernetes is working as expected.

[Taragolis]

Just figure out that we accidentally exclude #37458 from the changelog. We could add it after the release, because changes itself presented into the released package.

[gschuurman]

#38872 login is working as expected again with both email set and unset

[ellisms]

#37886 is good.

[eladkal]

Just figure out that we accidentally exclude #37458 from the changelog. We could add it after the release, because changes itself presented into the released package.

Its a mistake in the release process.
But its more than just change log. This PR is a feature while the rc1 is a bug fix release thus I will exclude yandex from this wave and cut rc2 afterwards

[jliu0812]

#38022 is good. ✅

[stiak]

#37736 is good.

[sudohainguyen]

#38223 and #38106 are good ✅

[SubhamSinghal]

Added a fix for #38741 in #38918. There was an extra slash in api.

[dondaum]

#38786 and #38496 are good ✅

[potiuk]

Checcked all my changes : all good !

[moiseenkov]

My changes work as expected:
#38251, #38458, #38454, #37998, #38404, #37818, #38404, #37938

[pankajkoti]

Checked SFTP provider for #38881 and is good!

[eladkal]

Added a fix for #38741 in #38918. There was an extra slash in api.

Databricks provider will be excluded from RC1

[kacpermuda]

I checked the whole OpenLineage provider + my PR for dbt.cloud #38033 - all is good.

[e-galan]

#37969 is good.

[dabla]

Just tested #38528 and I confirm it is working for ODBC and JDBC, we had in install issue of the rc provider hence why I dint't work at first but now I can confirm it works

[amoghrajesh]

Installed and RC with breeze and tested my changes:

• Setting use_beeline by default for hive cli connection (Setting use_beeline by default for hive cli connection #38763)
• Removing deprecated code in hive provider (Removing deprecated code in hive provider #38859)
• Adding support to hive hook for high availability Hive installations (Adding support to hive hook for high availability Hive installations #38651)
• Fix TRY002 for apache hive provider (Fix TRY002 for apache hive provider #38781)
• Add ssl context for verification of certs in FTPS hook (Add ssl context for verification of certs in FTPS hook #38266)

All these changes are working as expected.

[Lee-W]

Verified #38390, #38396, #38397, #38394, #38392, #38393, #38391

[eladkal]

Thank you everyone. Providers are released.
yandex and databricks providers are excluded and will followup with rc2

I invite everyone to help improve providers for the next release, a list of open issues can be found here.