Status of testing Providers that were prepared on August 19, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.28.0rc1

• Add incremental export and cross account export functionality in DynamoDBToS3Operator (#41304): @Ghoul-SSZ
  Linked issues:
  • Linked Issue #40737: @Kuhlmann-Itagyba-bah
• EKS Overrides for AWS Batch submit_job (#40718): @ssilb4
• Fix AwsTaskLogFetcher missing logs (#41515): @vincbeck
  Linked issues:
  • Linked Issue #40875: @yaningz
• Issue-41243 Unpin the moto dependency (#41256): @vikramaditya91
  Linked issues:
  • Linked Issue #41243: @potiuk
• Fix RedshiftDataOperator not running in deferred mode when it should (#41206): @borismo
• Partial fix for example_dynamodb_to_s3.py (#41517): @ferruzzi
• Remove deprecated code is AWS provider (#41407): @vincbeck
  Linked issues:
  • Linked Issue #41396: @eladkal
• Limit moto temporarily - 5.0.12 is breaking our tests (#41244): @potiuk

Provider apache.spark: 4.10.0rc1

• fix: resolve AirflowProviderDeprecationWarning in spark provider (#41358): @phi-friday
  Linked issues:

Provider celery: 3.8.0rc1

• Remove deprecated SubDags (#41390): @kaxil

Provider cncf.kubernetes: 8.4.0rc1

• Remove deprecated SubDags (#41390): @kaxil
• spark kubernetes operator arguments description reordering (#41372): @dirrao

Provider common.sql: 1.16.0rc1

• fix: rm deprecated import in common.sql (#41461): @phi-friday
  Linked issues:
  • Linked Issue #41460: @phi-friday

Provider docker: 3.13.0rc1

• feat(docker): Replace use_dill with serializer (#41356): @phi-friday

Provider elasticsearch: 5.5.0rc1

• add execute_sql API to fix ElasticSearchSQLHook (#41537): @Owen-CH-Leung
  Linked issues:
  • Linked Issue #41486: @Pooort

Provider fab: 1.3.0rc1

• Feature: Allow set Dag Run resource into Dag Level permission (#40703): @joaopamaral
• Remove deprecated SubDags (#41390): @kaxil

Provider google: 10.22.0rc1

• Add CloudRunServiceHook and CloudRunCreateServiceOperator (#40008): @jx2lee
  Linked issues:
  • Linked Issue #38760: @ophie200
• add missing sync_hook_class to CloudDataTransferServiceAsyncHook (#41417): @Lee-W
• Refactor DataprocCreateBatchOperator and Dataproc system tests (#41527): @moiseenkov
• Upgrade package gcloud-aio-auth>=5.2.0 (#41262): @moiseenkov

Provider microsoft.azure: 10.4.0rc1

• Microsoft Power BI operator to refresh the dataset (#40356): @ambika-garg
• Export Azure Container Instance log messages to XCOM (#41142): @perry2of5

Provider microsoft.mssql: 3.9.0rc1

• Add methodtools as dependency to mssql provider (#41392): @potiuk
  Linked issues:
  • Linked Issue #41330: @rawwar

Provider openlineage: 1.11.0rc1

• feat: [openlineage] add debug facet to all events (#41217): @kacpermuda
• feat: add fileloc to DAG info in AirflowRunFacet (#41311): @kacpermuda
• chore: remove openlineage client deprecated from_environment() method (#41310): @kacpermuda
• fix: get task dependencies without serializing task tree to string (#41494): @mobuchowski
• fix: return empty data instead of None when OpenLineage on_start method is missing (#41268): @kacpermuda

Provider papermill: 3.8.0rc1

• restore python 3.12 support for papermill (#41548): @morokosi

Provider snowflake: 5.7.0rc1

• Fix: Pass hook parameters to SnowflakeSqlApiHook and prep them for AP… (#41150): @BTeclaw
  Linked issues:
  • Linked Issue #39622: @alexcsolomon1

Provider yandex: 3.12.0rc1

• providers/yandex: fix typing (#40997): @got686-yandex

Provider ydb: 1.3.0rc1

• ydb provider: add database to table name in bulk upsert, use bulk upsert in system test (#41303): @uzhastik

All users involved in the PRs:
@morokosi @mobuchowski @borismo @dirrao @phi-friday @Ghoul-SSZ @joaopamaral @BTeclaw @potiuk @vikramaditya91 @uzhastik @ambika-garg @Owen-CH-Leung @moiseenkov @ssilb4 @got686-yandex @kaxil @vincbeck @le

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[moiseenkov]

Hi,
#41527, #41262 work as expected

[sc250072]

Hi, since there haven’t been any recent updates for the Teradata Provider beyond version 2.5.0, could you explain why a new release version is needed?

https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpypi.org%2Fproject%2Fapache-airflow-providers-teradata%2F2.6.0rc1%2F&data=05%7C02%7CSATISH.CHINTHANIPPU%40teradata.com%7Cc798de64676b411f9b7a08dcc0279df7%7C9151cbaafc6b4f4889bb8c4a71982138%7C0%7C0%7C638596523614516170%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=uL1wmpzD4EDpAJZPPvDPVbqC%2BbuDSIhwniZPqKyMQ3M%3D&reserved=0

[phi-friday]

1. feat(docker): Replace use_dill with serializer #41356 works fine.
2. fix: resolve AirflowProviderDeprecationWarning in spark provider #41358 I didn't use spark so I can't check, but given that _sql is simply an alias for sql, it should be fine.
3. fix: rm deprecated import in common.sql #41461 I couldn't find a good way to verify common.sql, but since we've only removed the part that generates a more detailed error message, it should be fine.

[potiuk]

Hi, since there haven’t been any recent updates for the Teradata Provider beyond version 2.5.0, could you explain why a new release version is needed?

According to our rules - periodically we bump all providers min-airflow version. https://github.com/apache/airflow/blob/main/PROVIDERS.rst#upgrading-minimum-supported-version-of-airflow -> then we relaease all providers with min-airflow version bumped - we also remove all pre-min-airflow backports, this allows to keep airflow providers free from back-compatibility issues.

[potiuk]

Checked that all my changes are in.

[jx2lee]

@eladkal
#40008 works fine! (unittest & example run)

[Owen-CH-Leung]

@eladkal the ElasticSearchSQLHook is now working as expected.

[vikramaditya91]

#41256: @vikramaditya91

Works fine

[kacpermuda]

#41494 tried to fix the OOM error in the scheduler that the OpenLineage can cause when generating a dag_tree from a huge DAG (related issue: #41505). It works but we've just got information about another production case where the scheduler went OOM with another complex DAG. There is a fix prepared in #41587 that will remove the dag_tree entirely so there will no more errors like this. I'd like to request an rc2 for OpenLineage provider (@eladkal) that will include that fix, as it is a bug that can cause some problems in bigger deployments.

[joaopamaral]

Tested #40703 with both access_control formats and it's working fine:

[eladkal]

@kacpermuda I will exclude openlineage from this release

[uzhastik]

ydb provider works fine: #41303

[ambika-garg]

Hi, #40356 work as expected

[dirrao]

Hi,
#41372 address the documentation changes. So, no functionality change.

[BTeclaw]

#41150 - Works as expected - unit test + functional check, details below:

1. Connection definition (different role and warehouse than is used on Operator definition)
   
2. DAG definition, mind the different warehouse, role and schema
   
3. Queries executed on a warehouse defined when declaring the SnowflakeSqlApiOperator and not the connection
   
4. DDL executing role is also properly forwarded through the SnowflakeSqlApiOperator
   

[perry2of5]

41142 passes test, but:

I noticed is that starting with 2.9.3 with microsoft-azure providers 10.3.0 the return value quits being put into XCOM (it is blank in the UI). Then with airflow 2.10.0 with microsoft-azure providers 10.3.0 the key shows up in XCOM but it says "No value found for XCom key". So something broke between 2.9.2 / 10.1.2 and 2.10.0/10.3.0.

With all that said, my actual change to put the last line (or all lines) of the logs into XCOM worked. So, I think we need a new defect logged to see why return value isn't showing up correctly any more.

Here is my operator. I'd been testing my changes with do_xcom_push=False since I didn't care about the normal return_value...obscured the fact something else broke :(

  aciOperator = AzureContainerInstancesOperator(
      ci_conn_id="azure-container-instance-conn-id",
      registry_conn_id="acr-conn-id",
      resource_group="redacted",
      name="http2blob{{ ds }}",
      image='redacted',
      region="WestUS2",
      environment_variables={
      redacted
      },
      volumes=[],
      memory_in_gb=1.0,
      cpu=1.0,
      task_id="start-download-aci",
      retries=0,
      do_xcom_push=True,
      # xcom_all=True,
      post_execute=_post_execute,
  )

Also, I did some more testing and multiple_outputs=True also fails back in 2.9.2 with microsoft-azure providers 10.1.2. This actually makes sense because the operator returns a single value, not a dictionary so I think this wasn't actually an issue. So, I'm saying this was tester error unless someone tells me otherwise.

[eladkal]

@perry2of5 there is only 1 question relevant here. Is there regression in apache-airflow-providers-microsoft-azure from 10.4.0rc1 to 10.3.0 ? All the rest is possible bugs that do not affect our decision about releasing.

Please clarify explicitly what worked on 10.3.0 and does not work anymore on 10.4.0rc1

[perry2of5]

I did not find any regression from 10.3.0 to 10.4.0rc1.

[eladkal]

I did not find any regression from 10.3.0 to 10.4.0rc1.

Then it's not blocking the release.
Feel free to raise PR to address the bugs you mentioned

[eladkal]

Thank you everyone. Providers are released.
Provider openlineage is excluded and will followup with rc2

I invite everyone to help improve providers for the next release, a list of open issues can be found here.