Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Status of testing Providers that were prepared on July 13, 2022 #25037

Closed
62 of 95 tasks
potiuk opened this issue Jul 13, 2022 · 35 comments
Closed
62 of 95 tasks

Status of testing Providers that were prepared on July 13, 2022 #25037

potiuk opened this issue Jul 13, 2022 · 35 comments
Labels
kind:meta High-level information important to the community testing status Status of testing releases

Comments

@potiuk
Copy link
Member

potiuk commented Jul 13, 2022 •
edited
Loading

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider airbyte: 3.1.0rc1

Provider alibaba: 2.0.1rc1

Provider amazon: 4.1.0rc1

Provider apache.drill: 2.1.0rc1

Provider apache.druid: 3.1.0rc1

Provider apache.hdfs: 3.0.1rc1

Provider apache.hive: 3.1.0rc1

Provider apache.pinot: 3.1.0rc1

Provider asana: 2.0.1rc1

Provider cncf.kubernetes: 4.2.0rc1

Provider databricks: 3.1.0rc1

Provider dbt.cloud: 2.0.1rc1

Provider docker: 3.1.0rc1

Provider elasticsearch: 4.1.0rc1

Provider exasol: 3.1.0rc1

Provider facebook: 3.0.1rc1

Provider ftp: 3.1.0rc1

Provider github: 2.1.0rc1

Provider google: 8.2.0rc1

Provider hashicorp: 3.0.1rc1

Provider http: 4.0.0rc1

Provider jdbc: 3.1.0rc1

Provider jira: 3.0.1rc1

Provider microsoft.azure: 4.1.0rc1

Provider microsoft.mssql: 3.1.0rc1

Provider mysql: 3.1.0rc1

Provider odbc: 3.1.0rc1

Provider oracle: 3.2.0rc1

Provider postgres: 5.1.0rc1

Provider presto: 3.1.0rc1

Provider qubole: 3.1.0rc1

Provider salesforce: 5.0.0rc1

Provider sftp: 3.1.0rc1 !!! NOTE this one s removed for breaking changes that need 4.0.0 release.

Provider slack: 5.1.0rc1

Provider snowflake: 3.1.0rc1

Provider sqlite: 3.1.0rc1

Provider ssh: 3.1.0rc1

Provider tableau: 3.0.1rc1

Provider trino: 3.1.0rc1

Provider vertica: 3.1.0rc1

The guidelines on how to test providers can be found in

Verify providers by contributors

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.
@potiuk potiuk added the kind:meta High-level information important to the community label Jul 13, 2022
@potiuk
Copy link
Member Author

potiuk commented Jul 13, 2022

@wylde780. - please install the RC1 SFTP provider from the link above.

@pankajkoti
Copy link
Member

I have a question regarding Provider http: 4.0.0rc1 and added it in the PR discussion here: #24967 (comment)

@potiuk
Copy link
Member Author

potiuk commented Jul 14, 2022

Thanks @pankajkoti - excellent point, I followed up with some extra comment/questions. I am happy to do either way (continue with self.hook removal, or restore it in RC2, and I think we should discuss what's best approach in general for deferred operators.

@phanikumv
Copy link
Contributor

Tested the below PR with the respective rc images and they are working fine.

Add test_connection method to Databricks hook (#24617): @phanikumv
Add test_connection method to GoogleBaseHook (#24682): @phanikumv
Add test_connection method to AzureCosmosDBHook (#25018): @phanikumv
Add test_connection method to AzureFileShareHook (#24843): @phanikumv
Add test_connection method to Azure WasbHook (#24771): @phanikumv

@bharanidharan14
Copy link
Contributor

Tested the following PR's with the respective rc image and its working fine.

Implement Azure service bus subscription (create, delete)Operators (#24625): @bharanidharan14
Implement Azure Service Bus Queue Operators (#24038): @bharanidharan14

@potiuk
Copy link
Member Author

potiuk commented Jul 14, 2022

I tested all my changes - mostly they were about common-sql extraction and dependencies. They all looks good!

@phanikumv
Copy link
Contributor

Completed testing the below PR and they are working fine.

Add test_connection method to Trino hook (#24583): @phanikumv
image

Add on_kill() to kill Trino query if Airflow task is killed (#24559): @phanikumv
Add TrinoOperator (#24415): @phanikumv

image

@igorborgest
Copy link
Contributor

I can confirm that the Amazon Appflow support is working as expected for apache-airflow-providers-amazon==4.1.0rc1.

image

@potiuk
Copy link
Member Author

potiuk commented Jul 14, 2022

cc: @Fokko - this is not mentioned here but it would be great if you test https://pypi.org/project/apache-airflow-providers-tabular/1.0.0rc1/

@josh-fell
Copy link
Contributor

cc @denimalpaca for https://pypi.org/project/apache-airflow-providers-common-sql/1.0.0rc1/ 🎉

@josh-fell
Copy link
Contributor

#24903 Looks good.

It's a little difficult to test #24582 since I don't have the means to test all of these providers in the wild, but I spot-checked the package files and 👍

@potiuk
Copy link
Member Author

potiuk commented Jul 14, 2022

It's a little difficult to test #24582 since I don't have the means to test all of these providers in the wild, but I spot-checked the package files and +1

Indeed.

@eskarimov
Copy link
Contributor

Tested new operators introduced in #24099, working correctly.

@alexkruc
Copy link
Contributor

Checked mine - #24663, #24895. Looks good.

@ishiis
Copy link
Contributor

ishiis commented Jul 14, 2022

#24473 is working as expected.

image

@bryan824
Copy link
Contributor

#23141 checked. Please see the details below for comparison.

apache-airflow-providers-google | 8.1.0

ValueError: When default_project is not set, table_id must be a fully-qualified ID in standard SQL format, e.g., "project.dataset_id.table_id", got test.table_to_delete

apache-airflow-providers-google | 8.2.0rc1

[2022-07-14 17:41:49,262] {bigquery.py:1182} INFO - Deleted table test.table_to_delete

@Taragolis
Copy link
Contributor

Working as expected: #24662 #24406 #24294
Also check depreciation warnings from #24670 - not listed here but included in apache-airflow-providers-amazon==4.1.0rc1

@potiuk
Copy link
Member Author

potiuk commented Jul 14, 2022

Yep #24670 has been removed from changelog as "not important" - we can re-add them later for the docs purpose see changelog with comments included https://raw.githubusercontent.com/apache/airflow/main/airflow/providers/amazon/CHANGELOG.rst)

@pauldalewilliams
Copy link
Contributor

@potiuk Re #24512 does the fact that I removed deprecated parameters necessitate a major version bump for the SFTP provider? See the second bullet point on my PR.

@potiuk
Copy link
Member Author

potiuk commented Jul 14, 2022

@potiuk Re #24512 does the fact that I removed deprecated parameters necessitate a major version bump for the SFTP provider? See the second bullet point on my PR.

Good point. Yes. We need to add a major release for it. I missed the removal. I will remove it from the release and make a new release for SFTP only.

@kevgeo
Copy link
Contributor

kevgeo commented Jul 14, 2022

I have tested #24860, works great!

@pasalkarsachin1
Copy link
Contributor

Looks okay to me #24726

@potiuk potiuk added the testing status Status of testing releases label Jul 15, 2022
@pankajastro
Copy link
Member

Tested #24306 with 4.1.0rc1 working fine.

@ankurbajaj9
Copy link
Contributor

ankurbajaj9 commented Jul 15, 2022 via email

@potiuk
Copy link
Member Author

potiuk commented Jul 15, 2022

I see an issue with (#24550) #24550.will try to raise a PR Regards Ankur Bajaj

How Serious? (BTW. it's no big problem we can remove it from the release and make rc2 right after - we are doing that with SFTP already).

@Fokko
Copy link
Contributor

Fokko commented Jul 15, 2022

I'll give the Tabular operator a swing right away. Thanks for running this @potiuk! 👏🏻

@Fokko
Copy link
Contributor

Fokko commented Jul 15, 2022

@potiuk would it be possible to include this patch into the RC2 as well? #25099

@potiuk
Copy link
Member Author

potiuk commented Jul 15, 2022

Sure. I will remove tabular from delivery too and add it to the list of rc2's

@alexott
Copy link
Contributor

alexott commented Jul 16, 2022

#24617 for Databricks has an issue because selected API is available only on AWS, and will fail on Azure/GCP...

More general question - is it possible to subscribe to all Databricks-related RPs?

@potiuk
Copy link
Member Author

potiuk commented Jul 16, 2022 •
edited
Loading

#24617 for Databricks has an issue because selected API is available only on AWS, and will fail on Azure/GCP...

Good check. That is not a regression though, test_connection is a new method/functionality added and it partially works, so it's not a reason to cancel it. It can be adressed in the next release.

More general question - is it possible to subscribe to all Databricks-related RPs?

We discussed it before, unfortunately CODEOWNER (which is a bad name BTW) only allows commiters to be marked as reviewer to part of the repo, which is a bummer (because it's not a codeowner by any means is just assigning people to be reviewers which has nothing to do with either being committer or owner).

For now, the best you can do (and it should be pretty accurate) is to subscribe to all notifiction and write a custom filter - if you are using gmail (that's what I'd do at least). It's actually pretty accurate. All PRs/commit messages with databrics contain list of changed files so subscribing to anything with "providers/databricks" should do the job nicely and you can choose how it should be surfaced in your email (and you can archive/store non-matching emails from commts@github elsewhere).

In the future (the mixed-governance was a pre-requisite to it) we plan to split providers to separate repos and then it will be much more flexible and you will be able to write more precise filters.

@ankurbajaj9
Copy link
Contributor

I see an issue with (#24550) #24550.will try to raise a PR Regards Ankur Bajaj

How Serious? (BTW. it's no big problem we can remove it from the release and make rc2 right after - we are doing that with SFTP already).

Not serious at all. It seems the authentication is missing since the original code only. Nothing is breaking , it just needs additional lines to be able to work in my environment.

@potiuk
Copy link
Member Author

potiuk commented Jul 16, 2022 •
edited
Loading

Not serious at all. It seems the authentication is missing since the original code only. Nothing is breaking , it just needs additional lines to be able to work in my environment.

Right. Then we can release it now and add fix in the next wave :)

@potiuk
Copy link
Member Author

potiuk commented Jul 16, 2022

@Fokko - I actually released 1.0.0 (will be easier) and release 1.0.1 right after - 1.0.0 was the first release, no regression there.

@potiuk
Copy link
Member Author

potiuk commented Jul 16, 2022

(both SFTP/Tabular will be rc1 then).

@potiuk
Copy link
Member Author

potiuk commented Jul 16, 2022

Closing this one as providers are released. Thanks everyone. you rock!

@potiuk potiuk closed this as completed Jul 16, 2022
@LaPetiteSouris LaPetiteSouris mentioned this issue Aug 11, 2022
Closed
98 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind:meta High-level information important to the community testing status Status of testing releases
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests