Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Status of testing Providers that were prepared on May 26, 2024 #39842

Closed
eladkal opened this issue May 26, 2024 · 20 comments
Closed

Status of testing Providers that were prepared on May 26, 2024 #39842

eladkal opened this issue May 26, 2024 · 20 comments
Labels
kind:meta High-level information important to the community testing status Status of testing releases

Comments

@eladkal
Copy link
Contributor

eladkal commented May 26, 2024 •
edited

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider alibaba: 2.8.1rc1

Provider amazon: 8.23.0rc1

Provider apache.beam: 5.7.1rc1

Provider apache.druid: 3.10.1rc1

Provider apache.hive: 8.1.1rc1

Provider apache.kafka: 1.4.1rc1

Provider apprise: 1.3.1rc1

Provider atlassian.jira: 2.6.1rc1

Provider celery: 3.7.1rc1

Provider cncf.kubernetes: 8.3.0rc1

Provider common.io: 1.3.2rc1

Provider common.sql: 1.14.0rc1

Provider databricks: 6.5.0rc1

Provider dbt.cloud: 3.8.1rc1

Provider docker: 3.12.0rc1

Provider elasticsearch: 5.4.1rc1

Provider fab: 1.1.1rc1

Provider google: 10.19.0rc1

Provider hashicorp: 3.7.1rc1

Provider microsoft.mssql: 3.7.1rc1

Provider openlineage: 1.8.0rc1

Provider pinecone: 2.0.0rc3

Provider postgres: 5.11.1rc1

Provider sftp: 4.10.1rc1

Provider snowflake: 5.5.1rc1

Provider yandex: 3.11.1rc1

All users involved in the PRs:
@pankajastro @dabla @sunank200 @e-galan @shahar1 @boraberke @JDarDagran @0x26res @ellisms @EricGao888 @grrolland @danielbe11 @tatiana @kacpermuda @gaurav7261 @gopidesupavan @mobuchowski @Lee-W @eladkal

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.
@eladkal eladkal added kind:meta High-level information important to the community testing status Status of testing releases labels May 26, 2024
@uzhastik
Copy link
Contributor

uzhastik commented May 26, 2024 •
edited

Provider yandex: 3.11.1rc1

Looks good, this version has been checked on my test dag and old active airflow server.
Is it possible to preview changes in doc? I have one PR in doc.

@gopidesupavan
Copy link
Contributor

Hi @eladkal , i have tested #39592 , it is working fine. thanks.

@potiuk
Copy link
Member

potiuk commented May 26, 2024

All my changes checked.

@shahar1
Copy link
Collaborator

shahar1 commented May 26, 2024

#39695 is working fine

airflow-oss-bot added a commit to astronomer/astronomer-providers that referenced this issue May 27, 2024
@airflow-oss-bot
Updating setup.cfg with RC provider packages on apache/airflow#39842
ea38d8a
airflow-oss-bot added a commit to astronomer/astro-sdk that referenced this issue May 27, 2024
@airflow-oss-bot
Updating setup.cfg with RC provider packages on apache/airflow#39842
5de9c08
@EricGao888
Copy link
Member

Hi @eladkal , I've tested #39627 and it is working fine. THX : )

@grrolland
Copy link
Contributor

grrolland commented May 27, 2024 via email

airflow-oss-bot added a commit to astronomer/astronomer-providers that referenced this issue May 27, 2024
@airflow-oss-bot
Updating setup.cfg with RC provider packages on apache/airflow#39842
d112117
airflow-oss-bot added a commit to astronomer/astronomer-providers that referenced this issue May 27, 2024
@airflow-oss-bot
Updating setup.cfg with RC provider packages on apache/airflow#39842
58c58fa
airflow-oss-bot added a commit to astronomer/astronomer-providers that referenced this issue May 27, 2024
@airflow-oss-bot
Updating setup.cfg with RC provider packages on apache/airflow#39842
fd9ae1e
airflow-oss-bot added a commit to astronomer/astronomer-providers that referenced this issue May 27, 2024
@airflow-oss-bot
Updating setup.cfg with RC provider packages on apache/airflow#39842
5c95a50
airflow-oss-bot added a commit to astronomer/astronomer-providers that referenced this issue May 27, 2024
@airflow-oss-bot
Updating setup.cfg with RC provider packages on apache/airflow#39842
bdc809a
airflow-oss-bot added a commit to astronomer/astronomer-providers that referenced this issue May 27, 2024
@airflow-oss-bot
Updating setup.cfg with RC provider packages on apache/airflow#39842
1cc803b
airflow-oss-bot added a commit to astronomer/astronomer-providers that referenced this issue May 27, 2024
@airflow-oss-bot
Updating setup.cfg with RC provider packages on apache/airflow#39842
ed3cad2
@Lee-W
Copy link
Member

Lee-W commented May 27, 2024

Hi, @vatsrahul1001 found an issue in #39112, and I just sent a hotfix for this #39874

@e-galan
Copy link
Contributor

e-galan commented May 27, 2024

#39640 , #39668 , #39752 are good

@e-galan
Copy link
Contributor

e-galan commented May 27, 2024

#39659 is good

@josh-fell
Copy link
Contributor

All of my changes look good! Thanks @eladkal!

@amoghrajesh
Copy link
Contributor

Tested my changes #39658, and tested few example DAGs as well. Looks good!

@kacpermuda
Copy link
Contributor

All my changes look good.

tatiana added a commit to astronomer/astronomer-cosmos that referenced this issue May 28, 2024
@tatiana
The issue we were trying to capture for no longer happens in the late…
e822176 
…st version of the Apache Airflow providers:

apache/airflow#39842

apache-airflow-providers-postgres==5.11.1rc1
@tatiana tatiana mentioned this issue May 28, 2024
Merged
tatiana added a commit to astronomer/astronomer-cosmos that referenced this issue May 28, 2024
@tatiana
The issue we were trying to capture for no longer happens in the late…
38a5f99 
…st version of the Apache Airflow providers:

apache/airflow#39842

apache-airflow-providers-postgres==5.11.1rc1
@ellisms
Copy link
Contributor

ellisms commented May 28, 2024

#38287 is good.

@VladaZakharova
Copy link
Contributor

Changes #39711 and #39371 @MaksYermak work well

@pankajastro
Copy link
Member

#38423 and #39583 looks good. thank you!

tatiana added a commit to astronomer/astronomer-cosmos that referenced this issue May 28, 2024
@tatiana
The issue we were trying to capture for no longer happens in the late…
509ae99 
…st version of the Apache Airflow providers:

apache/airflow#39842

apache-airflow-providers-postgres==5.11.1rc1
@tatiana tatiana mentioned this issue May 28, 2024
Merged
@boraberke
Copy link
Contributor

#38896 looks good. Thanks!

tatiana added a commit to astronomer/astronomer-cosmos that referenced this issue May 29, 2024
@tatiana
Fix CI issues (#1005)
32c47ca 
Fix three problems observed while running the tests in the CI recently:

1. Static check

Fix spelling issue captured by static check (#1000)

2. Integration test that's no longer needed

The issue we were trying to capture for no longer happens in the latest
version of the Apache Airflow provider
`apache-airflow-providers-postgres==5.11.1rc1`:
apache/airflow#39842

3. Skip a buggy version of OL

There was a breaking change between
`openlineage-integration-common==1.14.0` and
`openlineage-integration-common==1.15.0` . It may have been an
unintended side-effect of
OpenLineage/OpenLineage#2693.

This is an example of how Cosmos was using `DbtLocalArtifactProcessor` -
something that had been agreed upon in the past:

```
      openlineage_processor = DbtLocalArtifactProcessor(
            producer=OPENLINEAGE_PRODUCER,
            job_namespace=LINEAGE_NAMESPACE,
            project_dir=project_dir,
            profile_name=self.profile_config.profile_name,
            target=self.profile_config.target_name,
        )
        events = openlineage_processor.parse()
        for completed in events.completes:
            for output in getattr(completed, source):
                dataset_uri = output.namespace + "/" + output.name
                uris.append(dataset_uri)
```

In `openlineage-integration-common==1.14.0` and earlier versions, this
would create URIs in the format:
```
postgres://0.0.0.0:5432/postgres.public.stg_customers
```
Since openlineage-integration-common==1.15.0 , this leads to URIs being
created in the format:
```
postgres.public.stg_customers/postgres://0.0.0.0:5432
```

This was fixed in OpenLineage/OpenLineage#2735
and released as part of OL 1.16:
https://github.com/OpenLineage/OpenLineage/releases/tag/1.16.0
tatiana added a commit to astronomer/astronomer-cosmos that referenced this issue May 29, 2024
@pre-commit-ci @tatiana
⬆ [pre-commit.ci] pre-commit autoupdate (#1000)
d695c18 
<!--pre-commit.ci start-->
updates:
- [github.com/codespell-project/codespell: v2.2.6 →
v2.3.0](codespell-project/codespell@v2.2.6...v2.3.0)
- [github.com/astral-sh/ruff-pre-commit: v0.4.4 →
v0.4.5](astral-sh/ruff-pre-commit@v0.4.4...v0.4.5)
<!--pre-commit.ci end-->

Additionally, given the latest
`apache-airflow-providers-postgres==5.11.1rc1`, mentioned in
apache/airflow#39842, it no longer validates
the Dataset URI as it used to - so the test case we had in Cosmos is no
longer relevant.

Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
@Joffreybvn
Copy link
Contributor

#39690 is fine - No more warning logs when executing a SQL query !

@msoni1369
Copy link
Contributor

msoni1369 commented May 29, 2024 via email

@eladkal
Copy link
Contributor Author

eladkal commented May 30, 2024

Thank you everyone. Providers are released.
Provider cncf.kubernetes is excluded and will have rc2 shortly

I invite everyone to help improve providers for the next release, a list of open issues can be found here.

@eladkal eladkal closed this as completed May 30, 2024
@danielbe11
Copy link
Contributor

I see it's closed, but all good with me too. Thanks!

tatiana added a commit to astronomer/astronomer-cosmos that referenced this issue Jun 6, 2024
@tatiana
Fix CI issues (#1005)
895279e 
Fix three problems observed while running the tests in the CI recently:

1. Static check

Fix spelling issue captured by static check (#1000)

2. Integration test that's no longer needed

The issue we were trying to capture for no longer happens in the latest
version of the Apache Airflow provider
`apache-airflow-providers-postgres==5.11.1rc1`:
apache/airflow#39842

3. Skip a buggy version of OL

There was a breaking change between
`openlineage-integration-common==1.14.0` and
`openlineage-integration-common==1.15.0` . It may have been an
unintended side-effect of
OpenLineage/OpenLineage#2693.

This is an example of how Cosmos was using `DbtLocalArtifactProcessor` -
something that had been agreed upon in the past:

```
      openlineage_processor = DbtLocalArtifactProcessor(
            producer=OPENLINEAGE_PRODUCER,
            job_namespace=LINEAGE_NAMESPACE,
            project_dir=project_dir,
            profile_name=self.profile_config.profile_name,
            target=self.profile_config.target_name,
        )
        events = openlineage_processor.parse()
        for completed in events.completes:
            for output in getattr(completed, source):
                dataset_uri = output.namespace + "/" + output.name
                uris.append(dataset_uri)
```

In `openlineage-integration-common==1.14.0` and earlier versions, this
would create URIs in the format:
```
postgres://0.0.0.0:5432/postgres.public.stg_customers
```
Since openlineage-integration-common==1.15.0 , this leads to URIs being
created in the format:
```
postgres.public.stg_customers/postgres://0.0.0.0:5432
```

This was fixed in OpenLineage/OpenLineage#2735
and released as part of OL 1.16:
https://github.com/OpenLineage/OpenLineage/releases/tag/1.16.0
tatiana added a commit to astronomer/astronomer-cosmos that referenced this issue Jun 6, 2024
@pre-commit-ci @tatiana
⬆ [pre-commit.ci] pre-commit autoupdate (#1000)
df2b3ac 
<!--pre-commit.ci start-->
updates:
- [github.com/codespell-project/codespell: v2.2.6 →
v2.3.0](codespell-project/codespell@v2.2.6...v2.3.0)
- [github.com/astral-sh/ruff-pre-commit: v0.4.4 →
v0.4.5](astral-sh/ruff-pre-commit@v0.4.4...v0.4.5)
<!--pre-commit.ci end-->

Additionally, given the latest
`apache-airflow-providers-postgres==5.11.1rc1`, mentioned in
apache/airflow#39842, it no longer validates
the Dataset URI as it used to - so the test case we had in Cosmos is no
longer relevant.

Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind:meta High-level information important to the community testing status Status of testing releases
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests