Use apache-airflow-providers-openlineage, bump min Airflow version to 2.7 and Python version to 3.8
#2103
Conversation
pankajastro
force-pushed
the
update_ol_lib
branch
from
7110c40
to
c69ce5a
Compare
for more information, see https://pre-commit.ci
| @@ -87,7 +87,7 @@ ftp = [ | |||
| "apache-airflow-providers-ftp>=3.0.0", | |||
| "smart-open>=5.2.1", | |||
| ] | |||
| openlineage = ["openlineage-airflow>=0.17.0"] | |||
| openlineage = ["apache-airflow-providers-openlineage>=1.4.0"] | |||
There was a problem hiding this comment.
Note that the OpenLineage provider only supports Airflow versions greater than or equal to 2.7.0.
For compatibility with older Airflow versions, wouldn't there be dependency conflict on strict requirement with [openlineage] extra?
There was a problem hiding this comment.
yes, thanks a lot @JDarDagran for the suggestion
Bumping up min Airflow version to 2.7 and min Python version to 3.8
There was a problem hiding this comment.
most of the airflow provider has apache-airflow>=2.6.0 as a dependency. hopping bumping 2.7.0 will not impact users
There was a problem hiding this comment.
We will release a minor version of Astro SDK and it should not cause a problem IMHO. If users would like to use the new minor version, they would need to have Airflow 2.7. For earlier versions, they can continue using prior versions of Astro SDK.
Similar fashion is followed in Airflow OSS providers when bumping versions for dependencies
There was a problem hiding this comment.
Besides we cannot have apache-airflow-providers-openlineage without bumping Airflow to 2.7
pankajkoti
force-pushed
the
update_ol_lib
branch
from
343abea
to
b1397cb
Compare
pankajkoti
changed the title
openlineage-airflow to apache-airflow-providers-openlineageapache-airflow-providers-openlineage, bump min Airflow version to 2.7 and min Python version to 3.8
pankajkoti
changed the title
apache-airflow-providers-openlineage, bump min Airflow version to 2.7 and min Python version to 3.8apache-airflow-providers-openlineage, bump min Airflow version to 2.7 and Python version to 3.8
pankajkoti
requested review from
dimberman,
tatiana,
utkarsharma2 and
sunank200
as code owners
pankajkoti
requested review from
a team,
phanikumv,
ephraimbuddy,
pankajkoti and
JDarDagran
- Replace openlineage-airflow with Apache Airflow OSS provider apache-airflow-providers-openlineage #2103 - Bump up minimum version of apache-airflow to 2.7 #2103 - Bump up minimum version of Python to 3.8 #2103 - Limit pandas version to <2.2.0 due to an open issue for the release pandas==2.2.0 #2105
Fixes issue reported where deployments don't contain the
openlineage-airflowlibrary installed but instead contain
apache-airflow-providers-openlineage.To fix the issue we transform the usages from
openlineage-airflowtocorresponding available source code in the
apache-airflow-providers-openlineageAirflow OSS provider.
Since the apache-airflow-providers-openlineage depends on Airflow >= 2.7,
we also correspondingly bump the Airflow version.
Airflow 2.7.0 is also the first release that drops support for end-of-life Python 3.7,
and hence, we bump up the min Python version to 3.8
related: https://github.com/astronomer/issues-airflow/issues/503