airflow: Remove Airflow 2.8+ support

[kacpermuda]

Problem

As discussed on Slack channel, I believe it would be wise to avoid supporting Airflow 2.8+ in the openlineage-airflow package. This approach would encourage users to transition to the provider package.

Also, i think it should be mentioned that starting from Airflow 2.7, users should switch to provider package. There are some information about it in the Openlineage docs but i think there should be also some information included in main README file.

It's also important to clearly communicate that ongoing development and enhancements will be focused on the apache-airflow-providers-openlineage package, while the openlineage-airflow will primarily be updated for bug fixes.

Solution

I am not sure if proposed code changes are enough to "remove" the support for Airflow 2.8+. We are not including airflow itself in the requirements, so I see no single place to specify the exact constraints that would make it impossible to use with Airflow < 2.1.0, >2.8.0

I also adjusted the README so it's clear that this package will not receive as much attention as the native Airflow provider integration.

If there is any other place where this change should be mentioned, please let me know and i will adjust the code. The same goes for the way we want to communicate it, please feel free to leave a comment.

One-line summary:

Remove Airflow 2.8+ support, adjust documentation.

Checklist

• You've signed-off your work
• Your pull request title follows our guidelines
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• Your comment includes a one-liner for the changelog about the specific purpose of the change (if necessary)
• You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
• You've added a header to source files (if relevant)

---

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2023 contributors to the OpenLineage project

[codecov-commenter]

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (d573547) 84.43% compared to head (e55cf7b) 83.56%.

Files	Patch %	Lines
...ntegration/airflow/openlineage/airflow/__init__.py	66.66%	1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2330      +/-   ##
==========================================
- Coverage   84.43%   83.56%   -0.87%     
==========================================
  Files          59       54       -5     
  Lines        3348     3177     -171     
==========================================
- Hits         2827     2655     -172     
- Misses        521      522       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mobuchowski]

I don't think doing this in requirements is the right choice, as pypi works in a way that could result in reinstalling of Airflow for customers... and this is not something we want to do.

If we're doing is, the right way should be to not include listener in Airflow plugin, and warn users that this version of openlineage-airflow can't work with Airflow 2.8+

[kacpermuda]

@mobuchowski I removed the listener from the plugin for versions 2.8+, there is a warning emitted when Airflow 2.8+ is used.