Mark daskexecutor provider as removed

[potiuk]

Following the discussion in airflow devlist we mark daskexecutor as removed. See:
https://lists.apache.org/thread/fxv44cqqljrrhll3fdpdgc9h9fz5ghcy

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[pankajkoti]

CI is green

[potiuk]

ok. Now we just need to release the last one with the "remove" warnings in docs and pypi and then we can remove the code of it completely.

cc: @eladkal -> maybe we can make a separte ad-hoc relese for that one ? Not insisting, it can be done in a next wave as well, but I thought we want to remove it faster. I am even happy to do the release process myself (woudl be a great opportunity to walk through the process again - I have not done it for quite some time).

[eladkal]

, it can be done in a next wave

Next wave is really close as we are bumping min airflow version for providers today/tommorow

[potiuk]

Right.... Good timing :)

[MaiHoangViet1809]

I know it is late to comment this but we're using DaskExecutor here in production and it quite good, config easier than celery as it not require broker backend like redis.

[potiuk]

I know it is late to comment this but we're using DaskExecutor here in production and it quite good, config easier than celery as it not require broker backend like redis.

And you can still use it, it will just not be maintained any more by the community. You can keep latest apache-airflow-provider-daskexecutor provider installed in future versions of Airflow and keep using it. No problem with that - it will just not be maintained and tested in the main version of Airflow. We will keep on releasing and fixing security issues (if found) but we will not add new few features and will not keep up with latest dask libraries.

This is possible thanks to implementing "AIP-51 Remove Executor Coupling" https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airflow - where we designed and implemented public executor API that is intended to be backwards-compatible in the future. Thanks to that, latest released daskexecutor provider https://pypi.org/project/apache-airflow-providers-daskexecutor/ should work with future Airflow versions.

Of course if someone would like to keep on maintaining it, adding new libraries, possibly adding new features, and release a new executor based on the daskexecutor code that we have here tagged - they are absolutely free to do so. Our code is released with Apache Software Licence which allows anyone to do anything with the code and make a fork and release updated provider version on their own. We actually encourage that and will be happy to merge a change with link to such fork to our https://airflow.apache.org/ecosystem/ page where we have multiple providers, plugins, extensions, tools etc. We know also a number of people and teams are working on their own executors (again, following the clean API implemented and published with AIP-51), so having somoene developing and releasing Dask Executor, that someone would like to commit to maintain and release in the future would be one of those.

[vladvisan]

I feel like the following pages could be updated to make it clear that the Dask Executor was discontinued?

1. https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/dask.html
   • 2.8.4. branch version: https://github.com/apache/airflow/blob/2.8.4/docs/apache-airflow/core-concepts/executor/dask.rst
   • Page already removed from the "main" branch https://github.com/apache/airflow/blob/main/docs/apache-airflow/core-concepts/executor/dask.rst
   • Yet still appears on the airflow documentation
   • Maybe it would be good to include a "Dask was recently removed" sentence instead of removing the document entirely.
2. https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html
   • 2.8.4. branch version: https://github.com/apache/airflow/blob/2.8.4/docs/apache-airflow/core-concepts/executor/index.rst
   • Dask reference already removed from "main" branch https://github.com/apache/airflow/blob/main/docs/apache-airflow/core-concepts/executor/index.rst
   • Yet still appears on the Airflow documentation
   • Maybe it would be good to include a "Dask was recently removed" sentence instead of removing the reference entirely.

I spent some time searching for the DASK Executor install instructions, before realising that it had been removed.
This could avoid this confusion for other people in the future.
Thank you.

[Taragolis]

Page already removed from the "main" branch

Yet still appears on the airflow documentation

This changes will be reflected into Airflow 2.9, as part of #36269

[vladvisan]

Thank you!