fix: ExternalTaskSensor check_existence ignored in deferrable mode (Airflow < 3.0)

[dzpan0]

Root cause

In execute(), the deferrable path calls self.defer() directly without first calling _check_for_existence.

Fix

For Airflow < 3.0, call _check_for_existence inside a scoped create_session block before running self.defer() if check_existence flag is set to True.

This fix only works for versions below 3.0 since there is no direct way to assert DAG existence on higher versions. One possible solution would be creating a new RPC call to try to retrieve a specific DAG, but I'm not sure on creating it just for this reason.

closes: #40745

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Nataneljpwd]

Hello, I do not think there will be another provider release for airflow 2.x, if this does occur airflow 3, and it is a broader issue than just this sensor, a new rpc for the supervisor might be worth it
The direct db access won't work with airflow 3, as providers no longer have it, and so this will throw an error on airflow 3 and up, making it a breaking change that works only for lower versions

[dzpan0]

The check is encapsulated in the else block of "if AIRFLOW_V_3_0_PLUS", so it should not affect versions >=3.0. If the new rpc seems to be a reasonable new implementation, then I'll start working on it. Otherwise, deprecating this flag for higher versions would be an option, as it doesn't seem that impactful to have it.

[Nataneljpwd]

The check is encapsulated in the else block of "if AIRFLOW_V_3_0_PLUS", so it should not affect versions >=3.0. If the new rpc seems to be a reasonable new implementation, then I'll start working on it. Otherwise, deprecating this flag for higher versions would be an option, as it doesn't seem that impactful to have it.

Looks good, just add a test case and I think it's fine, when airflow 2 reaches end of life it can be removed, if you can also (in a different PR) fix it for airflow 3, it would be appreciated

Some of the failing tests are due to celery and unrelated, can be ignored for now, but I did not look at all of the checks

[Lee-W]

overall looks good

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.