Relax version checks for remote Edge Workers

[jscheffl]

A long lasting wish and desire was to be more relaxed on version checks between Edge/Core on the worker and on central site. We have learned now and gained in expertise such that now it is time to relax the exact checks. Versions to accept are now configurable as a starting point. Once we know better then we can adjust defaults in future.

DRAFT until tests have landed.

Version compatibility is now displayed via the extended system information capability that was added in Edge provider 3.5.0. Looks like this now:

Case 1 - version matching:

Case 2 - manually tweaked version to (non existing) 3.4.9:

Case 3 - Accepted an old Edge 3.3.0 worker which did not implement the system information push, thus no status known but online is accepted and can run jobs as compatible from API:

Case 4 - Used even older Edge 3.2.0 which I configured not to be accepted, fails starting as expected:

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[jscheffl]

FYI @AutomationDev85 - would be cool having a review from you as well.

[dheerajturaga]

@jscheffl, awesome! I think with globally distributed workers, this relaxing of the check is much needed.
One nit , the version mismatch message in system status can feel like something is wrong with the worker. Instead, I think we should probably still show the worker as healthy but with a "!" so that theres something more than just being healthy . Something like this. What do you think?

[jscheffl]

@jscheffl, awesome! I think with globally distributed workers, this relaxing of the check is much needed. One nit , the version mismatch message in system status can feel like something is wrong with the worker. Instead, I think we should probably still show the worker as healthy but with a "!" so that theres something more than just being healthy . Something like this. What do you think?

Mhm, I was so far populating the status based on logging levels (INFO, WARNING, ERROR) and did not plan adding more status levels just for this case.

As a counter proposal how about (1) relaxing the text to still report "Healty" but highlight the version mismatch

Or (2) also keep this informational and markt he version mismatch as info

If you dislike my original I'd rather pick (1) to inform admins that there is work to do.

[dheerajturaga]

@jscheffl, awesome! I think with globally distributed workers, this relaxing of the check is much needed. One nit , the version mismatch message in system status can feel like something is wrong with the worker. Instead, I think we should probably still show the worker as healthy but with a "!" so that theres something more than just being healthy . Something like this. What do you think?

Mhm, I was so far populating the status based on logging levels (INFO, WARNING, ERROR) and did not plan adding more status levels just for this case.

As a counter proposal how about (1) relaxing the text to still report "Healty" but highlight the version mismatch

Or (2) also keep this informational and markt he version mismatch as info

If you dislike my original I'd rather pick (1) to inform admins that there is work to do.

1 is fine, it doesn't essentially need to be green. As long as it states its healthy we should be good

[dheerajturaga]

Looks pretty good I was able to test this and it works as described. We are good to merge once the nit is addressed.
I was not able to cleanly test AIRFLOW__EDGE__MINIMUM_ACCEPTABLE_CORE_VERSION_FOR_WORKER
as the core version also needs to allow for mismatched version for testing

[jscheffl]

Looks pretty good I was able to test this and it works as described. We are good to merge once the nit is addressed. I was not able to cleanly test AIRFLOW__EDGE__MINIMUM_ACCEPTABLE_CORE_VERSION_FOR_WORKER as the core version also needs to allow for mismatched version for testing

Cool thing which I noticed is that with upgrading then also in this the existing can stay alive, so it is backwards compatible to installed base... tested down until Edge3 provider 3.3.0.

[dheerajturaga]

Looks pretty good I was able to test this and it works as described. We are good to merge once the nit is addressed. I was not able to cleanly test AIRFLOW__EDGE__MINIMUM_ACCEPTABLE_CORE_VERSION_FOR_WORKER as the core version also needs to allow for mismatched version for testing

Cool thing which I noticed is that with upgrading then also in this the existing can stay alive, so it is backwards compatible to installed base... tested down until Edge3 provider 3.3.0.

yes! thats a great relief

[dheerajturaga]

🚀