remove K8S 1.24 support

[raphaelauv]

end of GKE 1.24 support in 4 days -> https://endoflife.date/google-kubernetes-engine

[Taragolis]

[FIX] remove K8S 1.24 support what this PR fix? 🤔

[raphaelauv]

ok , I remove the prefix "fix"

[hussein-awala]

What about EKS users? (and AKS)

[raphaelauv]

Airflow policy is "until at least 2 major ( USA 😅😅 ) cloud provider support it

[hussein-awala]

( USA 😅😅 )

I think we should consider Alibaba Cloud as a major, no? Otherwise, we should replace "until at least 2 major" with "until at least 2 of (GCP, AWS, Azure)"

[potiuk]

I think we should consider Alibaba Cloud as a major, no? Otherwise, we should replace "until at least 2 major" with "until at least 2 of (GCP, AWS, Azure)"

If we include Alibaba (we did not when we set the policy), IMHO we should change to "at least 3 providers out of (GCP/AWS/Azure/Alibaba)" which does not change the outcome. PR maybe @hussein-awala explaining it ?

[hussein-awala]

I think we should consider Alibaba Cloud as a major, no? Otherwise, we should replace "until at least 2 major" with "until at least 2 of (GCP, AWS, Azure)"

If we include Alibaba (we did not when we set the policy), IMHO we should change to "at least 3 providers out of (GCP/AWS/Azure/Alibaba)" which does not change the outcome. PR maybe @hussein-awala explaining it ?

Okay, let's keep them 2 of the top 3, but IMHO, we should wait for the next minor version to do that, removing the support in a patch version doesn't make sense. If you agree, I will update the doc to explain what are the major cloud providers and when we can remove the support.

[potiuk]

I think we should consider Alibaba Cloud as a major, no? Otherwise, we should replace "until at least 2 major" with "until at least 2 of (GCP, AWS, Azure)"

If we include Alibaba (we did not when we set the policy), IMHO we should change to "at least 3 providers out of (GCP/AWS/Azure/Alibaba)" which does not change the outcome. PR maybe @hussein-awala explaining it ?

Okay, let's keep them 2 of the top 3, but IMHO, we should wait for the next minor version to do that, removing the support in a patch version doesn't make sense. If you agree, I will update the doc to explain what are the major cloud providers and when we can remove the support.

Absolutely Feel free and we can - I guess keep the discussion in PR and get few approvals for that.

Re - which version we drop support, we did not even plan to remove it in Patchlevel release. From our docs:

We drop support for those EOL versions in main right after EOL date, and it is effectively removed when we release the first new MINOR (Or MAJOR if there is no new MINOR version)

[raphaelauv]

we could just remove

Except for Kubernetes, a version stays supported by Airflow if two major cloud providers still provide support for it

since cloud providers are way more up to date with K8S versions and also Apache airflow is not astronomer and we could have the same policy that we have with python and postgres

wdyt ?

[potiuk]

since cloud providers are way more up to date with K8S versions and also Apache airflow is not astronomer and we could have the same policy that we have with python and postgres

I do not think the rule we have is "Astronomer" rule - it's been proposed by @ashb quite long time ago https://lists.apache.org/thread/fr3tdy4nc8ywo3p0hpy9h04nyv1kldsy, and a few people agreed it makes sense back then without opposal (coincidentally they were all from Astronomer + me :D) .

The discussion has (as I re-read it now) quite a good justification on why keep it until it is likely a number of people still use it. A lot of people keep they old clusters running for as long as they can and only forced migration will keep them to move. I believe k8s is a bit different than say "Postgres" - in the sense that it has many more internal components that might make things more difficult to migrate than say migrating the database. K8S is a conglomerate of about a hundred or so different - independently released components internally so there might be one small single thing that might prevent particular installation to migrate to a newer version without some serious pains.

But I agree, it would be quite a bit more consistent to have "EOL" as a cut-off time for "software" we release - because we do not run "service".

Maybe worth to restart the discussion on devlist and see what people think about it TODAY @raphaelauv ?. I'd support changing it to EOL only.

Just personally - I am (for now) generally fine with both approaches but I'd prefer following the "EOL" route. I see why it is good to keep it a little longer, but also I see how strictly following EOL from K8S might actually drive people to update things.
But this might be mandated in the future actually, so we might want to do it even now anticipating what will happen.

Putting my security hat on - we will likely in the future HAVE TO add the rule of not supporting EOL software because of security regulations. We cannot rely on Amazon or Google releasing fixes to K8S "to the public" and when we "put software on the market" (this is what making a release does) we have to make sure we do it without relying on software that does not have security fixes maintenance process (or us providing those fixes)

The landscape is changing a lot with Cyber Resilience Act and similar US legislations that all are going to be codified in 2024 and will likely have to be implemented 2/3 years from the moment they pass.

It will be legally and financially costly for companies NOT to upgrade the software with latest security fixes and use software that is known it will not receive security fixes. So basically what it might mean (this is what is the spirit of the regulations at least - the regulations and standards are not yet set in stone) that if someneone finds a security bug in K8S that is EOL and we (Airflow) rely on it, we (Airflow/ASF) will have to (somehow) make sure that our users will get a fix.

Making it clear that we do not support EOL software, removes that obligation from us. It is of course pretty messy with smaller libraries and their dependencies etc. But there, vendoring-in and relasing security fixes is much easier for us even if we need to, rather than advising our customers on how to patch their k8s. So explicitly marking EOL software as not supported in latest version of Airflow is simply smart thing to do to prevent any responsibilities we might get.

I think we will see our industry shifting in the coming years from "costly to migrate so let's not think about it" to "costly to NOT migrate so let's actively manage it". But we will yet see what effect those regulations might have. But anticipating that - I think using EOL as cut-off day for us is a good idea.

[ashb]

Yeah, both Amazon and Google have changed how they handle k8s versions more recently, so I'd be open to revisit this, but it should probably be an on-list discussion?

[raphaelauv]

@hussein-awala until rules update , we could already merge this ( since it follow current K8S policy )

[potiuk]

Yeah, both Amazon and Google have changed how they handle k8s versions more recently, so I'd be open to revisit this, but it should probably be an on-list discussion?

Absolutely.

[hussein-awala]

Yeah, both Amazon and Google have changed how they handle k8s versions more recently, so I'd be open to revisit this, but it should probably be an on-list discussion?

+1 to discuss this on dev list

[hussein-awala]

@hussein-awala until rules update , we could already merge this ( since it follow current K8S policy )

yes, I'm ok with that

[hussein-awala]

LGTM