Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow longer pod names for k8s executor / KPO #27736

Merged
merged 4 commits into from Dec 5, 2022

Conversation

dstandish
Copy link
Contributor

@dstandish dstandish commented Nov 17, 2022

Previously this limited pod length to 63 but that's the limit for label values not pod ids.

Meanwhile, a UUID was previously used which occupies 32 characters, much more space than necessary. This left you with a short and not very readable pod name.

Here we update the logic to use a suffix of 8 random alphanum chars, and set a longer max length of 80 characters.

One qualification on that. We set a max length of 80 when we control the pod name -- e.g. when using kube executor or KPO when pod name not supplied. Since we're generating it, 80 is a reasonable max. But, with KPO, when pod_id, and user asks for random suffix, we set max length to 253, to give user most control.

Along the way, we consolidate the name building logic into one function. Previously, one function would create the "pod_id" and then, only later in the process would a random suffix be added. Here, all of that is done in create_pod_id, and its behavior is controllable with params. cc @o-nikolas

Why 80 characters: that's about as much as can reasonably fit on a normal-sized terminal. Any more doesn't help with readability.

Why 8 character suffix: 8 random characters is more than enough. Probably fewer would be fine too.

Finally, a note.... I copied the core code to provider and we must keep it there until min airflow version catches up. And instead of duplicating test code, i parameterized the core test to also handle provider.


Now, an example.

Calling with

PodGenerator.make_unique_pod_id("this-is-my-semi-long-but-not-too-long-pod-id")

We would previously get this:

this-is-my-semi-long-but-not-t-c27ae4725b2049d6bf953e9f1f3d14c1

And now we'd get this:

this-is-my-semi-long-but-not-too-long-pod-id-l4hiiqxh

@boring-cyborg boring-cyborg bot added the provider:cncf-kubernetes Kubernetes provider related issues label Nov 17, 2022
@dstandish dstandish force-pushed the allow-longer-k8s-pod-ids branch 2 times, most recently from c4894d4 to ba90692 Compare November 18, 2022 07:08
@dstandish dstandish force-pushed the allow-longer-k8s-pod-ids branch 2 times, most recently from 3f70863 to fe06764 Compare November 20, 2022 07:44
@potiuk
Copy link
Member

potiuk commented Nov 25, 2022

Errors

@dstandish dstandish added the disable warnings Run pytest with --disable-warnings label Dec 2, 2022
Previously this limited pod length to 63 but that's the limit for label values not pod ids.  I assume that's just an atavism.  Meanwhile, a UUID was previously used which takes up 32 characters, much more space than necessary.  We use 8 random alphanum chars now which should be plenty.
@dstandish dstandish removed the disable warnings Run pytest with --disable-warnings label Dec 3, 2022
@potiuk potiuk merged commit d932406 into apache:main Dec 5, 2022
@dstandish dstandish added this to the Airflow 2.6.0 milestone Dec 8, 2022
@pierrejeambrun pierrejeambrun added the type:improvement Changelog: Improvements label Feb 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
provider:cncf-kubernetes Kubernetes provider related issues type:improvement Changelog: Improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants