Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KubernetesRuntime translates pulsar object names to values that are RFC1123-compliant when generating Kubernetes Resource Specs #9360

Closed
csthomas1 opened this issue Jan 28, 2021 · 1 comment · Fixed by #9556
Labels
type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages

Comments

@csthomas1
Copy link
Contributor

Is your enhancement request related to a problem? Please describe.
Currently, it's valid to (via pulsar-admin and the admin rest api) specify names that include the colon ':' character. For example, I can create a namespace via pulsar-admin and the rest api as something like 'my:example:namespace'. We've found this useful for purposes of adding structure to object names that is then easier to programmatically parse.

We've been able to use this format with no problems with the ProcessRuntime for deploying functions.

The KubernetesRuntime, however, is currently not compatible with pulsar object names that are not compliant with RFC1123, because KubernetesRuntime attempts to use these object names verbatim in the labels it attaches to the resource specifications.
Trying to use the KubernetesRuntime with the name format described above prevents the functions from deploying successfully.

More detail provided by @sijie :

there is a disconnection pulsar name verification and kubernetes name verification. Pulsar supports more legal characters than kubernetes naming convention. The disconnection causes an issue when using KubernetesRuntime to run functions and connectors. Kubernetes requires a name that can be used as a DNS subdomain name as defined in RFC 1123 (https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names). If the name doesn’t follow the rule, you will fail to run functions/connectors using kubernetes runtime.
However, we can’t change the pulsar naming convention to use the DNS subdomain name convention because there are already many applications built before we introduce functions.

Describe the solution you'd like
The KubernetesRuntime could translate the names of the pulsar objects to forms that are RFC1123-compliant. This could be as simple as replacing all illegal characters with a '-', for example. The KubernetesRuntime already does something similar in generating the pod names.

Describe alternatives you've considered
Without this feature, we'll need to either change our naming convention (which works well for us) or go back to using the ProcessRuntime. We'd really like to be able to use the KubernetesRuntime for the scaling benefits.

@freeznet
Copy link
Contributor

freeznet commented Feb 2, 2021

@csthomas1 yes we should create a conversion for namespace as well like we do for the pod names. But I think Pulsar already provide a ManifestCustomizer interface for k8s, so you can implement the interface to make your own customizer. Also pr is welcome to add the rfc1123 conversion for namespace in k8s runtime.

ref:

default String customizeNamespace(Function.FunctionDetails funcDetails, String currentNamespace) {

sijie pushed a commit that referenced this issue Feb 16, 2021
…c1123 compliant labels (#9556)

Fixes #9360

### Motivation

Currently, it's valid to (via pulsar-admin and the admin rest api) specify names that include the colon ':' character. For example, I can create a namespace via pulsar-admin and the rest api as something like 'my:example:namespace'. We've found this useful for purposes of adding structure to object names that is then easier to programmatically parse.

We've been able to use this format with no problems with the ProcessRuntime for deploying functions.

The KubernetesRuntime, however, is currently not compatible with pulsar object names that are not compliant with RFC1123, because KubernetesRuntime attempts to use these object names verbatim in the labels it attaches to the resource specifications.
Trying to use the KubernetesRuntime with the name format described above prevents the functions from deploying successfully.

### Modifications

Changed the KubernetesRuntime to translate the names of the pulsar objects to forms that are RFC1123-compliant for k8s resource labels. 

The rules for translating the Pulsar object names are:
- truncate to 63 chars
- replace any non alphanumeric character ([a-z0-9A-Z]), dashes (-), underscores(_), dot(.) with "-"
- replace beginning & end non-alphanumeric character with "0"

This change added tests and can be verified as follows:
  - Added an extra test to KubernetesRuntimeTest.java to make sure labels are created that are RFC1123-compliant AND also make sure the labels are no bigger than the k8s limit of 63 chars
codelipenghui pushed a commit that referenced this issue Feb 18, 2021
…c1123 compliant labels (#9556)

Fixes #9360

### Motivation

Currently, it's valid to (via pulsar-admin and the admin rest api) specify names that include the colon ':' character. For example, I can create a namespace via pulsar-admin and the rest api as something like 'my:example:namespace'. We've found this useful for purposes of adding structure to object names that is then easier to programmatically parse.

We've been able to use this format with no problems with the ProcessRuntime for deploying functions.

The KubernetesRuntime, however, is currently not compatible with pulsar object names that are not compliant with RFC1123, because KubernetesRuntime attempts to use these object names verbatim in the labels it attaches to the resource specifications.
Trying to use the KubernetesRuntime with the name format described above prevents the functions from deploying successfully.

### Modifications

Changed the KubernetesRuntime to translate the names of the pulsar objects to forms that are RFC1123-compliant for k8s resource labels. 

The rules for translating the Pulsar object names are:
- truncate to 63 chars
- replace any non alphanumeric character ([a-z0-9A-Z]), dashes (-), underscores(_), dot(.) with "-"
- replace beginning & end non-alphanumeric character with "0"

This change added tests and can be verified as follows:
  - Added an extra test to KubernetesRuntimeTest.java to make sure labels are created that are RFC1123-compliant AND also make sure the labels are no bigger than the k8s limit of 63 chars

(cherry picked from commit 5763c0e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages
Projects
None yet
2 participants