Rewrite services being injected to k8s to be lowercase because K8S DNS implementation sucks

[ejmarten]

I have my external consul doing a two way sync with my k8s (v1.15.0) service registry/coreDNS,
On inspecting the log for the consul-sync-catalog pod, I see an entry like the following for each service registered in consul with uppercase characters.

I jumped in the external-dns channel in k8s slack, and was told that kubernetes does not support uppercase names.

Well crud. Now I have to either write a wrapper for k8s-consul, create a docker image with a non-standard coredns binary that supports the rewrite plugin, or hotfix hundreds of consul services...

Guess which one I refuse to do?

2019-06-26T19:55:09.033Z [WARN ] to-k8s/sink: error creating service: name=consul-ORDER-API error="Service "consul-ORDER-API" is invalid: [metadata.name: Invalid value: "consul-ORDER-API": a DNS-1035 label must consist of lower case alphanumeric characters or '-', st
art with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for validation is 'a-z?'), spec.externalName: Invalid value: "ORDER-API.service.cluster.local": a DNS-1123 subdomain must consist
of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is 'a-z0-9?(.a-z0-9?)*')]"

I'm working on a workaround, but this is most definitely a bug, I'll share my results if I figure it out.

[lkysow]

I think we'll have to register them as lower case then. Note that in Consul registering services is case sensitive (you can have two services foo and FOO hashicorp/consul#5707) but then most of the endpoints like consul DNS will only return one of them.

[ejmarten]

This might work. I'll test and submit a PR if it does.

[root@dev]# grep -rl ToLower catalog/from-consul/source.go
catalog/from-consul/source.go
[root@dev]# grep -rn ToLower catalog/from-consul/source.go
72: services[s.Prefix+name] = fmt.Sprintf("%s.service.%s", strings.ToLower("name"), strings.ToLower("s.Domain"))

[ejmarten]

That compiled and seemed to work, but running it in a pod logs:
"Bad substitution"

...