kubernetes_service starts with invalid labels

[zmb3]

Today, we don't allow spaces in label keys.

If you attempt to start the SSH service with an invalid label, the service fails to start:

ssh_service:
  enabled: yes
  labels:
    'project id': invalid

This config produces the following failure:

2023-03-06T18:47:49-07:00 WARN [PROC:1]    Teleport process has exited with error. error:[
ERROR REPORT:
Original Error: *trace.BadParameterError invalid label key: "project id"
Stack Trace:
	github.com/gravitational/teleport/lib/srv/regular/sshserver.go:474 github.com/gravitational/teleport/lib/srv/regular.SetLabels.func1
	github.com/gravitational/teleport/lib/srv/regular/sshserver.go:738 github.com/gravitational/teleport/lib/srv/regular.New
	github.com/gravitational/teleport/lib/service/service.go:2313 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initSSH.func1
	github.com/gravitational/teleport/lib/service/supervisor.go:539 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
	github.com/gravitational/teleport/lib/service/supervisor.go:276 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
	runtime/asm_arm64.s:1172 runtime.goexit
User Message: invalid label key: "project id"] pid:19942.1 service:ssh.node service/supervisor.go:281
2023-03-06T18:47:49-07:00 ERRO [PROC:1]    Critical service ssh.node has exited with error invalid label key: "project id", aborting. pid:19942.1 service/signals.go:142

If you apply the same config to the Kubernetes service, the service starts correctly.

kubernetes_service:
  enabled: yes
  kubeconfig_file: /Users/zmb/t/kubeconfig
  listen_addr: 0.0.0.0:3027
  labels:
    'project id': invalid

On master, I do see heartbeat failures, but they only log warnings and the service otherwise appears to run.

2023-03-06T18:47:23-07:00 WARN [KUBERNETE] Heartbeat failed invalid label key: "project id". srv/heartbeat.go:282

Later on, when a user attempts to tsh login they receive an error:

If browser window does not open automatically, open it by clicking on the link:
 http://127.0.0.1:60382/9e0e9ee2-8896-4ffb-b814-7d0fac723203
ERROR: identity provider callback failed with error: Failed to create session certificate.
"\tinvalid label key: \"project id\""

At the same time, the kube agent shows the following:

ERROR REPORT:
Original Error: *trace.TraceErr invalid label key: "projejct id"
Stack Trace:
	github.com/gravitational/teleport/api@v0.0.0/client/client.go:2755 github.com/gravitational/teleport/api/client.GetResourcesWithFilters
	github.com/gravitational/teleport/api@v0.0.0/client/client.go:1113 github.com/gravitational/teleport/api/client.(*Client).GetKubeServices
	github.com/gravitational/teleport/lib/cache/collections.go:1909 github.com/gravitational/teleport/lib/cache.(*kubeService).fetch
	github.com/gravitational/teleport/lib/cache/cache.go:1268 github.com/gravitational/teleport/lib/cache.(*Cache).fetch.func2
	golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1
	runtime/asm_amd64.s:1594 runtime.goexit
User Message: failed to fetch resource: "kube_service"
	invalid label key: "project id"] cache/cache.go:831

Some thoughts:

• It is vastly preferable to fail early and refuse to start the service, rather than to fail user logins.
• Perhaps we should consider how to handle invalid label keys without such a hard failure.

[gclawes]

FYI, when this issue occurs in an agent, the teleport root cluster is unable to use the tctl kube ls command:

root@teleport-letsencrypt-us-east-1-79c99cdf87-d94gw:/# tctl kube ls
ERROR: invalid label key: "Application Id"