Add a concierge backend which uses service account tokens and an enhanced kubectl

[mattmoyer]

Is your feature request related to a problem? Please describe.
I would like to use Pinniped on my managed Kubernetes cluster (e.g., cloud provider) where the existing Concierge TokenCredentialRequest API is inoperable. I would also like to do this without needing to provision additional load balancer or ingress resources on my cluster.

Describe the solution you'd like
We could make this work with a custom kubectl binary which adds one small enhancement to the ExecCredential API: the ability to set additional headers.

The flow would look something like:

1. User installs custom build of kubectl or maybe some future upstream version which includes this enhancement.
2. User downloads the Pinniped kubeconfig for their cluster where the concierge is running.
3. User runs kubectl [...], which invokes the pinniped login [...] ExecCredential plugin.
4. The pinniped login command logs in the user and then sends a TokenCredentialRequest to the concierge.
5. The concierge authenticates the user's token and creates several cluster objects:
   1. A ServiceAccount named $user-service-account in the concierge namespace.
   2. A ClusterRole which grants the impersonate verb for the user's authenticated username and groups.
   3. A ClusterRoleBinding which binds the ClusterRole to the ServiceAccount.
6. The concierge uses the TokenRequest API to request a bound service account token for the newly-created ServiceAccount, which it returns in the status of the TokenCredentialRequest along with the authenticated username and groups.
7. The pinniped login command returns an ExecCredential status containing the token and extra Impersonate-User and Impersonate-Group headers.
8. When the custom kubectl receives the ExecCredential status, it makes requests using the ServiceAccount token but impersonating the user.
9. The Kubernetes audit log contains the ephemeral ServiceAccount and the impersonated user.

Some other notes:

• The objects in step 5 can also be cached and re-used across multiple logins.
• The kubectl enhancement is something we might be able to ship upstream. It would add a new requestHeaders field in the ExecCredential status.
• Our ExecCredential plugin could try to detect when it's running with an unsupported version of kubectl. In these cases we can prompt the user to install our custom version or ask them to add --as flags (or the equivalent YAML) to make things mostly work even with current 1.20 upstream kubectl.

Describe alternatives you've considered

Impersonation proxy via LoadBalancer service

This design is an alternative to the impersonation proxy design from #67/#339, which does not require changes to kubectl but does require additional ingress/load balancer resources.

Impersonation proxy via federated API

We attempted to work around this by serving the impersonation proxy via a federated API by creating an APIService. However, this does not work because when our ExecCredential plugin returns a token, it can only be sent in the Authorization header which is rejected by the federation API server and not passed through to the concierge.

Current certificate-based TokenCredentialRequest API

All of these designs are an alternative backend for the current TokenCredentialRequest API which mints short-lived certificates for users. This is only possible on clusters that allow TLS authentication and support some method of signing a short-lived certificate.

Are you considering submitting a PR for this feature?
Yes, but we need to make a team decision about which direction we want to go.

• How will this project improvement be tested?
  This would need extensive integration tests as well as acceptance environments in all major cloud providers.

• How does this change the current architecture?
  The current TokenCredentialRequest behavior would still be supported, at least initially.

• How will this change be backwards compatible?
  Yes.

• How will this feature be documented?
  We should consider writing some new documentation describing how to use Pinniped in managed cluster environments.

Additional context

N/A

[mattmoyer]

Other ideas:

• We could try to re-write the kubeconfig YAML to include the as/as-groups` fields. It wouldn't be super smooth because the first kubectl command would fail, but it would work without a custom kubectl.
• If you know the username/groups ahead of time, you can generate the kubeconfig with those as/as-groups fields in advance. This means your kubeconfig becomes outdated if your groups change.

Other tradeoffs/drawbacks in this proposal:

• Couldn't use alternative CLI clients, including oc or tools like stern that use client-go but aren't kubectl itself.
• There's a risk that our change would never be accepted upstream into client-go/kubectl.

[mattmoyer]

We ended up deciding not to do this for now, and to continue pursuing the impersonation proxy as a solution on managed cluster environments. We could still consider doing this in the future, especially if the upstream ExecCredential API adds custom header support.