Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubernetes support #1986

Closed
klizhentas opened this issue Jun 3, 2018 · 8 comments
Closed

Kubernetes support #1986

klizhentas opened this issue Jun 3, 2018 · 8 comments
Assignees

Comments

@klizhentas
Copy link
Contributor

@klizhentas klizhentas commented Jun 3, 2018

Description

Teleport adds support for Kubernetes protocol.

Rationale

We have received many requests to add support for capturing kubectl exec sessions.
Users will benefit from the unified SSH and Kubernetes access via the same endpoint connected to the same identity provider.

Design

Teleport proxy adds additional protocol explicitly turned on the proxy side.

Teleport Proxy Sequence Diagram

  • User uses tsh login with proxy, nothing changes compared to usual SSH flow.
  • Client receives short lived x509 certs and SSH certs issued by Telepoort CA. User's kubernetes configuration is updated with x509 certs and credentials.
  • Kubernetes API request goes to teleport proxy, as set up by kubeconfig
  • Before processing the request, proxy needs a valid certificate recognized by Kubernetes cluster. Proxy delegates this task to the auth server and passes
    the list of kubernetes groups derived from user roles.
  • Auth server uses Kubernetes native CSR API to send and approve request. Auth server acts both as a requester and approver.
  • Proxy can now use the certificates issued by Kubernetes CA to terminate the request and proxy it to the auth server, capture the exec traffic

This model allows Teleport proxy to work with any standard Kubernetes clusters supporting out of the box CSR API.

Design considerations

Client credentials

Client X509 certificates are short lived and controlled by the same max_session_ttl parameter. Client certificates can not be used directly with kubernetes cluster, as they are issued by Teleport certificate authority. This has additional bonus as users will be forced to go through the Teleport proxy bastion server.

Kubernetes certs

Proxy caches and stores Kubernetes certificates issued for every user in memory. Whenever user kubernetes_groups changes, kubernetes certificates will be re-issued to reflect the change.

Session capture

Teleport's session capture behavior for kubectl exec and kubectl attach is similar to session capture

Non-Terminating Mode

Unlike SSH implementation, Teleport does not support non-terminating mode for Kubernetes in the initial implementation.

Trusted clusters

Trusted clusters are supported as usual, in this case user inherits the roles as specified on the role_map and kubernetes_groups of the assumed local cluster roles will be mapped to the kubernetes groups.

Configuration

RBAC

User's RBAC adds one additional property - kubernetes_groups. Just like with SSH principals, this property sets kubernetes groups added to the issued certificate. It's up to identity provider to send the list of groups for every user.

kind: role
version: v3
metadata:
  name: clusteradmin
spec:
  # SSH options used for user sessions 
  options:
    # max_session_ttl defines the TTL (time to live) of SSH certificates 
    # issued to the users with this role.
    max_session_ttl: 8h

    # forward_agent controls either users are allowed to use SSH agent forwarding
    forward_agent: true

    # port_forwarding
    port_forwarding: true

  # allow section declares a list of resource/verb combinations that are
  # allowed for the users of this role. by default nothing is allowed.
  allow:
    # logins array defines the OS logins a user is allowed to use.
    # A few special variables are supported here (see below)
    logins: [root, sasha, '{{internal.logins}}']

    # a list of kubernetes groups to assign, the groups can be sent by SAML/OIDC provider
    kubernetes_groups: ['system:masters', '{{external.traits}}]]

    # node labels that a user can connect to. The wildcard ('*') means "any node"
    node_labels:
      '*': '*'

    # see below.
    rules:
    - resources: ['*']
      verbs: ['*']

  # the deny section uses the identical format as the 'allow' section.
  # the deny rules always override allow rules.
  deny: {}

Service

Proxy service adds two properties kube_listen_addr that specifies listening endpoint and if specified, turns on kubernetes proxy.

kube_api_addr is optional, by default is set to kubernetes.default.svc.cluster.local:443 and is changed for cases when proxy is deployed outside of the kubernetes cluster.

auth_service:
    # optional value for cases when auth service
    # is deployed outside of kubernetes cluster, 
    # auth service needs to have access to kubernetes CA certificate
    # default value is `/var/run/secrets/kubernetes.io/serviceaccount/ca.crt`
    # (the value that is always present in kubernetes pods)
    kubernetes_ca_cert_file: /home/sasha/scripts/teleport/local/kubeca.cert
proxy_service:
  enabled: yes
  # Kubernetes section configures kubernetes protocol
  # support for proxy service.
  kubernetes:
    # by default the kubernetes protocol is disabled
    enabled: yes
    # public_addr can be a scalar or list,
    # in case if specified, the first value of
    # public_addr is used by tsh client
    # and is set in kubeconfig
    public_addr: [custom.example.com:port]
    # optional value for cases when kubernetes proxy is deployed
    # outside of the cluster, default value points to kubernetes.default.svc.cluster.local:443
    # that will point to the cluster
    api_addr: kuberentes.example.com:443
    # Default listen address value is 0.0.0.0:3026
    listen_addr: localhost:3026

Community Edition Support

By default all OSS users are not assigned any kubernetes groups, so even if Kubernetes
support is turned in the cluster, users do nt

Github connector

Github connector now supports extra field that allows users from different
teams to get assigned to multiple kubernetes groups:

kind: github
version: v3
metadata:
  # connector name that will be used with `tsh --auth=github login`
  name: github
spec:
  # client ID of Github OAuth app
  client_id: example-3b1bdfdf0a79277648ab
  # client secret of Github OAuth app
  client_secret: example-e916c8bebfc4c11fce5c2cd2134e548e6731a7a6
  # connector display name that will be shown on web UI login screen
  display: Github
  # callback URL that will be called after successful authentication
  redirect_url: https://127.0.0.1:3080/v1/webapi/github/callback
  # mapping of org/team memberships onto allowed logins and roles
  teams_to_logins:
    - organization: example.com # Github organization name
      team: admins # Github team name within that organization
      # allowed logins for users in this org/team
      logins:
        - root
      kubernetes_groups:
        - 'system:masters'

Local users support

Additional flag --k8s-groups can be specified when inviting the user:

tctl users add alice alice,root --k8s-groups='system:masters'

Deployment

Both Proxy and Auth servers can be deployed both inside Kubernetes cluster and outside of it.

  • Proxy service, when deployed outside of Kubernetes cluster has to provide address via api_addr. When it's deployed inside a cluster, no extra config is necessary.

  • Auth service, when deployed outside of K8s cluster, has to have access to public certificate of K8s certificate authority, placed in /var/run/secrets/kubernetes.io/serviceaccount/ca.crt or specified in auth service kubernetes_ca_cert_file parameter. and have access to Kubernetes configuration file with proper access credentials. The environment variable KUBECONFIG must point to valid kubeconfig file that auth server has access to when deployed outside of Kubernetes cluster.

When deployed inside K8s, no extra configuration is necessary. In both deployment modes Auth server has to have K8s RBAC to allow it to receive and approve CSR.

Domain based routing

In case of multiple remote clusters, teleport updates kubeconfig based on kubernetes section public_addr property, cluster name and subdomain.

For example, if kubernetes public_addr is example.com, the listen addr is specified to 3026 and there are two clusters, main and remote, here is how the requests will be routed:

https://example.com:3026 -> will get routed to main cluster
https://main.example.com:3026 -> will get routed to main cluster
https://remote.example.com:3026 -> will get routed to remote cluster.

Note that to support remote clusters cases X509 certificates for example.com should be wildcard.

@klizhentas klizhentas added this to the 2.7.0 "San Antonio" milestone Jun 3, 2018
@klizhentas

This comment has been minimized.

Copy link
Contributor Author

@klizhentas klizhentas commented Jun 3, 2018

cc @kontsevoy take a look

klizhentas added a commit that referenced this issue Jun 3, 2018
This issue updates #1986.

This is intial, experimental implementation that will
be updated with tests and edge cases prior to production 2.7.0 release.

Teleport proxy adds support for Kubernetes API protocol.
Auth server uses Kubernetes API to receive certificates
issued by Kubernetes CA.

Proxy intercepts and forwards API requests to the Kubernetes
API server and captures live session traffic, making
recordings available in the audit log.

Tsh login now updates kubeconfig configuration to use
Teleport as a proxy server.
klizhentas added a commit that referenced this issue Jun 3, 2018
This issue updates #1986.

This is intial, experimental implementation that will
be updated with tests and edge cases prior to production 2.7.0 release.

Teleport proxy adds support for Kubernetes API protocol.
Auth server uses Kubernetes API to receive certificates
issued by Kubernetes CA.

Proxy intercepts and forwards API requests to the Kubernetes
API server and captures live session traffic, making
recordings available in the audit log.

Tsh login now updates kubeconfig configuration to use
Teleport as a proxy server.
klizhentas added a commit that referenced this issue Jun 3, 2018
This issue updates #1986.

This is intial, experimental implementation that will
be updated with tests and edge cases prior to production 2.7.0 release.

Teleport proxy adds support for Kubernetes API protocol.
Auth server uses Kubernetes API to receive certificates
issued by Kubernetes CA.

Proxy intercepts and forwards API requests to the Kubernetes
API server and captures live session traffic, making
recordings available in the audit log.

Tsh login now updates kubeconfig configuration to use
Teleport as a proxy server.
klizhentas added a commit that referenced this issue Jun 3, 2018
This issue updates #1986.

This is intial, experimental implementation that will
be updated with tests and edge cases prior to production 2.7.0 release.

Teleport proxy adds support for Kubernetes API protocol.
Auth server uses Kubernetes API to receive certificates
issued by Kubernetes CA.

Proxy intercepts and forwards API requests to the Kubernetes
API server and captures live session traffic, making
recordings available in the audit log.

Tsh login now updates kubeconfig configuration to use
Teleport as a proxy server.
klizhentas added a commit that referenced this issue Jun 3, 2018
This issue updates #1986.

This is intial, experimental implementation that will
be updated with tests and edge cases prior to production 2.7.0 release.

Teleport proxy adds support for Kubernetes API protocol.
Auth server uses Kubernetes API to receive certificates
issued by Kubernetes CA.

Proxy intercepts and forwards API requests to the Kubernetes
API server and captures live session traffic, making
recordings available in the audit log.

Tsh login now updates kubeconfig configuration to use
Teleport as a proxy server.
klizhentas added a commit that referenced this issue Jun 3, 2018
This issue updates #1986.

This is intial, experimental implementation that will
be updated with tests and edge cases prior to production 2.7.0 release.

Teleport proxy adds support for Kubernetes API protocol.
Auth server uses Kubernetes API to receive certificates
issued by Kubernetes CA.

Proxy intercepts and forwards API requests to the Kubernetes
API server and captures live session traffic, making
recordings available in the audit log.

Tsh login now updates kubeconfig configuration to use
Teleport as a proxy server.
mumoshu added a commit to mumoshu/teleport that referenced this issue Jun 16, 2018
With instructions to test it locally using minikube + ngrok. I was just trying to run Teleport on Kubernetes for gravitational#1986 and thought a more idiomatic helm chart would be helpful as a foundation towards the 2.7.0 release :)
mumoshu added a commit to mumoshu/teleport that referenced this issue Jun 18, 2018
With instructions to test it locally using minikube + ngrok. I was just trying to run Teleport on Kubernetes for gravitational#1986 and thought a more idiomatic helm chart would be helpful as a foundation towards the 2.7.0 release :)
@mumoshu

This comment has been minimized.

Copy link
Contributor

@mumoshu mumoshu commented Jun 28, 2018

@klizhentas Hey!

Non-Terminating Mode

Unlike SSH implementation, Teleport does not support non-terminating mode for Kubernetes in the initial implementation.

What do you mean by non-terminating mode? I'm unable to figure that even after reading throughout the teleport docs.

@klizhentas

This comment has been minimized.

Copy link
Contributor Author

@klizhentas klizhentas commented Aug 4, 2018

@kontsevoy I have updated this ticket with latest configuration changes.

@kontsevoy

This comment has been minimized.

Copy link
Contributor

@kontsevoy kontsevoy commented Sep 20, 2018

@alexwolfe see the diagram above. Can we have a nicer looking version of it which would fit into the restricted width of our Teleport docs?

@kontsevoy

This comment has been minimized.

Copy link
Contributor

@kontsevoy kontsevoy commented Sep 25, 2018

Proposed configuration

auth_service:
    # optional value for cases when the auth service is deployed outside a k8s pod:
    kubeconfig_file: /path/to/kubeconfig
proxy_service:
  enabled: yes
  # Kubernetes section configures kubernetes protocol
  # support for proxy service.
  kubernetes:
    # by default the kubernetes protocol is disabled
    enabled: yes
    # public_addr can be a scalar or list,
    # in case if specified, the first value of
    # public_addr is used by tsh client
    # and is set in kubeconfig
    public_addr: [custom.example.com:port]
    # Default listen address value is 0.0.0.0:3026
    listen_addr: localhost:3026

Changes

  • kubeconfig_file is added to the "auth_service" section.
  • kubernetes_ca_cert_file is removed from the "auth_service" section. it becomes redundant because of kubeconfig_path.
  • api_addr is removed from the "proxy_service" section. The proxy will fetch this information from the auth server.
@kontsevoy kontsevoy assigned klizhentas and unassigned alexwolfe Sep 25, 2018
@kontsevoy

This comment has been minimized.

Copy link
Contributor

@kontsevoy kontsevoy commented Sep 25, 2018

@klizhentas see the comment above.

@alexwolfe

This comment has been minimized.

Copy link
Contributor

@alexwolfe alexwolfe commented Sep 25, 2018

@kontsevoy how's this one?
sequence-diagram

@kontsevoy

This comment has been minimized.

Copy link
Contributor

@kontsevoy kontsevoy commented Sep 25, 2018

The docs are done. @klizhentas you can close it when the config improvements are in.

@kontsevoy kontsevoy removed their assignment Sep 25, 2018
klizhentas added a commit that referenced this issue Sep 26, 2018
Fixes #1986

When deployed outside of the kubernetes cluster,
teleport now reads all configuration from kubernetes
config file, supplied via parameter.

Auth server then passes information about
target api server back to the proxy.
klizhentas added a commit that referenced this issue Sep 26, 2018
Fixes #1986

When deployed outside of the kubernetes cluster,
teleport now reads all configuration from kubernetes
config file, supplied via parameter.

Auth server then passes information about
target api server back to the proxy.
klizhentas added a commit that referenced this issue Sep 26, 2018
Fixes #1986

When deployed outside of the kubernetes cluster,
teleport now reads all configuration from kubernetes
config file, supplied via parameter.

Auth server then passes information about
target api server back to the proxy.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.