Skip to content

Commit

Permalink
Use SPIRE to manage cert for CRD k8s-workload-registrar webhook (#2321)
Browse files Browse the repository at this point in the history 
Signed-off-by: Faisal Memon <f.memon@f5.com>
  • Loading branch information
@faisal-memon @evan2645
faisal-memon authored and evan2645 committed Sep 2, 2021
1 parent ef839d0 commit da3dbde
Show file tree
Hide file tree
Showing 18 changed files with 1,495 additions and 300 deletions.
131 changes: 7 additions & 124 deletions support/k8s/k8s-workload-registrar/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,21 +44,6 @@ The following configuration directives are specific to `"webhook"` mode:
| `cacert_path` | string | required | Path on disk to the CA certificate used to verify the client (i.e. API server) | `"cacert.pem"` |
| `insecure_skip_client_verification` | boolean | required | If true, skips client certificate verification (in which case `cacert_path` is ignored). See [Security Considerations](#security-considerations) for more details. | `false` |

The following configuration directives are specific to `"crd"` mode:

| Key | Type | Required? | Description | Default |
| -------------------------- | --------| ---------| ----------------------------------------- | ------- |
| `add_svc_dns_name` | bool | optional | Enable adding service names as SAN DNS names to endpoint pods | `true` |
| `leader_election` | bool | optional | Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager. | `false` |
| `metrics_bind_addr` | string | optional | The address the metric endpoint binds to. The special value of "0" disables metrics. | `":8080"` |
| `pod_controller` | bool | optional | Enable auto generation of SVIDs for new pods that are created | `true` |
| `webhook_enabled` | bool | optional | Enable a validating webhook to ensure CRDs are properly formatted and there are no duplicates. Only needed if manually creating entries | `false` |
| `webhook_cert_dir` | string | optional | Directory for certificates when enabling validating webhook. The certificate and key must be named tls.crt and tls.key. | `"/run/spire/serving-certs"` |
| `webhook_port` | int | optional | The port to use for the validating webhook. | `9443` |
| `identity_template` | string | optional | The template for custom [Identity Template Based Workload Registration](#identity-template-based-workload-registration) | `ns/{{.Pod.Namespace}}/sa/{{.Pod.ServiceAccount}}` |
| `identity_template_label` | string | optional | Pod label for selecting pods that get SVIDs whose SPIFFE IDs are defined by `identity_template` format. If not set, applies to all the pods when `identity_template` is set | |
| `context` | map[string]string | optional | The map of key/value pairs of arbitrary string parameters to be used by `identity_template` | |

The following configuration directives are specific to `"reconcile"` mode:

| Key | Type | Required? | Description | Default |
Expand All @@ -69,6 +54,8 @@ The following configuration directives are specific to `"reconcile"` mode:
| `add_pod_dns_names` | bool | optional | Enable/disable adding k8s DNS names to pod SVIDs. | false |
| `cluster_dns_zone` | string | optional | The DNS zone used for services in the k8s cluster. | `"cluster.local"` |

For CRD configuration directives see [CRD Mode Configuration](mode-crd/README.md#configuration)

### Example

```
Expand Down Expand Up @@ -179,58 +166,7 @@ Pods that don't contain the pod annotation are ignored.

### Identity Template Based Workload Registration

Identity template based workload registration provides a way to customize the format of SPIFFE IDs. The identity format is scoped to a cluster.
The template formatter is using Golang
[text/template](https://pkg.go.dev/text/template) conventions,
and it can reference arbitrary values provided in the `context` map of strings
in addition to the following Pod-specific arguments:
* Pod.Name
* Pod.UID
* Pod.Namespace
* Pod.ServiceAccount
* Pod.Hostname
* Pod.NodeName

For example if the registrar was configured with the following:
```
identity_template = "region/{{.Context.Region}}/cluster/{{.Context.ClusterName}}/sa/{{.Pod.ServiceAccount}}/pod_name/{{.Pod.pod_name}}"
context {
Region = "US-NORTH"
ClusterName = "MYCLUSTER"
}
```
and the _example-workload_ pod was deployed in _production_ namespace and _myserviceacct_ service account, the following registration entry would be created:
```
Entry ID : 200d8b19-8334-443d-9494-f65d0ad64eb5
SPIFFE ID : spiffe://example.org/region/US-NORTH/cluster/MYCLUSTER/sa/myserviceacct/pod_name/example-workload
Parent ID : ...
TTL : default
Selector : k8s:ns:production
Selector : k8s:pod-name:example-workload-98b6b79fd-jnv5m
```

If `identity_template_label` is defined in the registrar configuration:

```
identity_template_label = "enable_identity_template"
```

only pods with the same label set to `true` would get identity SVID.

```yaml
apiVersion: v1
kind: Pod
metadata:
labels:
enable_identity_template: true
spec:
containers:
...
```
Pods that don't contain the pod label are ignored.

If `identity_template_label` is empty or omitted, all the pods will receive the identity.

This is specific to the `crd` mode. See [Identity Template Based Workload Registration](mode-crd/README.md#identity-template-based-workload-registration) in the `crd` mode documentation.

## Deployment

Expand All @@ -243,32 +179,13 @@ shared volume containing the socket file.


### Reconcile Mode Configuration

To use reconcile mode you need to create appropriate roles and bind them to the ServiceAccount you intend to run the controller as.
An example can be found in `mode-reconcile/config/role.yaml`, which you would apply with `kubectl apply -f mode-reconcile/config/role.yaml`

### CRD Mode Configuration

The following configuration is required before `"crd"` mode can be used:

1. The SpiffeId CRD needs to be applied: `kubectl apply -f mode-crd/config/spiffeid.spiffe.io_spiffeids.yaml`
* The SpiffeId CRD is namespace scoped
1. The appropriate ClusterRole need to be applied. `kubectl apply -f mode-crd/config/crd_role.yaml`
* This creates a new ClusterRole named `spiffe-crd-role`
1. The new ClusterRole needs a ClusterRoleBinding to the SPIRE Server ServiceAccount. Change the name of the ServiceAccount and then: `kubectl apply -f mode-crd/config/crd_role_binding.yaml`
* This creates a new ClusterRoleBinding named `spiffe-crd-rolebinding`
1. If you would like to manually create SpiffeId custom resources, then a validating webhook is needed to prevent misconfigurations and improve security: `kubectl apply -f mode-crd/config/webhook.yaml`
* This creates a new ValidatingWebhookConfiguration and Service, both named `k8s-workload-registrar`
* Make sure to add your CA Bundle to the ValidatingWebhookConfiguration where it says `<INSERT BASE64 CA BUNDLE HERE>`
* Additionally a Secret that volume mounts the certificate and key to use for the webhook. See `webhook_cert_dir` configuration option above.
1. The CRD mode allows custom format of the SVID via `identity_template` with Pod specific values, and provided `context` map of string arguments. See [Identity Template Based Workload Registration](#identity-template-based-workload-registration)

#### CRD mode Security Considerations
It is imperative to only grant trusted users access to manually create SpiffeId custom resources. Users with access have the ability to issue any SpiffeId
to any pod in the namespace.

If allowing users to manually create SpiffeId custom resources it is important to use the Validating Webhook. The Validating Webhook ensures that
registration entries created have a namespace selector that matches the namespace the resource was created in. This ensures that the manually created
entries can only be consumed by workloads within that namespace.
See [Quick Start for CRD Kubernetes Workload Registrar](mode-crd/README.md#quick-start)

### Webhook Mode Configuration
The registrar will need access to its server keypair and the CA certificate it uses to verify clients.
Expand Down Expand Up @@ -300,12 +217,14 @@ the risks.


#### Migrating away from the webhook

The k8s ValidatingWebhookConfiguration will need to be removed or pods may fail admission. If you used the default
configuration this can be done with:

`kubectl validatingwebhookconfiguration delete k8s-workload-registrar-webhook`

## DNS names

Both `"reconcile"` and `"crd"` mode provide the ability to add DNS names to registration entries for pods. They
currently have different ideas about what names should be added, with `"reconcile"` adding every possible name that can
be used to access a pod (via a service or directly), and `"crd"` mode limiting itself to `<service>.<namespace>.svc`.
Expand Down Expand Up @@ -339,39 +258,3 @@ less configuration.
registrar, but may also be manually created to allow creation of arbitrary Spire Entries. If you intend to manage
SpiffeID custom resources directly then it is strongly encouraged to run the controller with the `"crd"` mode's webhook
enabled.

## SPIFFE ID Custom Resources
A sample SPIFFE ID custom resource for `"crd"` mode is below:

```
apiVersion: spiffeid.spiffe.io/v1beta1
kind: SpiffeID
metadata:
name: my-spiffe-id
namespace: my-namespace
spec:
dnsNames:
- my-dns-name
federatesWith:
- example-third-party.org
selector:
namespace: my-namespace
podName: my-pod-name
spiffeId: spiffe://example.org/my-spiffe-id
parentId: spiffe://example.org/spire/server
```

The supported selectors are:
- arbitrary -- Arbitrary selectors
- containerName -- Name of the container
- containerImage -- Container image used
- namespace -- Namespace to match for this SPIFFE ID
- nodeName -- Node name to match for this SPIFFE ID
- podLabel -- Pod label name/value to match for this SPIFFE ID
- podName -- Pod name to match for this SPIFFE ID
- podUID -- Pod UID to match for this SPIFFE ID
- serviceAccount -- ServiceAccount to match for this SPIFFE ID

Note: Specifying DNS Names or Federation Domains is optional.

Spire enforces that spiffeId+parentId+selectors are unique. The optional `"crd"` mode webhook
93 changes: 76 additions & 17 deletions support/k8s/k8s-workload-registrar/config_crd.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,30 @@ package main

import (
"context"
"errors"
"fmt"
"io/ioutil"
"os"
"strings"

"github.com/hashicorp/hcl"
svidv1 "github.com/spiffe/spire-api-sdk/proto/spire/api/server/svid/v1"
spiffeidv1beta1 "github.com/spiffe/spire/support/k8s/k8s-workload-registrar/mode-crd/api/spiffeid/v1beta1"
"github.com/spiffe/spire/support/k8s/k8s-workload-registrar/mode-crd/controllers"
"github.com/spiffe/spire/support/k8s/k8s-workload-registrar/mode-crd/webhook"
"github.com/zeebo/errs"

"golang.org/x/sys/unix"
ctrl "sigs.k8s.io/controller-runtime"
)

const (
defaultAddSvcDNSName = true
defaultPodController = true
defaultMetricsBindAddr = ":8080"
defaultWebhookCertDir = "/run/spire/serving-certs"
defaultWebhookPort = 9443
namespaceFile = "/var/run/secrets/kubernetes.io/serviceaccount/namespace"
defaultAddSvcDNSName = true
defaultPodController = true
defaultMetricsBindAddr = ":8080"
defaultWebhookCertDir = "/run/spire/serving-certs"
defaultWebhookPort = 9443
defaultWebhookServiceName = "k8s-workload-registrar"
namespaceFile = "/var/run/secrets/kubernetes.io/serviceaccount/namespace"
)

type CRDMode struct {
Expand All @@ -28,9 +34,10 @@ type CRDMode struct {
LeaderElection bool `hcl:"leader_election"`
MetricsBindAddr string `hcl:"metrics_bind_addr"`
PodController bool `hcl:"pod_controller"`
WebhookEnabled bool `hcl:"webhook_enabled"`
WebhookCertDir string `hcl:"webhook_cert_dir"`
WebhookEnabled bool `hcl:"webhook_enabled"`
WebhookPort int `hcl:"webhook_port"`
WebhookServiceName string `hcl:"webhook_service_name"`
IdentityTemplate string `hcl:"identity_template"`
IdentityTemplateLabel string `hcl:"identity_template_label"`
Context map[string]string `hcl:"context"`
Expand All @@ -55,6 +62,10 @@ func (c *CRDMode) ParseConfig(hclConfig string) error {
c.WebhookPort = defaultWebhookPort
}

if c.WebhookServiceName == "" {
c.WebhookServiceName = defaultWebhookServiceName
}

if c.IdentityTemplate != "" && (c.PodAnnotation != "" || c.PodLabel != "") {
return errs.New("workload registration configuration is incorrect, can only use one of identity_template, pod_annotation, or pod_label")
}
Expand All @@ -63,6 +74,7 @@ func (c *CRDMode) ParseConfig(hclConfig string) error {
if c.Context == nil && c.IdentityTemplate != "" && strings.Contains(strings.ReplaceAll(c.IdentityTemplate, " ", ""), "{{.Context.") {
return errs.New("identity_template references non-existing context")
}

return nil
}

Expand All @@ -77,13 +89,14 @@ func (c *CRDMode) Run(ctx context.Context) error {
if err != nil {
return errs.New("failed to dial server: %v", err)
}
svidClient := svidv1.NewSVIDClient(c.serverAPI.serverConn)

mgr, err := controllers.NewManager(c.LeaderElection, c.MetricsBindAddr, c.WebhookCertDir, c.WebhookPort)
if err != nil {
return err
}

myNamespace, err := getNamespace()
myPodNamespace, err := getMyPodNamespace()
if err != nil {
return err
}
Expand All @@ -102,11 +115,41 @@ func (c *CRDMode) Run(ctx context.Context) error {
}

if c.WebhookEnabled {
// Backwards compatibility check
exists, err := c.certDirExistsAndReadOnly()
if err != nil {
return fmt.Errorf("checking webhook certificate directory permissions: %w", err)
}
if exists {
log.Warn("Detected statically mounted webhook certificate directory, support for this will be removed in a future version. " +
"Refer to README for instructions on using SPIRE Server to populate webhook certificates.")
} else {
webhookSVID, err := webhook.NewSVID(ctx, webhook.SVIDConfig{
Cluster: c.Cluster,
S: svidClient,
Log: log,
Namespace: myPodNamespace,
TrustDomain: c.TrustDomain,
WebhookCertDir: c.WebhookCertDir,
WebhookServiceName: c.WebhookServiceName,
})
if err != nil {
return err
}
if err = webhookSVID.MintSVID(ctx, nil); err != nil {
return err
}
go func() {
if err := webhookSVID.SVIDRotator(ctx); err != nil {
log.Fatalf("failed rotating webhook certificate: %v", err)
}
}()
}
err = spiffeidv1beta1.AddSpiffeIDWebhook(spiffeidv1beta1.SpiffeIDWebhookConfig{
Ctx: ctx,
Log: log,
Mgr: mgr,
Namespace: myNamespace,
Namespace: myPodNamespace,
E: entryClient,
TrustDomain: c.TrustDomain,
})
Expand All @@ -121,7 +164,7 @@ func (c *CRDMode) Run(ctx context.Context) error {
Cluster: c.Cluster,
Ctx: ctx,
Log: log,
Namespace: myNamespace,
Namespace: myPodNamespace,
Scheme: mgr.GetScheme(),
TrustDomain: c.TrustDomain,
}).SetupWithManager(mgr)
Expand Down Expand Up @@ -152,7 +195,7 @@ func (c *CRDMode) Run(ctx context.Context) error {
}

if c.AddSvcDNSName {
err := controllers.NewEndpointReconciler(controllers.EndpointReconcilerConfig{
err = controllers.NewEndpointReconciler(controllers.EndpointReconcilerConfig{
Client: mgr.GetClient(),
Ctx: ctx,
DisabledNamespaces: c.DisabledNamespaces,
Expand All @@ -168,11 +211,27 @@ func (c *CRDMode) Run(ctx context.Context) error {
return mgr.Start(ctrl.SetupSignalHandler())
}

func getNamespace() (string, error) {
content, err := os.ReadFile(namespaceFile)
if err != nil {
return "", err
func (c *CRDMode) certDirExistsAndReadOnly() (bool, error) {
err := unix.Access(c.WebhookCertDir, unix.W_OK)
switch {
case err == nil, errors.Is(err, unix.ENOENT):
return false, nil
case errors.Is(err, unix.EROFS):
return true, nil
default:
return false, err
}
}

func getMyPodNamespace() (string, error) {
namespace, ok := os.LookupEnv("MY_POD_NAMESPACE")
if !ok {
content, err := ioutil.ReadFile(namespaceFile)
if err != nil {
return "", fmt.Errorf("unable to get MY_POD_NAMESPACE; ensure downward API is configured for this pod: %w", err)
}
return string(content), nil
}

return string(content), nil
return namespace, nil
}
Loading

0 comments on commit da3dbde

Please sign in to comment.