Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Towards support for federated K8s OIDC #212

Closed
mattmoor opened this issue Oct 24, 2021 · 6 comments
Closed

Towards support for federated K8s OIDC #212

mattmoor opened this issue Oct 24, 2021 · 6 comments
Labels
enhancement New feature or request

Comments

@mattmoor
Copy link
Member

Description

tl;dr We should allow Fulcio to accept OIDC tokens from some (see below) federated Kubernetes OIDC issuers.

Background and High-level proposal

Kubernetes has a relatively new feature called "Service Account Projected Volumes", which allows a workload to generate short-lived, audience-scoped OIDC tokens for a given serviceaccount, e.g.

{
  "header": {
    "alg": "RS256",
    "kid": "oDK51cHaX3V8crffIaSQG8Sx1Vys3xHLD6moTkivqaM"
  },
  "payload": {
    "aud": [
      "something"
    ],
    "exp": 1634744688,
    "iat": 1634744088,
    "iss": "https://container.googleapis.com/v1/projects/mattmoor-credit/locations/us-west1-b/clusters/tenant-cluster",
    "kubernetes.io": {
      "namespace": "default",
      "pod": {
        "name": "oidc-test",
        "uid": "49ad3572-b3dd-43a6-8d77-5858d3660275"
      },
      "serviceaccount": {
        "name": "default",
        "uid": "f5720c1d-e152-4356-a897-11b07aff165d"
      }
    },
    "nbf": 1634744088,
    "sub": "system:serviceaccount:default:default"
  },
  "signature": "..."
}

At least GKE and EKS expose public issuer endpoints for these, which enable these tokens to be used with OIDC federation to authenticate with off-cluster services.

Corollary: if we support this, then we can actually start to leverage KinD to run e2e tests for keyless flows

Broken assumptions and work needed

In general, the subject (sub) of an OIDC flow is only unique within the scoping of the issuer (iss), but today Fulcio assumes a handful of fairly central OIDC issuers with disjoint subjects. This assumption manifests in the subject being the only piece of information included in the issued certificate. If we don't address this assumption, then we can't meaningfully add support for tokens issued by clusters because any cluster could sign things as system:serviceaccount:default:default!

AI: So the first order of business is to make the implicit issuer URLs we have today explicit. I'd propose we start to unconditionally include the iss as the first URI in the URIs section of the certificate we issue.

AI: Once it is safe to start including Kubernetes OIDC in the issuer configuration, I propose that we add support for Kubernetes OIDC issuers (not configured in the public instance!), and use it to stand up some basic KinD e2e testing for Fulcio following: https://github.com/mattmoor/kind-oidc. Once this lands, we can also use this same setup to start doing presubmit testing for keyless flows downstream in cosign (cc @dekkagaijin).

AI: Once we have support for directly configured Kubernetes OIDC issuers, I propose that we refactor the current issuer configuration to support a degree of fuzzy matching, and start adding well-known public prefixes like GKE and EKS.

AI: Once we have support for Kubernetes OIDC issuers, I propose we augment the cosign OIDC "magic" to support a particular mount path (e.g. /var/run/sigstore/cosign/oidc-token)

I believe that with the sum total of these features, we should be able to support fairly broad K8s "workload identity" based flows (both public Fulcio w/ public cloud vendor, and private Fulcios on-prem), and start supporting "keyless" presubmit testing (upstream and downstream in tools 🤩).

cc @dlorenc @lukehinds

@mattmoor mattmoor added the enhancement New feature or request label Oct 24, 2021
mattmoor added a commit to mattmoor/fulcio that referenced this issue Oct 24, 2021
This change incorporates the OIDC token issuer into the certificates issued by Fulcio.

There is a much more detailed write-up in the linked issue laying out the motivations, justifications, and long-term goals of this change.

Related: sigstore#212

Signed-off-by: Matt Moore <mattomata@gmail.com>
@dlorenc
Copy link
Member

dlorenc commented Oct 24, 2021

AI: So the first order of business is to make the implicit issuer URLs we have today explicit. I'd propose we start to unconditionally include the iss as the first URI in the URIs section of the certificate we issue.

@bobcallaway has been working on this sort of - see #204 We use dex in front of Fulcio in prod, so the issuer is actually always the same dex endpoint.

@mattmoor
Copy link
Member Author

@dlorenc I assume you mean for 3LOs? #211 is my PR 😁

@dlorenc
Copy link
Member

dlorenc commented Oct 24, 2021

Yeah sorry - switched it to #204

mattmoor added a commit to mattmoor/fulcio that referenced this issue Oct 25, 2021
This change incorporates the OIDC token issuer into the certificates issued by Fulcio.

There is a much more detailed write-up in the linked issue laying out the motivations, justifications, and long-term goals of this change.

Related: sigstore#212

Signed-off-by: Matt Moore <mattomata@gmail.com>
@mattmoor
Copy link
Member Author

I believe with #204 in and deployed, this work should be unblocked!

@dekkagaijin
Copy link
Member

dekkagaijin commented Oct 26, 2021

/lgtm

Talked about it offline, and it's not a blocker, but we need to figure out good UX for disambiguating and/or explicitly trusting OIDC providers during verification to prevent evil providers from minting id_tokens for valid subjects (as mentioned). Including the provider in the cert should make it fairly easy to do this with e.g. --expected-oidc-provider

mattmoor added a commit to mattmoor/fulcio that referenced this issue Oct 27, 2021
There are several parts to this change:
1. Implement a new `ephemeralca` that just generates an in-memory certificate,
1. Rename `pkg/ca/pkcs11ca` to `pkg/ca/x509ca` since it had nothing `PKCS11` specific (shared with `ephemeralca` logic),
1. Add support for Kubernetes OIDC via Service Account Projected Volumes,
1. Have the KinD e2e test use `ephemeralca` and `cosign sign` an image.

I can split some of these pieces apart, but wanted to get this all working end-to-end, since a key goal was enabling e2e testing on KinD.

This follows a lot of the ideas from: https://github.com/mattmoor/kind-oidc

Related: sigstore#212
Fixes: sigstore#194

Signed-off-by: Matt Moore <mattomata@gmail.com>
mattmoor added a commit to mattmoor/cosign that referenced this issue Oct 27, 2021
This change adds an ambient OIDC provider that will enable when the following
filesystem path is populated: `/var/run/sigstore/cosign/oidc-token`.  The intended
use of this (primarily) is to enable consuming Kubernetes OIDC tokens produced
through Service Account Projected Volumes.

To consume this you would add the following to your Kubernetes pod spec:
```yaml
      containers:
      - name: my-container-name
        image: ...
        volumeMounts:
        - name: oidc-info
          mountPath: /var/run/sigstore/cosign

      volumes:
        - name: oidc-info
          projected:
            sources:
              - serviceAccountToken:
                  path: oidc-token
                  expirationSeconds: 600 # Use as short-lived as possible.
                  audience: sigstore
```

This would also work with Tekton step definitions, or other things that permit
the use of projected volumes.

Note: Fulcio doesn't currently allow any Kubernetes provider OIDC tokens on the
public instance, but one of the things I plan to look at next is supporting the
endpoints from GKE and EKS (both of which have public discovery endpoints).

Related: sigstore/fulcio#219
Related: sigstore/fulcio#212

Signed-off-by: Matt Moore <mattomata@gmail.com>
mattmoor added a commit to mattmoor/cosign that referenced this issue Oct 27, 2021
This change adds an ambient OIDC provider that will enable when the following
filesystem path is populated: `/var/run/sigstore/cosign/oidc-token`.  The intended
use of this (primarily) is to enable consuming Kubernetes OIDC tokens produced
through Service Account Projected Volumes.

To consume this you would add the following to your Kubernetes pod spec:
```yaml
      containers:
      - name: my-container-name
        image: ...
        volumeMounts:
        - name: oidc-info
          mountPath: /var/run/sigstore/cosign

      volumes:
        - name: oidc-info
          projected:
            sources:
              - serviceAccountToken:
                  path: oidc-token
                  expirationSeconds: 600 # Use as short-lived as possible.
                  audience: sigstore
```

This would also work with Tekton step definitions, or other things that permit
the use of projected volumes.

Note: Fulcio doesn't currently allow any Kubernetes provider OIDC tokens on the
public instance, but one of the things I plan to look at next is supporting the
endpoints from GKE and EKS (both of which have public discovery endpoints).

Related: sigstore/fulcio#219
Related: sigstore/fulcio#212

Signed-off-by: Matt Moore <mattomata@gmail.com>
dlorenc pushed a commit to sigstore/cosign that referenced this issue Oct 28, 2021
This change adds an ambient OIDC provider that will enable when the following
filesystem path is populated: `/var/run/sigstore/cosign/oidc-token`.  The intended
use of this (primarily) is to enable consuming Kubernetes OIDC tokens produced
through Service Account Projected Volumes.

To consume this you would add the following to your Kubernetes pod spec:
```yaml
      containers:
      - name: my-container-name
        image: ...
        volumeMounts:
        - name: oidc-info
          mountPath: /var/run/sigstore/cosign

      volumes:
        - name: oidc-info
          projected:
            sources:
              - serviceAccountToken:
                  path: oidc-token
                  expirationSeconds: 600 # Use as short-lived as possible.
                  audience: sigstore
```

This would also work with Tekton step definitions, or other things that permit
the use of projected volumes.

Note: Fulcio doesn't currently allow any Kubernetes provider OIDC tokens on the
public instance, but one of the things I plan to look at next is supporting the
endpoints from GKE and EKS (both of which have public discovery endpoints).

Related: sigstore/fulcio#219
Related: sigstore/fulcio#212

Signed-off-by: Matt Moore <mattomata@gmail.com>
dlorenc pushed a commit that referenced this issue Oct 28, 2021
There are several parts to this change:
1. Implement a new `ephemeralca` that just generates an in-memory certificate,
1. Rename `pkg/ca/pkcs11ca` to `pkg/ca/x509ca` since it had nothing `PKCS11` specific (shared with `ephemeralca` logic),
1. Add support for Kubernetes OIDC via Service Account Projected Volumes,
1. Have the KinD e2e test use `ephemeralca` and `cosign sign` an image.

I can split some of these pieces apart, but wanted to get this all working end-to-end, since a key goal was enabling e2e testing on KinD.

This follows a lot of the ideas from: https://github.com/mattmoor/kind-oidc

Related: #212
Fixes: #194

Signed-off-by: Matt Moore <mattomata@gmail.com>
mattmoor added a commit to mattmoor/fulcio that referenced this issue Oct 28, 2021
This change starts to refactor the way we interact with theh Fulcio `Config`
to support a forthcoming change to support what I've taken to calling "meta URLs".

I want to allow the public Fulcio instance to support the following *classes*
of Issuer URL, which are exposed for EKS and GKE clusters:
 * https://oidc.eks.us-west-2.amazonaws.com/id/B02C93B6A2D30341AD01E1B6D48164CB
 * https://container.googleapis.com/v1/projects/mattmoor-credit/locations/us-west1-b/clusters/tenant-cluster

To this end, I want to support adding to the Fulcio configuration "meta URLs"
of the form (still a strawman):
```
https://oidc.eks.*.amazonaws.com/id/*
https://container.googleapis.com/v1/projects/*/locations/*/clusters/*
```

However, in this change, I'm still just consolidating how we access the OIDC
issuer information in `FulcioConfig` so that lookups are performed via a new
`GetIssuer` method, and verifiers are instantiated via `GetVerifier()`.

The only remaining place I see iterating over `cfg.OIDCIssuers` is for the
`realm` in the `WWW-Authenticate` header of `401` responses (seems right to
not include the meta URLs here).  There are also still a handful of places
handling OIDCIssuers directly, but most are in `config.go`, or associated
validation logic (e.g. `federation/main.go`).

Related: sigstore#212

Signed-off-by: Matt Moore <mattomata@gmail.com>
mattmoor added a commit to mattmoor/fulcio that referenced this issue Oct 28, 2021
This change starts to refactor the way we interact with theh Fulcio `Config`
to support a forthcoming change to support what I've taken to calling "meta URLs".

I want to allow the public Fulcio instance to support the following *classes*
of Issuer URL, which are exposed for EKS and GKE clusters:
 * https://oidc.eks.us-west-2.amazonaws.com/id/B02C93B6A2D30341AD01E1B6D48164CB
 * https://container.googleapis.com/v1/projects/mattmoor-credit/locations/us-west1-b/clusters/tenant-cluster

To this end, I want to support adding to the Fulcio configuration "meta URLs"
of the form (still a strawman):
```
https://oidc.eks.*.amazonaws.com/id/*
https://container.googleapis.com/v1/projects/*/locations/*/clusters/*
```

However, in this change, I'm still just consolidating how we access the OIDC
issuer information in `FulcioConfig` so that lookups are performed via a new
`GetIssuer` method, and verifiers are instantiated via `GetVerifier()`.

The only remaining place I see iterating over `cfg.OIDCIssuers` is for the
`realm` in the `WWW-Authenticate` header of `401` responses (seems right to
not include the meta URLs here).  There are also still a handful of places
handling OIDCIssuers directly, but most are in `config.go`, or associated
validation logic (e.g. `federation/main.go`).

Related: sigstore#212

Signed-off-by: Matt Moore <mattomata@gmail.com>
mattmoor added a commit to mattmoor/fulcio that referenced this issue Oct 28, 2021
These are separate from the fixed OIDC issuers, and they represent templates for *classes* of OIDC endpoints that we want to support, e.g. for EKS:
```
https://oidc.eks.*.amazonaws.com/id/*
```

The `*` character here will be used to match `[a-zA-Z0-9_-]+`, so no host or path delimiting characters are allowed to prevent attacks like:

```
https://oidc.eks.mattmoor.io/pwned.amazonaws.com/id/does-not-matter
```

We do NOT maintain OIDC verifiers for all of the possible endpoints, but we do keep an LRU cache to avoid the expensive discovery process on each request.

Related: sigstore#212
Signed-off-by: Matt Moore <mattomata@gmail.com>
dlorenc pushed a commit that referenced this issue Oct 28, 2021
* Refactor the way we access `Config`

This change starts to refactor the way we interact with theh Fulcio `Config`
to support a forthcoming change to support what I've taken to calling "meta URLs".

I want to allow the public Fulcio instance to support the following *classes*
of Issuer URL, which are exposed for EKS and GKE clusters:
 * https://oidc.eks.us-west-2.amazonaws.com/id/B02C93B6A2D30341AD01E1B6D48164CB
 * https://container.googleapis.com/v1/projects/mattmoor-credit/locations/us-west1-b/clusters/tenant-cluster

To this end, I want to support adding to the Fulcio configuration "meta URLs"
of the form (still a strawman):
```
https://oidc.eks.*.amazonaws.com/id/*
https://container.googleapis.com/v1/projects/*/locations/*/clusters/*
```

However, in this change, I'm still just consolidating how we access the OIDC
issuer information in `FulcioConfig` so that lookups are performed via a new
`GetIssuer` method, and verifiers are instantiated via `GetVerifier()`.

The only remaining place I see iterating over `cfg.OIDCIssuers` is for the
`realm` in the `WWW-Authenticate` header of `401` responses (seems right to
not include the meta URLs here).  There are also still a handful of places
handling OIDCIssuers directly, but most are in `config.go`, or associated
validation logic (e.g. `federation/main.go`).

Related: #212

Signed-off-by: Matt Moore <mattomata@gmail.com>

* Make `ParseConfig` private, shift verification.

Signed-off-by: Matt Moore <mattomata@gmail.com>
mattmoor added a commit to mattmoor/fulcio that referenced this issue Oct 28, 2021
These are separate from the fixed OIDC issuers, and they represent templates for *classes* of OIDC endpoints that we want to support, e.g. for EKS:
```
https://oidc.eks.*.amazonaws.com/id/*
```

The `*` character here will be used to match `[a-zA-Z0-9_-]+`, so no host or path delimiting characters are allowed to prevent attacks like:

```
https://oidc.eks.mattmoor.io/pwned.amazonaws.com/id/does-not-matter
```

We do NOT maintain OIDC verifiers for all of the possible endpoints, but we do keep an LRU cache to avoid the expensive discovery process on each request.

Related: sigstore#212
Signed-off-by: Matt Moore <mattomata@gmail.com>
mattmoor added a commit to mattmoor/fulcio that referenced this issue Oct 28, 2021
These are separate from the fixed OIDC issuers, and they represent templates for *classes* of OIDC endpoints that we want to support, e.g. for EKS:
```
https://oidc.eks.*.amazonaws.com/id/*
```

The `*` character here will be used to match `[a-zA-Z0-9_-]+`, so no host or path delimiting characters are allowed to prevent attacks like:

```
https://oidc.eks.mattmoor.io/pwned.amazonaws.com/id/does-not-matter
```

We do NOT maintain OIDC verifiers for all of the possible endpoints, but we do keep an LRU cache to avoid the expensive discovery process on each request.

Related: sigstore#212
Signed-off-by: Matt Moore <mattomata@gmail.com>
mattmoor added a commit to mattmoor/fulcio that referenced this issue Oct 28, 2021
These are separate from the fixed OIDC issuers, and they represent templates for *classes* of OIDC endpoints that we want to support, e.g. for EKS:
```
https://oidc.eks.*.amazonaws.com/id/*
```

The `*` character here will be used to match `[a-zA-Z0-9_-]+`, so no host or path delimiting characters are allowed to prevent attacks like:

```
https://oidc.eks.mattmoor.io/pwned.amazonaws.com/id/does-not-matter
```

We do NOT maintain OIDC verifiers for all of the possible endpoints, but we do keep an LRU cache to avoid the expensive discovery process on each request.

Related: sigstore#212
Signed-off-by: Matt Moore <mattomata@gmail.com>
bobcallaway pushed a commit that referenced this issue Oct 28, 2021
These are separate from the fixed OIDC issuers, and they represent templates for *classes* of OIDC endpoints that we want to support, e.g. for EKS:
```
https://oidc.eks.*.amazonaws.com/id/*
```

The `*` character here will be used to match `[a-zA-Z0-9_-]+`, so no host or path delimiting characters are allowed to prevent attacks like:

```
https://oidc.eks.mattmoor.io/pwned.amazonaws.com/id/does-not-matter
```

We do NOT maintain OIDC verifiers for all of the possible endpoints, but we do keep an LRU cache to avoid the expensive discovery process on each request.

Related: #212
Signed-off-by: Matt Moore <mattomata@gmail.com>
mattmoor added a commit to mattmoor/fulcio that referenced this issue Oct 28, 2021
These are separate from the fixed OIDC issuers, and they represent templates for *classes* of OIDC endpoints that we want to support, e.g. for EKS:
```
https://oidc.eks.*.amazonaws.com/id/*
```

The `*` character here will be used to match `[a-zA-Z0-9_-]+`, so no host or path delimiting characters are allowed to prevent attacks like:

```
https://oidc.eks.mattmoor.io/pwned.amazonaws.com/id/does-not-matter
```

We do NOT maintain OIDC verifiers for all of the possible endpoints, but we do keep an LRU cache to avoid the expensive discovery process on each request.

Related: sigstore#212
Signed-off-by: Matt Moore <mattomata@gmail.com>
@mattmoor
Copy link
Member Author

So I was just able to run the following job against an EKS cluster (Graviton) and a GKE cluster (amd64):

apiVersion: batch/v1
kind: Job
metadata:
  name: check-oidc
spec:
  template:
    spec:
      restartPolicy: Never
      automountServiceAccountToken: false
      containers:
      - name: check-oidc
        image: ghcr.io/mattmoor/cosign@sha256:1a359871ef0dc34df81b8a33506bb51338b013a9827837aee678e4c118090084
        args: [
            "sign",
            # Skip upload because I'm too lazy to deal with credentials.
            "--upload=false",
            ghcr.io/mattmoor/cosign@sha256:1a359871ef0dc34df81b8a33506bb51338b013a9827837aee678e4c118090084
        ]
        env:
        - name: COSIGN_EXPERIMENTAL
          value: "true"
        volumeMounts:
        - name: oidc-info
          mountPath: /var/run/sigstore/cosign
      volumes:
        - name: oidc-info
          projected:
            sources:
              - serviceAccountToken:
                  path: oidc-token
                  expirationSeconds: 600 # Use as short-lived as possible.
                  audience: sigstore

So I think we are good here 🎉

wlynch pushed a commit to wlynch/sigstore that referenced this issue Jun 3, 2022
This change adds an ambient OIDC provider that will enable when the following
filesystem path is populated: `/var/run/sigstore/cosign/oidc-token`.  The intended
use of this (primarily) is to enable consuming Kubernetes OIDC tokens produced
through Service Account Projected Volumes.

To consume this you would add the following to your Kubernetes pod spec:
```yaml
      containers:
      - name: my-container-name
        image: ...
        volumeMounts:
        - name: oidc-info
          mountPath: /var/run/sigstore/cosign

      volumes:
        - name: oidc-info
          projected:
            sources:
              - serviceAccountToken:
                  path: oidc-token
                  expirationSeconds: 600 # Use as short-lived as possible.
                  audience: sigstore
```

This would also work with Tekton step definitions, or other things that permit
the use of projected volumes.

Note: Fulcio doesn't currently allow any Kubernetes provider OIDC tokens on the
public instance, but one of the things I plan to look at next is supporting the
endpoints from GKE and EKS (both of which have public discovery endpoints).

Related: sigstore/fulcio#219
Related: sigstore/fulcio#212

Signed-off-by: Matt Moore <mattomata@gmail.com>
wlynch pushed a commit to wlynch/sigstore that referenced this issue Jun 6, 2022
This change adds an ambient OIDC provider that will enable when the following
filesystem path is populated: `/var/run/sigstore/cosign/oidc-token`.  The intended
use of this (primarily) is to enable consuming Kubernetes OIDC tokens produced
through Service Account Projected Volumes.

To consume this you would add the following to your Kubernetes pod spec:
```yaml
      containers:
      - name: my-container-name
        image: ...
        volumeMounts:
        - name: oidc-info
          mountPath: /var/run/sigstore/cosign

      volumes:
        - name: oidc-info
          projected:
            sources:
              - serviceAccountToken:
                  path: oidc-token
                  expirationSeconds: 600 # Use as short-lived as possible.
                  audience: sigstore
```

This would also work with Tekton step definitions, or other things that permit
the use of projected volumes.

Note: Fulcio doesn't currently allow any Kubernetes provider OIDC tokens on the
public instance, but one of the things I plan to look at next is supporting the
endpoints from GKE and EKS (both of which have public discovery endpoints).

Related: sigstore/fulcio#219
Related: sigstore/fulcio#212

Signed-off-by: Matt Moore <mattomata@gmail.com>
Signed-off-by: Billy Lynch <billy@chainguard.dev>
wlynch pushed a commit to wlynch/sigstore that referenced this issue Jun 8, 2022
This change adds an ambient OIDC provider that will enable when the following
filesystem path is populated: `/var/run/sigstore/cosign/oidc-token`.  The intended
use of this (primarily) is to enable consuming Kubernetes OIDC tokens produced
through Service Account Projected Volumes.

To consume this you would add the following to your Kubernetes pod spec:
```yaml
      containers:
      - name: my-container-name
        image: ...
        volumeMounts:
        - name: oidc-info
          mountPath: /var/run/sigstore/cosign

      volumes:
        - name: oidc-info
          projected:
            sources:
              - serviceAccountToken:
                  path: oidc-token
                  expirationSeconds: 600 # Use as short-lived as possible.
                  audience: sigstore
```

This would also work with Tekton step definitions, or other things that permit
the use of projected volumes.

Note: Fulcio doesn't currently allow any Kubernetes provider OIDC tokens on the
public instance, but one of the things I plan to look at next is supporting the
endpoints from GKE and EKS (both of which have public discovery endpoints).

Related: sigstore/fulcio#219
Related: sigstore/fulcio#212

Signed-off-by: Matt Moore <mattomata@gmail.com>
Signed-off-by: Billy Lynch <billy@chainguard.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants