mtls: SPIRE server and agent installation #24765

sayboras · 2023-04-05T13:26:28Z

Description

This is to install SPIRE server and agent for cilium mTLS.

TODO

Figure out the way to register the below identities as part of the installation
- init sidecar

Temp script

kubectl exec -n cilium-spire spire-server-0 -- \
  /opt/spire/bin/spire-server entry create \
  -spiffeID spiffe://spiffe.cilium/ns/spire/sa/spire-agent \
  -selector k8s_psat:agent_ns:cilium-spire \
  -selector k8s_psat:agent_sa:spire-agent \
  -node

# Then add the cilium-agent identity to the server (required for the delegated identity to work)
kubectl exec -n cilium-spire spire-server-0 -- \
  /opt/spire/bin/spire-server entry create \
  -spiffeID spiffe://spiffe.cilium/cilium-agent \
  -parentID spiffe://spiffe.cilium/ns/spire/sa/spire-agent \
  -selector k8s:ns:kube-system \
  -selector k8s:label:k8s-app:spiffetest

kubectl exec -n cilium-spire spire-server-0 -- \
  /opt/spire/bin/spire-server entry create \
  -spiffeID spiffe://spiffe.cilium/dclient \
  -parentID spiffe://spiffe.cilium/ns/spire/sa/spire-agent \
  -selector k8s:ns:kube-system \
  -selector k8s:sa:cilium

kubectl exec -n cilium-spire spire-server-0 -- \
  /opt/spire/bin/spire-server entry create \
  -spiffeID spiffe://spiffe.cilium/cilium-operator \
  -parentID spiffe://spiffe.cilium/ns/spire/sa/spire-agent \
  -selector k8s:ns:kube-system \
  -selector k8s:sa:cilium-operator

Fixes: #23806

Testing

Testing was done with fresh cilium installation with mTLS enabled.

auth.mTLS.spire.enabled=true
auth.mTLS.spire.install.enabled=true

$ cilium connectivity test --test auth
ℹ️  Single-node environment detected, enabling single-node connectivity test
ℹ️  Monitor aggregation detected, will skip some flow validation steps
...
[=] Skipping Test [client-egress-l7-set-header]
[=] Test [echo-ingress-auth-always-fail]
........
[=] Test [echo-ingress-auth-mtls-spiffe]
........
✅ All 2 tests (16 actions) successful, 35 tests skipped, 0 scenarios skipped.

mtls: SPIRE server and agent installation

sayboras · 2023-04-10T15:20:43Z

/test

sayboras · 2023-04-11T02:20:03Z

/test-1.16-4.19

sayboras · 2023-04-11T02:20:11Z

/test-1.24-5.4

sayboras · 2023-04-11T02:20:31Z

/test-1.25-4.19

sayboras · 2023-04-11T02:20:49Z

/test-1.26-net-next

sayboras · 2023-04-11T02:43:53Z

/test

sayboras · 2023-04-11T03:21:28Z

/test-1.26-net-next

meyskens · 2023-04-18T12:47:53Z

Adding a note that i got this error on my system with this PR on the spire agent, it could be me but better verify a later change did not break it.

time="2023-04-18T12:46:11Z" level=error msg="no identity issued" method=SubscribeToX509SVIDs service=spire.api.agent.delegatedidentity.v1.DelegatedIdentity subsystem_name=debug_api
time="2023-04-18T12:46:11Z" level=error msg="Failed to collect all selectors for PID" error="workload attestor \"k8s\" failed: rpc error: code = Internal desc = workloadattestor(k8s): unable to perform request: Get \"https://127.0.0.1:10250/pods\": x509: certificate signed by unknown authority" pid=5656 subsystem_name=workload_attestor
time="2023-04-18T12:46:11Z" level=error msg="no identity issued" method=SubscribeToX509Bundles service=spire.api.agent.delegatedidentity.v1.DelegatedIdentity subsystem_name=debug_api

install/kubernetes/cilium/templates/spire/server/clusterrole.yaml

squeed · 2023-04-18T12:53:28Z

install/kubernetes/cilium/values.yaml.tmpl

+          # -- SPIRE agent init containers
+          initContainers:
+          - name: init
+            image: cgr.dev/chainguard/wait-for-it@sha256:ecb58e3a2ffbdb732bb9049987e06eaf826d945410e167f31d6ffe28fab259f4


why does this need to be configurable?

FY I plan to revert this change in #24897

what @meyskens says 👍, we probably either change the default here, or just remove it from being configurable.

squeed · 2023-04-18T12:54:01Z

install/kubernetes/cilium/values.yaml.tmpl

+        # -- SPIRE agent configuration
+        agent:
+          # -- SPIRE agent image
+          image: ghcr.io/spiffe/spire-agent:1.5.1@sha256:40228af4d9a094f0fef2d7a303a3b6a689c4b4eba2fa9f7da5125b81d2d68ec8


Are we going to mirror these containers to a repository in the cilium organization?

most of users are kind of doing that i.e. copy public images to private registry (e.g. ecr, google, etc), so I think we don't need to do it ourselves.

squeed · 2023-04-18T12:57:47Z

How does this affect cluster bootstrap? Is a running SPIRE agent + server required for cilium agent start / restart / upgrade?

If a cluster is using an in-cluster-hosted CSI driver, the statefulset volume will not be available until after the network starts up, for example. Is that going to be a problem here?

This is highlighted as part of CFP. Signed-off-by: Tam Mach <tam.mach@cilium.io>

This is to avoid any confusion on valid DNS name, which could cause any assumption on external traffic/lookup as part of Auth process. Signed-off-by: Tam Mach <tam.mach@cilium.io>

This is to simplify the identity registration as part of bootstrap. Signed-off-by: Tam Mach <tam.mach@cilium.io>

sayboras · 2023-04-18T13:54:22Z

How does this affect cluster bootstrap? Is a running SPIRE agent + server required for cilium agent start / restart / upgrade?

The current implementation makes sure that cilium will not crash or wait for SPIRE component to bootstrap. Cilium will perform connection retry with back-off to SPIRE.

If a cluster is using an in-cluster-hosted CSI driver, the statefulset volume will not be available until after the network starts up, for example. Is that going to be a problem here?

Same as above, Cilium will automatically retry with backoff, so once SPIRE components are up, things will be converged.

maintainer-s-little-helper · 2023-04-18T13:56:05Z

Commit fbc1e1e75f0131da33c167ce451da94633a4b4d0 does not contain "Signed-off-by".

Please follow instructions provided in https://docs.cilium.io/en/stable/contributing/development/contributing_guide/#developer-s-certificate-of-origin

maintainer-s-little-helper · 2023-04-18T13:56:37Z

Commit fbc1e1e75f0131da33c167ce451da94633a4b4d0 does not contain "Signed-off-by".

Please follow instructions provided in https://docs.cilium.io/en/stable/contributing/development/contributing_guide/#developer-s-certificate-of-origin

ldelossa · 2023-04-18T14:19:07Z

@sayboras should this be marked ready to merge? Some tests seem to be still running, and we are failing a smoke testing.

sayboras · 2023-04-18T14:23:08Z

should this be marked ready to merge? Some tests seem to be still running, and we are failing a smoke testing.

Thanks, I didn't know that we need to remove ready-to-merge manually. I pushed one small change to incorporate review comments.

This commit is just to put the layout for current structure of spire configuration. All generated docs and files are included. Signed-off-by: Tam Mach <tam.mach@cilium.io>

This commit is to add all required manifests for both spire agent and server components. Signed-off-by: Tam Mach <tam.mach@cilium.io>

This commit is to register all cilium related identities in spire, so that other components (e.g. spire-agent, cilium agent, cilium operator) are having required permissions. Currently, a small bash script is used for simplicity, once things are getting more complicated, we can move these steps to a small golang utility, like what we have with cilium-mount or cilium-sysctlfix. Kind note that shareProcessNamespace is enabled, so that init container can cooperate with main spire-server and perform required identity registration. https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/ Signed-off-by: Tam Mach <tam.mach@cilium.io>

sayboras · 2023-04-19T10:10:41Z

/test

maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Apr 5, 2023

sayboras added the release-note/minor This PR changes functionality that users may find relevant to operating Cilium. label Apr 5, 2023

maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Apr 5, 2023

sayboras added area/helm Impacts helm charts and user deployment experience dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. labels Apr 5, 2023

maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Apr 5, 2023

sayboras added the area/servicemesh GH issues or PRs regarding servicemesh label Apr 5, 2023

sayboras force-pushed the tam/spire-installation branch 7 times, most recently from a315cd7 to 5a1d388 Compare April 10, 2023 15:02

sayboras marked this pull request as ready for review April 10, 2023 15:03

sayboras requested review from a team as code owners April 10, 2023 15:03

sayboras requested review from youngnick, squeed and zacharysarah April 10, 2023 15:03

maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Apr 18, 2023

squeed reviewed Apr 18, 2023

View reviewed changes

install/kubernetes/cilium/templates/spire/server/clusterrole.yaml Show resolved Hide resolved

squeed reviewed Apr 18, 2023

View reviewed changes

sayboras added 3 commits April 18, 2023 23:41

spire: Rename /cilium-id to /identity

5391ad5

This is highlighted as part of CFP. Signed-off-by: Tam Mach <tam.mach@cilium.io>

spire: Rename trust domain to spiffe.cilium

4b78ac1

This is to avoid any confusion on valid DNS name, which could cause any assumption on external traffic/lookup as part of Auth process. Signed-off-by: Tam Mach <tam.mach@cilium.io>

auth: Replace default Parent ID by /cilium-operator

cb4e1b6

This is to simplify the identity registration as part of bootstrap. Signed-off-by: Tam Mach <tam.mach@cilium.io>

sayboras force-pushed the tam/spire-installation branch from ddebd44 to fbc1e1e Compare April 18, 2023 13:56

maintainer-s-little-helper bot added the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label Apr 18, 2023

sayboras force-pushed the tam/spire-installation branch from fbc1e1e to c6bbee1 Compare April 18, 2023 13:56

sayboras removed the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label Apr 18, 2023

sayboras removed the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Apr 18, 2023

sayboras added 3 commits April 19, 2023 00:24

helm: Adjust the mTLS value structure

f439581

This commit is just to put the layout for current structure of spire configuration. All generated docs and files are included. Signed-off-by: Tam Mach <tam.mach@cilium.io>

spire: Add all required manifests for spire

30903d4

This commit is to add all required manifests for both spire agent and server components. Signed-off-by: Tam Mach <tam.mach@cilium.io>

sayboras force-pushed the tam/spire-installation branch from c6bbee1 to 71e000a Compare April 18, 2023 14:32

meyskens mentioned this pull request Apr 18, 2023

helm: Replace wait-for-it with a busybox script #24959

Merged

sayboras requested a review from squeed April 19, 2023 10:10

squeed approved these changes Apr 19, 2023

View reviewed changes

maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Apr 19, 2023

sayboras merged commit c268e97 into cilium:main Apr 19, 2023
57 checks passed

sayboras deleted the tam/spire-installation branch April 19, 2023 12:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mtls: SPIRE server and agent installation #24765

mtls: SPIRE server and agent installation #24765

sayboras commented Apr 5, 2023 •

edited

sayboras commented Apr 10, 2023

sayboras commented Apr 11, 2023

sayboras commented Apr 11, 2023

sayboras commented Apr 11, 2023

sayboras commented Apr 11, 2023

sayboras commented Apr 11, 2023

sayboras commented Apr 11, 2023

meyskens commented Apr 18, 2023

squeed Apr 18, 2023

meyskens Apr 18, 2023

sayboras Apr 18, 2023

squeed Apr 18, 2023

sayboras Apr 18, 2023

squeed commented Apr 18, 2023

sayboras commented Apr 18, 2023

maintainer-s-little-helper bot commented Apr 18, 2023

maintainer-s-little-helper bot commented Apr 18, 2023

ldelossa commented Apr 18, 2023

sayboras commented Apr 18, 2023

sayboras commented Apr 19, 2023

mtls: SPIRE server and agent installation #24765

mtls: SPIRE server and agent installation #24765

Conversation

sayboras commented Apr 5, 2023 • edited

Description

TODO

Testing

sayboras commented Apr 10, 2023

sayboras commented Apr 11, 2023

sayboras commented Apr 11, 2023

sayboras commented Apr 11, 2023

sayboras commented Apr 11, 2023

sayboras commented Apr 11, 2023

sayboras commented Apr 11, 2023

meyskens commented Apr 18, 2023

squeed Apr 18, 2023

Choose a reason for hiding this comment

meyskens Apr 18, 2023

Choose a reason for hiding this comment

sayboras Apr 18, 2023

Choose a reason for hiding this comment

squeed Apr 18, 2023

Choose a reason for hiding this comment

sayboras Apr 18, 2023

Choose a reason for hiding this comment

squeed commented Apr 18, 2023

sayboras commented Apr 18, 2023

maintainer-s-little-helper bot commented Apr 18, 2023

maintainer-s-little-helper bot commented Apr 18, 2023

ldelossa commented Apr 18, 2023

sayboras commented Apr 18, 2023

sayboras commented Apr 19, 2023

sayboras commented Apr 5, 2023 •

edited