k8s quickstart client not able to get SVID

[JaminenB]

Ran through the quickstart with k8s setup. Using the initial values provided (kept example.org trust domain and everything). No changes or updates made to the repo after cloning. When it came time to apply client-deployment.yaml it had problems receiving it's SVID which drove me bonkers because everything was copypasta. This is the error I received when the client workload came online from the pod logs:
rpc error: code = PermissionDenied desc = no identity issued

Eventually, I checked the spire-server-0 logs just to see what the problem could be and came across the client deployment selectors and failure (to make things easier to view I put each selector type on a new line):

time="2023-09-15T05:42:03Z" level=debug msg="PID attested to have selectors" pid=3149843 selectors="[
type:\"unix\" value:\"uid:0\" 
type:\"unix\" value:\"gid:0\" 
type:\"unix\" value:\"supplementary_gid:0\" 
type:\"k8s\" value:\"sa:default\" 
type:\"k8s\" value:\"ns:spire\" 
type:\"k8s\" value:\"node-name:aks-default-xxxxxx\" 
type:\"k8s\" value:\"pod-uid:xxxxx-xxxxx-xxxx\" 
type:\"k8s\" value:\"pod-name:client-98d8b6bfb-6fnt7\" 
type:\"k8s\" value:\"pod-image-count:1\" 
type:\"k8s\" value:\"pod-init-image-count:0\" 
type:\"k8s\" value:\"pod-image:ghcr.io/spiffe/spire-agent@sha256:xxxxxx\"
type:\"k8s\" value:\"pod-image:ghcr.io/spiffe/spire-agent:1.5.1\" 
type:\"k8s\" value:\"pod-label:pod-template-hash:98d8b6bfb\" 
type:\"k8s\" value:\"pod-label:app:client\" 
type:\"k8s\" value:\"pod-owner:ReplicaSet:client-98d8b6bfb\" 
type:\"k8s\" value:\"pod-owner-uid:ReplicaSet:xxxxxx-xxxxx-xxxx\" 
type:\"k8s\" value:\"container-name:client\" 
type:\"k8s\" value:\"container-image:ghcr.io/spiffe/pire-agent@sha256:xxxxxxxxxx\" 
type:\"k8s\" value:\"container-image:ghcr.io/spiffe/spire-agent:1.5.1\"]" 
subsystem_name=workload_attestor

time="2023-09-15T05:42:03Z" level=error msg="No identity issued" method=FetchX509SVID pid=3149843 
registered=false service=WorkloadAPI subsystem_name=endpoints

From the above log output the two selectors of importance to the quickstart are type:\"k8s\" value:\"sa:default\" and type:\"k8s\" value:\"ns:spire\".

Here is the entry command provided by the quickstart:

kubectl exec -n spire spire-server-0 -- \
    /opt/spire/bin/spire-server entry create \
    -spiffeID spiffe://example.org/ns/default/sa/default \
    -parentID spiffe://example.org/ns/spire/sa/spire-agent \
    -selector k8s:ns:default \
    -selector k8s:sa:default

This shows one of the selectors to be k8s:ns:default but it should be k8s:ns:spire instead. After creating a new entry with that change, the client was able to get an SVID:

kubectl exec -n spire spire-server-0 -- \
    /opt/spire/bin/spire-server entry create \
    -spiffeID spiffe://example.org/ns/default/sa/default \
    -parentID spiffe://example.org/ns/spire/sa/spire-agent \
    -selector k8s:ns:spire \ <-------- CHANGE MADE HERE
    -selector k8s:sa:default

Now we can see the container get it's SVID:

$kubectl exec -it $(kubectl get pods -o=jsonpath='{.items[0].metadata.name}' \
   -l app=client)  -- /opt/spire/bin/spire-agent api fetch -socketPath /run/spire/sockets/agent.sock

rpc error: code = PermissionDenied desc = no identity issued
rpc error: code = PermissionDenied desc = no identity issued
rpc error: code = PermissionDenied desc = no identity issued
rpc error: code = PermissionDenied desc = no identity issued
rpc error: code = PermissionDenied desc = no identity issued
rpc error: code = PermissionDenied desc = no identity issued
Received 1 svid after 9h29m5.086759695s

SPIFFE ID:              spiffe://example.org/ns/default/sa/default
SVID Valid After:       2023-09-15 15:10:33 +0000 UTC
SVID Valid Until:       2023-09-15 16:10:43 +0000 UTC
CA #1 Valid After:      2023-09-15 05:38:38 +0000 UTC
CA #1 Valid Until:      2023-09-16 05:38:48 +0000 UTC

Suggestion to edit the provided entry create command in the quickstart k8s portion to include the correct selector k8s:ns:spire.