Enabling remote administration with Helm chart install

[larsks]

I'm trying to get started with step-ca running under Kubernetes. I'm installing using the helm chart version 1.18.2+20220324 and the client version 0.22.0.

I'm initializing the helm chart like this:

#!/bin/sh

for pwfile in ca-password.txt provisioner-password.txt; do
	if ! [ -f "$pwfile" ]; then
		apg -m25 -n1 > "$pwfile"
	fi

	tr -d '\n' < "$pwfile" | base64 > "${pwfile%.txt}.b64"
done

step ca init \
		--helm \
		--deployment-type=standalone \
		--name "Example Apps" \
		--dns ca.apps.example \
		--address :9000 \
		--provisioner admin \
		--password-file ca-password.txt \
		--provisioner-password-file provisioner-password.txt > values.yaml

Then I install step-ca:

helm install -f values.yaml \
  --set inject.secrets.ca_password=$(cat ca-password.b64) \
  --set inject.secrets.provisioner_password=$(cat provisioner-password.b64) \
  --set service.targetPort=9000 \
  step-certificates smallstep/step-certificates

And add a Traefik IngressRoute with passthrough TLS so that I can reach it:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: step-ca
spec:
  entryPoints:
    - websecure
  routes:
  - match: HostSNI(`ca.apps.example`)
    services:
    - name: step-certificates
      port: https
  tls:
    passthrough: true

And bootstrap my client using step ca bootstrap ....

This all seems to complete successfully. The CA seems healthy:

$ step ca health
ok

I can list provisioners:

$ step ca provisioner list
[
   {
      "type": "JWK",
      "name": "admin",
      "key": {
         "use": "sig",
         "kty": "EC",
         "kid": "...",
         "crv": "P-256",
         "alg": "ES256",
         "x": "...",
         "y": "..."
      },
      "encryptedKey": "...",
      "options": {
         "x509": {},
         "ssh": {}
      }
   }
]

And I can issue a certificate (using the password from provisioner-password.txt):

$ step ca certificate foo.apps.example foo.crt foo.key
✔ Provisioner: admin (JWK) [kid: trn1f8DnakEV4E9pMqAMup2wDDW-jaQACHNtdp01a_o]
Please enter the password to decrypt the provisioner key:
✔ CA: https://ca.apps.example
✔ Certificate: foo.crt
✔ Private Key: foo.key

Now, all the guides suggest that to enable ACME support I simply need to run:

$ step ca provisioner add acme --type ACME

There is no mention of authentication in the docs, but when I run this command I see:

No admin credentials found. You must login to execute admin commands.
✗ Please enter admin name/subject (e.g., name@example.com): █

What am I supposed to enter here...is this looking for the name of an existing provisioner? I tried entering admin here, which then prompts:

✔ Provisioner: admin (JWK) [kid: trn1f8DnakEV4E9pMqAMup2wDDW-jaQACHNtdp01a_o]
Please enter the password to decrypt the provisioner key: █

And this is where things fail. If I use the same password I used when performing the earlier step ca certificate command, it fails with an unhelpful error message:

json: cannot unmarshal number into Go value of type ca.AdminClientError

Am I doing something wrong? Is the documentation wrong?

When I respond to the "Please enter admin name/subject" prompt, I see this request in the logs:

{"duration":"40.45µs","duration-ns":40450,"fields.time":"2022-09-26T21:34:23Z","level":"info","method":"GET","msg":"","name":"ca","path":"/provisioners?limit=100","protocol":"HTTP/2.0","referer":"","remote-address":"10.244.0.32","request-id":"ccp1lnrfgsds739f2t90","size":900,"status":200,"time":"2022-09-26T21:34:23Z","user-agent":"Smallstep CLI/0.22.0 (linux/amd64)","user-id":""}
{"duration":"14.686µs","duration-ns":14686,"fields.time":"2022-09-26T21:34:23Z","level":"info","method":"GET","msg":"","name":"ca","path":"/provisioners/trn1f8DnakEV4E9pMqAMup2wDDW-jaQACHNtdp01a_o/encrypted-key","protocol":"HTTP/2.0","referer":"","remote-address":"10.244.0.32","request-id":"ccp1lnrfgsds739f2t9g","size":586,"status":200,"time":"2022-09-26T21:34:23Z","user-agent":"Smallstep CLI/0.22.0 (linux/amd64)","user-id":""}

And when I enter the password, I see:

{"certificate":"...","duration":"18.470129ms","duration-ns":18470129,"fields.time":"2022-09-26T21:34:27Z","issuer":"Oddbit Apps Intermediate CA","level":"info","method":"POST","msg":"","name":"ca","ott":"...","path":"/sign","protocol":"HTTP/2.0","provisioner":"admin (trn1f8DnakEV4E9pMqAMup2wDDW-jaQACHNtdp01a_o)","public-key":"ECDSA P-256","referer":"","remote-address":"10.244.0.32","request-id":"ccp1lorfgsds739f2tb0","serial":"221864705580655265740322077991489091158","size":3249,"status":201,"subject":"admin","time":"2022-09-26T21:34:27Z","user-agent":"Smallstep CLI/0.22.0 (linux/amd64)","user-id":"","valid-from":"2022-09-26T21:33:27Z","valid-to":"2022-09-27T21:34:27Z"}
{"duration":"9.218µs","duration-ns":9218,"fields.time":"2022-09-26T21:34:27Z","level":"warning","method":"POST","msg":"","name":"ca","path":"/admin/provisioners","protocol":"HTTP/2.0","referer":"","remote-address":"10.244.0.32","request-id":"ccp1lorfgsds739f2tbg","size":19,"status":404,"time":"2022-09-26T21:34:27Z","user-agent":"Smallstep CLI/0.22.0 (linux/amd64)","user-id":""}

[tashian]

Hi @larsks, the error message could definitely be clearer here.

The issue I think you're running into is, by default the step ca provisioner add/update/remove command groups will try to locally modify the $(step path)/config/ca.json CA config file. Outside of the container, step won't be able to find that file and it will attempt remote provisioner management instead.

You have a couple options here.

1. You could enable remote provisioner management (see the Remote Provisioner Management docs for instructions). This is especially useful if you will regularly need to modify provisioner configurations.
2. Or, you could run the step ca provisioner add command inside the container where the CA is running.

Be sure to restart the CA (or send it a HUP signal) to reload the configuration.

Hope this helps.

[larsks]

The issue I think you're running into is, by default the step ca provisioner add/update/remove command groups will try to locally modify the $(step path)/config/ca.json CA config file.

Thanks!

This is actually not terribly clear anywhere in the docs (especially given that several commands in the "Getting started" document are operating against a remote endpoint (via --ca-url).

I think it might be worth being explicit about that, especially since there are several deployment options that will -- by default -- result in the server not running in the same place as the client. Both the Docker and Helm documentation explicitly ask the deployer to install the step client locally.

[larsks]

Of course, trying to run step ca provisioner add acme --type ACME in the deployed Kubernetes pod fails with:

error opening /home/step/config/ca.json: open /home/step/config/ca.json: read-only file system

And that's because /home/step/config is mounted from a ConfigMap:

      - mountPath: /home/step/config
        name: config
        readOnly: true
[...]
    - configMap:
        defaultMode: 420
        name: step-certificates-config
      name: config

So it is apparently not possible to run that command local to the server, nor is it possible to run it remotely without additional configuration.

It looks like the only options ie enabling remote provision management, so I'm off to read those docs.

[larsks]

In our continuing story...

The remote provisioner management documentation suggests:

• Stop step-ca if it is running.
• Start step-ca from the command line. You will be prompted for a password that will encrypt the Admin JWK provisioner.

This is problematic: when deployed into a containerized environment such as Kubernetes, if we stop step-ca, then of course the container exits. It's not really possible to interact with step-ca via stdin. What's the correct way of setting this up?

To sum up where we are so far:

• We can't run the step client inside the container (because the config file is mounted via a ConfigMap which makes it read-only)

• We can't run the step client remotely because remote provisioner management is disabled by default

• We can configure ca.json via the helm chart values file like this...

  inject:
    config:
      files:
        ca.json:
          authority:
            provisioners: []
            enableAdmin: true
  

• ...but we can't ever set the password for the Admin JWK provisioner because there are only instructions for interactively configuring this.

What's the correct way to get this set up? The only thing I can come up with right now (and honestly I'm not at my best; I had two vaccinations earlier today and I can feel them starting to kick in) is to:

1. Arrange to mount the step-certificates-config path on a different location
2. Mount a PersistentVolume on /home/step/config
3. Add an initContainer to the StatefuleSet that will copy the files into /home/step/config

This will give us a writable /home/step/config so that we can run the step ca provisioner add... command locally, but it requires some substantial patching of helm output.

[tashian]

Hi Lars,

So, I just tested this out and I think configuring ca.json as you did, via the Helm chart, is the way to go.
To set the password, pass a --password-file into step-ca.
The first time you start up step-ca, the Admin JWK provisioner will be created using the password from the password file.
This password will be the password for your Super Admin (the username will be step).
Note that this is the same password that's used to decrypt the intermediate CA signing key.
(Unfortunately, there's no way to use a different password here yet.)

We will clarify the docs on this.
Especially since remote provisioner management is a great fit for Helm deployments!

[larsks]

@tashian, thanks for your help. Something still isn't right. By removing the provisioner configuration from ca.json, I now have the Admin JWK provisoiner as expected:

[
   {
      "type": "JWK",
      "name": "Admin JWK",
      "key": {
         ...
      },
      "encryptedKey": "...",
      "claims": {
         "defaultTLSCertDuration": "5m0s",
         "disableRenewal": false
      },
      "options": {
         "x509": {},
         "ssh": {}
      }
   }
]

The invocation of step-ca in the StatefulSet alreayd includes a --password-file option:

containers:
- command:
  - /usr/local/bin/step-ca
  - --password-file
  - /home/step/secrets/passwords/password
  - /home/step/config/ca.json

And /home/step/secrets/passwords/password is mounted from the step-certificates-ca-password secret; in the StatefulSet we see:

containeers:
[...]
  volumeMounts:
    - mountPath: /home/step/secrets/passwords
      name: ca-password
      readOnly: true
[...]
volumes:
  - name: ca-password
    secret:
      secretName: step-certificates-ca-password

But once the CA is running, I'm unable to add a new admin:

$ step ca admin add larsks "Admin JWK" \
  --password-file secrets/step-certificates-ca-password
No admin credentials found. You must login to execute admin commands.
✔ Please enter admin name/subject (e.g., name@example.com): step█
✔ Provisioner: Admin JWK (JWK) [kid: JzIZafoPFf94WyB1Vclfa8Xfg48PRCZrmeltUm68ihU]
x5c.authorizeToken; invalid x5c claims: square/go-jose/jwt: validation failed, invalid issuer claim (iss)

In the above command secrets/step-certificates-ca-password is the same files that ends up in the step-certificates-ca-password Secret.

I'm not sure what "invalid issuer claim" means or what to try next.

[tashian]

Hi Lars,

I'm not sure what's going on there.
Maybe it would be helpful to write up how I'm testing this locally.
I'm not using Helm or any containers, but the overall setup is otherwise the same.

echo "123456" > pw
step ca init --context remote-admin \
   --provisioner-password-file pw \
   --password-file pw \
   --deployment-type standalone \
   --name "Remote Admin Test" \
   --address :443 \
   --dns localhost \
   --provisioner test 

Output:

Generating root certificate... done!
Generating intermediate certificate... done!

✔ Root certificate: /Users/carl/.step/authorities/remote-admin/certs/root_ca.crt
✔ Root private key: /Users/carl/.step/authorities/remote-admin/secrets/root_ca_key
✔ Root fingerprint: bb9a208d515c4cec55470eac0e42e37c02b4b7c2999a25bd5b39a35ce6f35ace
✔ Intermediate certificate: /Users/carl/.step/authorities/remote-admin/certs/intermediate_ca.crt
✔ Intermediate private key: /Users/carl/.step/authorities/remote-admin/secrets/intermediate_ca_key
✔ Database folder: /Users/carl/.step/authorities/remote-admin/db
✔ Default configuration: /Users/carl/.step/authorities/remote-admin/config/defaults.json
✔ Default profile configuration: /Users/carl/.step/profiles/remote-admin/config/defaults.json
✔ Certificate Authority configuration: /Users/carl/.step/authorities/remote-admin/config/ca.json

Your PKI is ready to go. To generate certificates for individual services see 'step help ca'.

FEEDBACK 😍 🍻
  The step utility is not instrumented for usage statistics. It does not phone
  home. But your feedback is extremely valuable. Any information you can provide
  regarding how you’re using `step` helps. Please send us a sentence or two,
  good or bad at feedback@smallstep.com or join GitHub Discussions
  https://github.com/smallstep/certificates/discussions and our Discord
  https://u.step.sm/discord.

Then:

vi $(step path)/config/ca.json
# remove provisioner and enable remote admin
step-ca --password-file pw

Now step-ca starts up. In another window:

step ca admin add carl "Admin JWK" --password-file pw --admin-subject step

Output:

No admin credentials found. You must login to execute admin commands.
✔ Provisioner: Admin JWK (JWK) [kid: Mg2nFgGtPCdpFYWsnwqSTqFcDtN4Yh5mw4m9CA1lvWc]
SUBJECT PROVISIONER     TYPE
carl    Admin JWK (JWK) ADMIN

Does this help?

[maraino]

Hi @larsks, I haven't looked at all the threads, but @hslatman is working on a couple of PRs to be able to configure the admin interface when you run step ca init --helm, right now requires to do some changes in your ca.json

[larsks]

I have discovered the primary problem:

The most recent version of the helm chart installs step-ca version 0.18.2, and it looks like that doesn't have the remote admin support (or at least, it doesn't interoperate with a more recent client). The most recent version of the app appears to be 0.22.1; If I override the image installed by the helm chart, it appears to work as expected.

Is the helm chart still maintained?

step ca admin add carl "Admin JWK" --password-file pw --admin-subject step

Something that isn't clear in the documentation: after I create an admin user like this, how do I authenticate as that user? The admin add command never prompts for any sort of password. After showing the admin add command, the docs say:

You're all set. 🎉

You can use the step ca provisioner commands, from any client, to modify your CA's provisioner configuration.

[tashian]

Ahh yes. Great catch, thank you for pointing this out.
I've updated the docs on this piece.

There are no passwords associated with administrators.
An administrator is only a subject name and a provisioner name.
You authenticate as an administrator via the authentication scheme of the provisioner.
So, if you are able to get a certificate using a given subject name/provisioner pair, you can also administer the CA if that pair is registered as an Admin or Super Admin.
This adds a lot of flexibility to how Admins authenticate.
For example, Single Sign-On can be used for Admin users by adding an OIDC provisioner and associating an Admin user with it.

[tashian]

As for the Helm chart, we keep it updated regularly, but it's not in lockstep with step-ca releases.