[Feature Request] Put Vault token in Raft and provide an API to set it.

[the-maldridge]

It would be nice to have an API to set the nomad server vault token so that it doesn't need to be placed into a config file on the server machines. My current use case is that this would allow the use of Nomad in places where remote configuration management is not a clean solution, but talking to the nomad API is.

It would also make it significantly easier to rotate the token.

[OneCricketeer]

What if you used Consul template to create the nomad config?

[the-maldridge]

While technically possible, this only moves the problem one layer further down. I'm working in an environment where I would like to not place a secret into the configuration archive, and rather would like to remotely provide it to nomad during bootstrapping.

[jrasell]

Hi @the-maldridge, thanks for raising this issue. It is certainly an interesting idea and would make the UX in handling certain token related tasks nicer. That being said, this would be something we would need to discuss seeing as there are security implications.

[the-maldridge]

@jrasell For sure. There are also security implications for having a very high powered token sitting in a file for Nomad to pick up, and I'm hopeful to avoid that.

With the release of Vault 1.4 and integrated storage, Nomad becomes the last part of the Hashicorp stack that by default can form a bootstrapping loop with another service when collocated. I'm "solving" this right now by doing the bootstrap of a cluster in two passes and rebooting in between, but this is less than ideal.

[apollo13]

Would this also be fixed via #15617 @the-maldridge?

[the-maldridge]

@apollo13 it would to an extend. The problem I have with the solution outlined in #15617 is that it is predicated on having full TLS setup in an environment, which significantly raises the bar to just try out Nomad.

[lgfa29]

it is predicated on having full TLS setup in an environment

Luckily it doesn't require a full TLS setup. I've been testing it locally just fine, and I'm too lazy to start a full mTLS cluster every time 😅

I'm curious to which part made you think that, we may need to adjust the description.

[the-maldridge]

@lgfa29 my understanding is that OIDC needs full TLS (note TLS, not mTLS) to be able to perform key discovery and import. I had always understood this to be a spec conformance item, but perhaps the Nomad team has allowed that key import to happen over an insecure channel?

[lgfa29]

Ah ok, yeah so the issue is a bit light on the details, probably because the implementation wasn't fully clear by then.

The initial plan is to use Nomad's JWKS URL instead of OIDC. The workflow would be something like this:

1. Configure a JWT auth method in Vault pointing to Nomad's JWKS URL, something like this:

   {
     "jwks_url": "http://localhost:4646/.well-known/jwks.json",
     "jwt_supported_algs": ["EdDSA"],
     "default_role": "nomad-workloads"
   }

2. Create a role used to configure the Vault tokens derived for Nomad tasks.

   {
     "role_type": "jwt",
     "bound_audiences": "vault.io",
     "user_claim": "/nomad_namespace",
     "user_claim_json_pointer": true,
     "claim_mappings": {
       "nomad_namespace": "nomad_namespace",
       "nomad_job_id": "nomad_job_id"
     },
     "token_ttl": "30m",
     "token_type": "service",
     "token_period": "72h",
     "token_policies": ["nomad-workloads"]
   }

3. Create a templated Vault ACL policy to define access permission for Nomad taks. For example, this policy grants automatic access to secrets in the path secret/<job namespace>/<job ID>.

   path "secret/data/{{identity.entity.aliases.auth_jwt_X.metadata.nomad_namespace}}/{{identity.entity.aliases.auth_jwt_X.metadata.nomad_job_id}}/*" {
     capabilities = ["read"]
   }
   
   path "secret/data/{{identity.entity.aliases.auth_jwt_X.metadata.nomad_namespace}}/{{identity.entity.aliases.auth_jwt_X.metadata.nomad_job_id}}" {
     capabilities = ["read"]
   }
   
   path "secret/metadata/{{identity.entity.aliases.auth_jwt_X.metadata.nomad_namespace}}/*" {
     capabilities = ["list"]
   }
   
   path "secret/metadata/*" {
     capabilities = ["list"]
   }

4. Run a job with a vault block.

So for each task with a vault block Nomad will mint an JWT attesting that it is a valid Nomad task. The Nomad client then presents this JWT to Vault to exchange it for a Vault ACL token for the task. Vault can verify the JWT is valid using the Nomad JWKS URL configured in the auth method.

This way the Nomad servers are no longer involved in the token creation and renewal process, so there's no need to provide any Vault token anywhere.

There's a bit of setup needed to properly configure Vault, but we're thinking of ways to help with that.

[tgross]

Starting in Nomad 1.7.0-beta.1 we've deprecated the use of Vault tokens in the Nomad agent configuration. Nomad will use workload identities to sign into Vault for purposes of getting Vault tokens for those workloads. Using Vault tokens in the Nomad configuration will be removed entirely in Nomad 1.9.

[the-maldridge]

For future people finding this issue, could you put a link to the learn guide for setting up workload identity, and/or a migration guide for going from the legacy workflow to the new one?

[tgross]

Oh, I expected the 1.7-beta to publish docs as well, but it looks like the pipeline bug we have around publishing docs for prereleases is still unfixed. In the meantime, there's some documentation describing the new workflow in https://github.com/hashicorp/nomad/blob/main/website/content/docs/integrations/vault-integration.mdx and a full tutorial for Vault and Consul coming out before GA. I'll make sure to circle back here once those are published.

[lgfa29]

@the-maldridge here's a preview of the tutorials:

• https://tutorials-git-1796-hashicorp.vercel.app/nomad/tutorials/integrate-consul/consul-acl
• https://tutorials-git-1796-hashicorp.vercel.app/nomad/tutorials/integrate-vault/vault-acl

The full documentation is still a work in progress, but let me know if you have any early feedback on these 🙂

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.