feat: Add support for HCP Vault Secrets

[twpayne]

Fixes #3061.

WIP. Not ready for review or merge yet.

[halostatue]

I’m playing with vlt myself and I’m not sure that --format json provides anything useful at this point; the metadata is limited and there doesn’t appear to be a way to get the secret value in the returned JSON:

$ vlt secrets get --format json test_secret | jq . | pbcopy
{
  "created_at": "2023-06-28T01:55:18.686Z",
  "created_by": {
    "email": "[REDACTED]",
    "name": "Austin Ziegler",
    "type": "TYPE_USER"
  },
  "latest_version": "2",
  "name": "test_secret",
  "version": {
    "created_at": "2023-06-28T01:56:11.053Z",
    "created_by": {
      "email": "[REDACTED]",
      "name": "Austin Ziegler",
      "type": "TYPE_USER"
    },
    "type": "kv",
    "version": "2"
  }
}

$ vlt secrets get --plaintext --format json test_secret | jq .
{
  "created_at": "2023-06-28T01:55:18.686Z",
  "created_by": {
    "email": "austin@zieglers.ca",
    "name": "Austin Ziegler",
    "type": "TYPE_USER"
  },
  "latest_version": "2",
  "name": "test_secret"
}

$ vlt secrets get --plaintext test_secret
real-test-value

vlt feels like a "cloud-enhanced" chezmoi secrets keyring. Still worth doing, but unless there’s a good use case or unless they add the ability to put the secret and application information into the metadata, I’m not sure that implementing the …Json template command is worthwhile.

[halostatue]

Another thought: vlt config init --local is not the default behaviour, but it feels generally safe enough that we could check that into the working tree or in the working tree directory specified by .chezmoiroot.

Unfortunately, it looks like chezmoi never changes $PWD, and that would require explicit specification or a single global configuration.

$ go run . execute-template '{{ hcpVaultSecrets "test_secret" }}'
chezmoi: template: arg1:1:3: executing "arg1" at <hcpVaultSecrets "test_secret">: error calling hcpVaultSecrets: /opt/homebrew/bin/vlt secrets get --plaintext test_secret: exit status 1
Error: organization ID must be set via `vlt config init` or passed in

exit status 1
$ go run . execute-template '{{ output "pwd" }}'
/Users/austin/dev/oss/forks/chezmoi

(I currently have .vlt.json in .chezmoiroot, but I think that the better location is just the chezmoi worktree root. Others might feel differently if they have two effective roots like in felipecrs/dotfiles.

[halostatue]

I’ve provided feedback with feature requests linking back to this draft PR:

I’m a contributor on twpayne/chezmoi and we’ve had someone ask to implement support for vlt / Vault Secrets, which the primary maintainer has started in #3067.

1. vlt secrets get --plaintext key does exactly what is required to support the minimum use case (putting some sort of secret value in place). This would be done in chezmoi with a template like {{ hcpVaultSecrets "key" "app-name" "project" "organization" }}`.

2. vlt secrets get --format json key doesn’t provide much useful information and there appears to be no way to get the JSON shape of the key and the secret value included in the JSON (I can understand where --format json might not be desired, but perhaps --format json+value). In the event that vlt config init or vlt config init --local has been run, it would be useful to include information like the app name, project name/id, and organization name/id. This would be a template command like hcpVaultSecretsJson, but would be most useful if the password were available in the JSON.

3. There appears to be no way to pass a path to a configuration file.

(This last would make it so that chezmoi could check for $CHEZMOIROOT/.vlt.json and pass --config-path $CHEZMOIROOT/.vlt.json instead of needing to change $PWD.)

[kartiklunkad26]

@halostatue thank you for providing your feedback! Would you be up for a conversation to make sure we understand the improvements request & correspondingly do the work to fix it?

My calendar in case you'd be up for it.

[twpayne]

Thanks very much for testing @halostatue!

I've updated the PR with a few changes based on your work and feedback:

• Added hcpVaultSecrets.organization, hcpVaultSecrets.project, and hcpVaultSecrets.appName config variables that are used as defaults if you don't specify an organization, project, or application name. Effectively, this allows you to provide a default --organization argument.
• Ensured that the working directory when running vlt is run is the destination directory (typically the user's home directory) so vlt can find .vlt.json. Not sure if the working directory should always be the user's home directory instead.
• Template functions renamed to hcpVaultSecret and hcpVaultSecretJson (singular, no s) to better reflect that fact that they return data from a single secret.
• Added hcpVaultSecrets.args to pass extra optional arguments to vlt in all cases.

@kartiklunkad26 I've booked 30 minutes with you next Friday.

[twpayne]

@arrrgi do you have any input on this?

[arrrgi]

Thanks for the invite to review progress @twpayne

If I'm to understand the WIP so far:

• the new feature requires the vlt CLI to be available on the target
• a Chezmoi user can provide a default configuration to be used if none are provided as overrides
• a template function allows a specific application/project/organisation to act as the root to fetch a name secret from

What I wasn't able to follow was the link to the .vlt.json file as I'm unsure where that fits into the usage pattern. Is that a file that needs to be checked into the local repo with Chezmoi to ensure portability?

I think there is also an undocumented assumption that before this works, you need to have installed the vlt CLI and exec vlt login which builds the oAuth session for subsequent requests with the client and this typically happens interactively with a browser. This might need a callout in the docs.

In a headless environment, will Chezmoi support the convention of respecting variables defined before the chezmoi command, eg. HCP_CLIENT_ID=xyz HCP_CLIENT_SECRET=foo chezmoi apply ?

Other than my couple of knowledge gap questions, it's a great implementation @twpayne and @halostatue !! Love it!

[arrrgi]

@kartiklunkad26 thanks also for supporting the Chezmoi community's feedback and requests. Sorry I didn't reach out to you directly as I consider myself a hobbyist consumer, not your usual mid-market/Ent. customer :)

[twpayne]

Thanks for taking a look!

A couple of answers:

What I wasn't able to follow was the link to the .vlt.json file as I'm unsure where that fits into the usage pattern. Is that a file that needs to be checked into the local repo with Chezmoi to ensure portability?

I don't think you need a .vlt.json file. I think (but have not verified) that you can instead set the hcpVaultSecrets.organization, hcpVaultSecrets.project, and hcpVaultSecrets.appName variables in your chezmoi configuration file.

I think there is also an undocumented assumption that before this works, you need to have installed the vlt CLI and exec vlt login which builds the oAuth session for subsequent requests with the client and this typically happens interactively with a browser. This might need a callout in the docs.

This is correct. You have to install the vlt CLI and run vlt login before running chezmoi. I'll update the docs.

In a headless environment, will Chezmoi support the convention of respecting variables defined before the chezmoi command, eg. HCP_CLIENT_ID=xyz HCP_CLIENT_SECRET=foo chezmoi apply ?

Yes, chezmoi passes its environment on to subcommands (like vlt) when it executes the subscommands.

[kartiklunkad26]

@arrrgi

Sorry I didn't reach out to you directly as I consider myself a hobbyist consumer, not your usual mid-market/Ent. customer :)

Our intent with Vault Secrets is to make it valuable for hobbyist consumers and SMB organizations, so your use-case is very much our focus! Nevertheless, let me know if you have other feedback points that you'd like to see improve in the product.

[twpayne]

This is now ready for review.

Thank you @arrrgi for the initial input and thank you @halostatue for the thorough investigation.

@kartiklunkad26 if you would be able to review the documentation, that would be fantastic.

[twpayne]

@kartiklunkad26 I'd like to do a new release of chezmoi by the end of this week. If you could review the docs by Thursday evening your time, that would be awesome.

[kartiklunkad26]

@twpayne Hey Tom. Sorry I could only get to this now. I'll review it today and leave any comments.