agent does not respect kv ttl

[deitch]

Describe the bug

vault agent does not respect a ttl setting in kv secrets

To Reproduce

Steps to reproduce the behavior:

1. Configure a vault agent template
2. Run vault kv put secret/foo a=b ttl=30s
3. Run vault agent
4. See it render the template
5. Run vault kv patch secret/foo a=q
6. Wait at least 5 mins, sometimes more / sometimes less, for it to re-render the template

Expected behavior

Respect the ttl for kv, or have a different method for tweaking it.

Environment:

• Vault Server Version (retrieve with vault status):1.3.2
• Vault CLI Version (retrieve with vault version): v1.3.2
• Server Operating System/Architecture: macOS 10.15.1

Vault server configuration file(s):

storage "file" {
  path = "./data"
}

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_cert_file = "./server/cert.pem"
  tls_key_file = "./server/key.pem"
}

Additional context

The docs on kv state that having a key of type ttl creates an implied lease duration, which should be respected by clients. Vault agent should respect that and re-render templates that use those secrets.

I find the ttl a bit odd, as it mixes actual data with metadata, and can lead to accidental overwrite/deletion; ttl is a property of the secret, not a member of it. Even so, agent should respect it.

[catsby]

Hello! The ttl key is only offered in KV version 1:

• https://www.vaultproject.io/docs/secrets/kv/kv-v1/#ttls

I assume you're using KV version 2 because you mention patch, which is only offered in KV 2. Unfortunately KV 2 does not support adding a ttl to a version of a secret. Unless I'm mistaken, when you patch with the ttl value above, the ttl key is just part of the data and not a TTL value for the metadata

[deitch]

I am indeed using v2.

So what is the right way to control agent refresh of kv secrets - and therefore template generation - for v1 and for v2?

[catsby]

At this time there is not much control available for KV 2. Vault Agent uses Consul-Template internally to manage the rendered templates, which calculates when to check for updated versions. For secrets that do not have a lease (such as KV version 2), Consul-Template calculates a random time based off of a default duration of 5 minutes, as shown here:

https://github.com/hashicorp/consul-template/blob/d675da81accde69ae7d0fdc9da78c104f774dce4/dependency/vault_common.go#L160-L163

Offering users better control on the refresh rate seems like a reasonable feature, however it would need to be added upstream to Consul Template.

[deitch]

So it is calculated within the template? I didn’t realize that.

• can we add to the vault-agent documentation pages that the templates are refreshed based on consul template rules, see (link to docs about that)? Save others the time I spent going through it (and bothering you 😃)
• if you can point me to a starting point for getting control into a template, I’m happy to look at upstreaming it
• would something without a lease belong in the template? Or would it be something the agent should decide?

[catsby]

Hey @deitch - thank you for offering to contribute here! As far as starting point in consul template, I'm afraid I may not be the best to guide here. Perhaps post an issue on https://github.com/hashicorp/consul-template/issues asking?

would something without a lease belong in the template?

I don't see why not. It's true that there is little control offered here, but Consul Template still refreshes on a best effort based on a default duration. Exposing a knob that allows users to customize that duration (or otherwise set a refresh period) seems reasonable, but would first need to be added into Consul Template itself.

I'm going to close this issue for now. Thanks again!