Secret ttl handling

[kalafut]

The secret IDs from AppRole don't have leases, but they may have a TTL. A number of users have requested that Vault Agent (which consumes consul-template's templating logic) base fetching new secret IDs on the role's TTL, and not just the default 5 minute cadence for non-leased secrets. For users with shorter TTLs, this change will ensure the secret ID isn't expired. For those with much longer TTLs, it will reduced unnecessary early fetching of new secret IDs.

This PR pairs with (but doesn't require) hashicorp/vault#10826 which will start embedding the role TTL into the secret ID response.

[eikenb]

Thanks for the PR @kalafut and the review @calvn.

The code and tests all looks good but would you mind either writing some context or linking to some... something to say why this change was made. Thanks.

[kalafut]

@eikenb Whoops... yep. Missed adding some context when promoting this from draft. Updated.

[calvn]

Tested this manually as well, and the secret ID renewal honors the secret_id_ttl value (set to 10s).

Manual run output

› consul-template -config=config.hcl -log-level=debug
2021-02-08T17:46:23.103-0800 [INFO] consul-template v0.25.1 (210cb410)
2021-02-08T17:46:23.103-0800 [INFO] (runner) creating new runner (dry: false, once: false)
2021-02-08T17:46:23.104-0800 [DEBUG] (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":17,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"debug","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","CreateDestDirs":true,"Destination":"render.txt","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"template.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionDenylist":[],"SandboxPath":""}],"Vault":{"Address":"http://localhost:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":true,"Key":"","ServerName":"","Verify":true},"Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":17,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"BlockQueryWaitTime":60000000000}
2021-02-08T17:46:23.104-0800 [INFO] (runner) creating watcher
2021-02-08T17:46:23.104-0800 [INFO] (runner) starting
2021-02-08T17:46:23.104-0800 [DEBUG] (runner) running initial templates
2021-02-08T17:46:23.104-0800 [DEBUG] (runner) initiating run
2021-02-08T17:46:23.104-0800 [DEBUG] (runner) checking template ce1351b83544e8d21ddbcc7d457079f4
2021-02-08T17:46:23.104-0800 [DEBUG] (runner) missing data for 1 dependencies
2021-02-08T17:46:23.104-0800 [DEBUG] (runner) missing dependency: vault.write(auth/approle/role/my-role/secret-id -> ab03a894)
2021-02-08T17:46:23.104-0800 [DEBUG] (runner) add used dependency vault.write(auth/approle/role/my-role/secret-id -> ab03a894) to missing since isLeader but do not have a watcher
2021-02-08T17:46:23.104-0800 [DEBUG] (runner) was not watching 1 dependencies
2021-02-08T17:46:23.104-0800 [DEBUG] (watcher) adding vault.write(auth/approle/role/my-role/secret-id -> ab03a894)
2021-02-08T17:46:23.104-0800 [TRACE] (watcher) vault.write(auth/approle/role/my-role/secret-id -> ab03a894) starting
2021-02-08T17:46:23.104-0800 [DEBUG] (runner) diffing and updating dependencies
2021-02-08T17:46:23.104-0800 [DEBUG] (runner) watching 1 dependencies
2021-02-08T17:46:23.105-0800 [TRACE] (view) vault.write(auth/approle/role/my-role/secret-id -> ab03a894) starting fetch
2021-02-08T17:46:23.105-0800 [TRACE] vault.write(auth/approle/role/my-role/secret-id -> ab03a894): PUT /v1/auth/approle/role/my-role/secret-id?stale=true&wait=1m0s
2021-02-08T17:46:23.107-0800 [DEBUG] Found secret_id and set lease duration to 11 seconds
2021-02-08T17:46:23.107-0800 [TRACE] vault.write(auth/approle/role/my-role/secret-id -> ab03a894): non-renewable secret, set sleep for 9.376647586s
2021-02-08T17:46:23.107-0800 [TRACE] (view) vault.write(auth/approle/role/my-role/secret-id -> ab03a894) marking successful data response
2021-02-08T17:46:23.107-0800 [TRACE] (view) vault.write(auth/approle/role/my-role/secret-id -> ab03a894) successful contact, resetting retries
2021-02-08T17:46:23.215-0800 [TRACE] (view) vault.write(auth/approle/role/my-role/secret-id -> ab03a894) received data
2021-02-08T17:46:23.215-0800 [TRACE] (view) vault.write(auth/approle/role/my-role/secret-id -> ab03a894) starting fetch
2021-02-08T17:46:23.216-0800 [DEBUG] (runner) receiving dependency vault.write(auth/approle/role/my-role/secret-id -> ab03a894)
2021-02-08T17:46:23.216-0800 [DEBUG] (runner) initiating run
2021-02-08T17:46:23.216-0800 [DEBUG] (runner) checking template ce1351b83544e8d21ddbcc7d457079f4
2021-02-08T17:46:23.216-0800 [DEBUG] (runner) rendering "template.ctmpl" => "render.txt"
2021-02-08T17:46:23.249-0800 [INFO] (runner) rendered "template.ctmpl" => "render.txt"
2021-02-08T17:46:23.249-0800 [DEBUG] (runner) diffing and updating dependencies
2021-02-08T17:46:23.249-0800 [DEBUG] (runner) vault.write(auth/approle/role/my-role/secret-id -> ab03a894) is still needed
2021-02-08T17:46:23.249-0800 [DEBUG] (runner) watching 1 dependencies
2021-02-08T17:46:23.249-0800 [DEBUG] (runner) all templates rendered
2021-02-08T17:46:32.597-0800 [TRACE] vault.write(auth/approle/role/my-role/secret-id -> ab03a894): PUT /v1/auth/approle/role/my-role/secret-id?index=1612835183&stale=true&wait=1m0s
2021-02-08T17:46:32.598-0800 [DEBUG] (cli) receiving signal "urgent I/O condition"
2021-02-08T17:46:32.600-0800 [DEBUG] Found secret_id and set lease duration to 11 seconds
2021-02-08T17:46:32.600-0800 [TRACE] vault.write(auth/approle/role/my-role/secret-id -> ab03a894): non-renewable secret, set sleep for 9.952091794s
2021-02-08T17:46:32.600-0800 [TRACE] (view) vault.write(auth/approle/role/my-role/secret-id -> ab03a894) marking successful data response
2021-02-08T17:46:32.600-0800 [TRACE] (view) vault.write(auth/approle/role/my-role/secret-id -> ab03a894) successful contact, resetting retries
2021-02-08T17:46:32.600-0800 [TRACE] (view) vault.write(auth/approle/role/my-role/secret-id -> ab03a894) received data
2021-02-08T17:46:32.600-0800 [TRACE] (view) vault.write(auth/approle/role/my-role/secret-id -> ab03a894) starting fetch
2021-02-08T17:46:32.600-0800 [DEBUG] (runner) receiving dependency vault.write(auth/approle/role/my-role/secret-id -> ab03a894)
2021-02-08T17:46:32.600-0800 [DEBUG] (runner) initiating run
2021-02-08T17:46:32.600-0800 [DEBUG] (runner) checking template ce1351b83544e8d21ddbcc7d457079f4
2021-02-08T17:46:32.600-0800 [DEBUG] (runner) rendering "template.ctmpl" => "render.txt"
2021-02-08T17:46:32.626-0800 [INFO] (runner) rendered "template.ctmpl" => "render.txt"
2021-02-08T17:46:32.626-0800 [DEBUG] (runner) diffing and updating dependencies
2021-02-08T17:46:32.626-0800 [DEBUG] (runner) vault.write(auth/approle/role/my-role/secret-id -> ab03a894) is still needed
2021-02-08T17:46:32.626-0800 [DEBUG] (runner) watching 1 dependencies
2021-02-08T17:46:32.626-0800 [DEBUG] (runner) all templates rendered
2021-02-08T17:46:32.626-0800 [DEBUG] (cli) receiving signal "urgent I/O condition"
2021-02-08T17:46:42.555-0800 [TRACE] vault.write(auth/approle/role/my-role/secret-id -> ab03a894): PUT /v1/auth/approle/role/my-role/secret-id?index=1612835192&stale=true&wait=1m0s
2021-02-08T17:46:42.556-0800 [DEBUG] (cli) receiving signal "urgent I/O condition"
2021-02-08T17:46:42.557-0800 [DEBUG] Found secret_id and set lease duration to 11 seconds
2021-02-08T17:46:42.557-0800 [TRACE] vault.write(auth/approle/role/my-role/secret-id -> ab03a894): non-renewable secret, set sleep for 9.798086159s
2021-02-08T17:46:42.558-0800 [TRACE] (view) vault.write(auth/approle/role/my-role/secret-id -> ab03a894) marking successful data response
2021-02-08T17:46:42.558-0800 [TRACE] (view) vault.write(auth/approle/role/my-role/secret-id -> ab03a894) successful contact, resetting retries
2021-02-08T17:46:42.558-0800 [TRACE] (view) vault.write(auth/approle/role/my-role/secret-id -> ab03a894) received data
2021-02-08T17:46:42.558-0800 [TRACE] (view) vault.write(auth/approle/role/my-role/secret-id -> ab03a894) starting fetch
2021-02-08T17:46:42.558-0800 [DEBUG] (runner) receiving dependency vault.write(auth/approle/role/my-role/secret-id -> ab03a894)
2021-02-08T17:46:42.558-0800 [DEBUG] (runner) initiating run
2021-02-08T17:46:42.558-0800 [DEBUG] (runner) checking template ce1351b83544e8d21ddbcc7d457079f4
2021-02-08T17:46:42.558-0800 [DEBUG] (runner) rendering "template.ctmpl" => "render.txt"
2021-02-08T17:46:42.600-0800 [INFO] (runner) rendered "template.ctmpl" => "render.txt"
2021-02-08T17:46:42.600-0800 [DEBUG] (runner) diffing and updating dependencies
2021-02-08T17:46:42.600-0800 [DEBUG] (runner) vault.write(auth/approle/role/my-role/secret-id -> ab03a894) is still needed
2021-02-08T17:46:42.600-0800 [DEBUG] (runner) watching 1 dependencies
2021-02-08T17:46:42.600-0800 [DEBUG] (runner) all templates rendered

[calvn]

LGTM!

[eikenb]

I'll merge this when I review what to add to the point release I'll be making.

[eikenb]

LGTM