Vault agent doesn’t respect the number of retries #15725
Comments
|
This may have been fixed by #15204, available in Vault 1.11.0. |
|
Thank you. I'll test it and let you know. |
|
Hi @jasonodonnell, Unless I'm missing something the issue still persist: / $ vault version
Vault v1.11.1 (0f634755745f4adf62ec0723a0b93d6dce5bc33e), built 2022-07-19T20:16:47Z
/ $ cat /vault/config.json | grep -A 6 vault\":
"vault": {
"address": "https://vault.service.intra:8200",
"retry": [
{
"num_retries": "3"
}
]
/ $ cat /vault/config.json | grep -A 5 template_config
"template_config": [
{
"error_on_missing_key": false,
"exit_on_retry_failure": false
}
],
/ $Logs: |
|
I will take a closer look, but unfortunately the retry logic is complicated due to the interaction of sub systems in agent as per the documentation: https://www.vaultproject.io/docs/agent#retry-stanza |
|
I did find this note in the code for template: vault/command/agent/template/template.go Line 267 in 637d4bd |
|
Hi there! I'm pretty sure my PR #16970 will fix this issue. To describe the current behaviour: the issue was that when caching was enabled, configured retries were not respected. If This issue should be resolved soon and fixed in 1.12.0. Thanks for the report! |
|
Hi there! I'm going to close this issue as I just merged #16970 which should fix the issue identified here. It should now respect the number of retries in all cases, regardless of if caching is enabled. This should release in 1.12. Thanks for the bug report! |
Describe the bug
Looks like the Vault agent doesn’t respect the number of retries.
I also tried setting the
VAULT_MAX_RETRIESENV variable.To Reproduce
Steps to reproduce the behavior:
Please see the configuration in the environment section below.
Expected behavior
According to the docs, I expect that the Vault agent won’t exit on an error and that the number of retries will be three. This is what I’m getting:
I tried to set the VAULT_MAX_RETRIES ENV variable but no luck.
There is an option to pass the Consul template retry config but the problem is that I don’t understand how to pass it from the Vault config.
This is the Consul template config for the Vault config shown below:
[DEBUG] (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","TokenFile":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":33,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"DEBUG","FileLog":{"LogFilePath":"","LogRotateBytes":0,"LogRotateDuration":86400000000000,"LogRotateMaxFiles":0},"MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":[],"CommandTimeout":30000000000,"Contents":"{{ with secret \"/dev/kv/secrets/app1/secret-new\" }}{{ .Data.data.value }}{{ end }}","CreateDestDirs":true,"Destination":"/secrets/secret-new","ErrMissingKey":false,"ErrFatal":true,"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"User":null,"Uid":null,"Group":null,"Gid":null,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}","FunctionDenylist":[],"SandboxPath":""}],"TemplateErrFatal":null,"Vault":{"Address":"http://127.0.0.1:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"CustomDialer":{},"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":33,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false,"DefaultLeaseDuration":300000000000,"LeaseRenewalThreshold":0.9},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"ParseOnly":false,"BlockQueryWaitTime":60000000000}I can tell that the config is being parsed properly, because if I pass two values in the retry stanza I got an error that only one is allowed.
Environment:
vault status):1.10.1vault version):1.10.3Vault agent configuration file(s):
{ "auto_auth": { "method": { "type": "kubernetes", "mount_path": "auth/kube_cluster_c5", "config": { "role": "secrets-test-dev" } }, "sink": [ { "type": "file", "config": { "path": "/vault/vault-token" } } ] }, "cache": { "use_auto_auth_token": "true" }, "exit_after_auth": false, "pid_file": "/vault/vault.pid", "vault": { "address": "https://vault.service:8200", "retry": [ { "num_retries": "3" } ] }, "template_config": [ { "error_on_missing_key": false, "exit_on_retry_failure": false } ], "template": [ { "destination": "/secrets/secret-new", "contents": "{{ with secret \"/dev/kv/secrets/app1/secret-new\" }}{{ .Data.data.value }}{{ end }}", "left_delimiter": "{{", "right_delimiter": "}}" } ], "listener": [ { "type": "tcp", "address": "127.0.0.1:8200", "tls_disable": true } ] }Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: