Vault agent certificate authentication error #10835

datadude816 · 2021-02-03T21:39:54Z

Describe the bug
When using the vault agent for auto-auth with cert auth

To Reproduce
Steps to reproduce the behavior:

create and sign a certificate
enable cert auth in vault
upload the certificate to vault
configure vault agent to use certificate
run the the agent with the configuration
sudo -u vault /usr/local/bin/vault agent -config=/etc/vault.d/vault.hcl

Expected behavior
I expect the successful authentication with the token populated in the file specified in the config file.

Environment:

Vault Server Version (retrieve with vault status): 1.6.2 and 1.5.3
Vault CLI Version (retrieve with vault version): 1.6.2 and 1.5.3
Server Operating System/Architecture: RHEL 8

Vault server configuration file(s):

Vault agent config
vault { tls_disable = false client_key = “/etc/pki/tls/certs/machine.key” client_cert = “/etc/pki/tls/certs/machine.pem” ca_cert = “/etc/pki/ca-trust/source/anchors/CA.pem” address = “https://active.vault.service.consul.domain.us:8200” } pid_file = “/var/vault/.pidfile” exit_after_auth = false auto_auth { method “cert” { name = “p5520” mount_path = “cert/certs/test0” } sink “file” { config = { path = “/var/vault/token” } } }

Additional context
Using the same certificate as the agent I can authenticate just fine with the CLI and the API using curl

`sudo -u vault curl \

--request POST
--cert "/etc/pki/tls/certs/machine.pem"
--key "/etc/pki/tls/certs/machine.key"
--data '{"name": "p5520"}'
https://active.vault.service.consul.domain.us:8200/v1/auth/cert/certs/test0/login
{“request_id”:“0cb89ae4-f353-f548-fef6-836f5d06d02d”,“lease_id”:"",“renewable”:false,“lease_duration”:0,“data”:null,“wrap_info”:null,“warnings”:null,“auth”:{“client_token”:“s.IQHi4lE3OrG8A0vlskVuQG3L”,“accessor”:“9fxoyhm3yPl0dq7PQhCEjSTz”,“policies”:[“test”],“token_policies”:[“test”],“metadata”:{“authority_key_id”:“06:d9:9f:85:c0:07:11:04:f2:72:49:74:20:22:25:35:9e:99:1a:3c”,“cert_name”:“p5520”,“common_name”:“p5520-ada.domain.us”,“serial_number”:“345185377614830858”,“subject_key_id”:“51:77:01:60:d8:22:a9:01:1e:79:05:68:f2:77:02:9b:3b:31:e3:00”},“lease_duration”:2764800,“renewable”:true,“entity_id”:“5fcb2a3e-f22e-ceaa-d31b-9d387529d657”,“token_type”:“service”,“orphan”:true}}`

`sudo -u vault -E /usr/local/bin/vault login -method=cert -path=cert/certs/test0/ -client-cert=/etc/pki/tls/certs/machine.pem -client-key=/etc/pki/tls/certs/machine.key name=p5520
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run “vault login”
again. Future Vault requests will automatically use this token.

Key Value

token s.6qIIi9T7gvoOmLyVW7123456
token_accessor REhHEhJv89F3OYizwqr12345
token_duration 768h
token_renewable true
token_policies [“test”]
identity_policies
policies [“test”]
token_meta_common_name p5520-ada.domain.us
token_meta_serial_number 345185377614830858
token_meta_subject_key_id 51:77:01:60:d8:22:a9:01:1e:79:05:68:f2:77:02:9b:3b:31:e3:00
token_meta_authority_key_id 06:d9:9f:85:c0:07:11:04:f2:72:49:45:20:22:25:35:9e:45:1a:3c
token_meta_cert_name p5520`

The text was updated successfully, but these errors were encountered:

datadude816 · 2021-02-16T19:55:13Z

Here is the error

==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

   Api Address 1: http://127.0.0.1:8200
             Cgo: disabled
       Log Level: info
         Version: Vault v1.6.2
     Version Sha: be65a227ef2e80f8588b3b13584b5c0d9238c1d7

2021-02-02T13:59:23.485-0700 [INFO] sink.file: creating file sink
2021-02-02T13:59:23.486-0700 [INFO] sink.file: file sink configured: path=/var/vault/token mode=-rw-r-----
2021-02-02T13:59:23.486-0700 [INFO] template.server: starting template server
2021-02-02T13:59:23.486-0700 [INFO] template.server: no templates found
2021-02-02T13:59:23.486-0700 [INFO] auth.handler: starting auth handler
2021-02-02T13:59:23.486-0700 [INFO] auth.handler: authenticating
2021-02-02T13:59:23.486-0700 [INFO] sink.server: starting sink server
2021-02-02T13:59:23.576-0700 [ERROR] auth.handler: error authenticating: error="Error making API request.

URL: PUT https://active.vault.service.consul.domain.us:8200/v1/cert/certs/test0/login
Code: 403. Errors:

1 error occurred:
permission denied
" backoff=1.990850627

calvn · 2021-02-16T21:05:15Z

Since you're providing a mount_path explicitly, you will need to provide the full path, including the auth/ prefix like so:

method “cert” { name = “p5520” mount_path = “auth/cert/certs/test0” }

datadude816 · 2021-02-16T21:18:37Z

Then I get.....

sudo -u vault /usr/local/bin/vault agent -config=/etc/vault.d/vault.hcl
==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                 Cgo: disabled
           Log Level: info
             Version: Vault v1.6.2
         Version Sha: be65a227ef2e80f8588b3b13584b5c0d9238c1d7

2021-02-16T14:12:43.981-0700 [INFO] sink.file: creating file sink
2021-02-16T14:12:43.983-0700 [INFO] sink.file: file sink configured: path=/var/vault/token mode=-rw-r-----
2021-02-16T14:12:43.984-0700 [INFO] sink.server: starting sink server
2021-02-16T14:12:43.984-0700 [INFO] template.server: starting template server
2021-02-16T14:12:43.985-0700 [INFO] template.server: no templates found
2021-02-16T14:12:43.985-0700 [INFO] auth.handler: starting auth handler
2021-02-16T14:12:43.985-0700 [INFO] auth.handler: authenticating
2021-02-16T14:12:44.081-0700 [ERROR] auth.handler: error authenticating: error="Error making API request.

URL: PUT https://active.vault.service.consul.domain.us:8200/v1/auth/cert/certs/test0/login
Code: 400. Errors:

client certificate must be supplied" backoff=2.182675565

calvn · 2021-02-16T21:23:55Z

You'll also have to include the client_cert in the method “cert” {..} stanza itself. It looks like you're using agent solely for auto-auth and token sink, so it's strongly recommended to move all the TLS params to the auto-auth stanza (client_key and ca_cert).

datadude816 · 2021-02-16T21:35:01Z

Actually I had all the cert config under the "cert" stanza....so it either want both or under the "vault" stanza

So what I am finding is that it currently requires the cert configuration info under the "vault" stanza and the full path as you described.
This works:

`vault {
tls_disable = false
address = "https://active.vault.service.consul.domain.us:8200"
client_key = "/etc/pki/tls/certs/machine.key"
client_cert = "/etc/pki/tls/certs/machine.pem"
ca_cert = "/etc/pki/ca-trust/source/anchors/CA.pem"

}
pid_file = "/var/vault/.pidfile"
exit_after_auth = false
log_level = "trace"
auto_auth {
method "cert" {
name = "p5520"
mount_path = "auth/cert/certs/test0"
}

sink "file" {
config = {
path = "/var/vault/token"
}
}
}
`

This does not"
`vault {
tls_disable = false
address = "https://active.vault.service.consul.domain.us:8200"
}
pid_file = "/var/vault/.pidfile"
exit_after_auth = false
log_level = "trace"
auto_auth {
method "cert" {
name = "p5520"
mount_path = "auth/cert/certs/test0"
client_key = "/etc/pki/tls/certs/machine.key"
client_cert = "/etc/pki/tls/certs/machine.pem"
ca_cert = "/etc/pki/ca-trust/source/anchors/CA.pem"
}

sink "file" {
config = {
path = "/var/vault/token"
}
}
}
`

calvn · 2021-02-16T21:47:01Z

What's the error that you get when using the second config?

calvn · 2021-02-16T21:53:06Z

If the server is not is not set up with mTLS, try providing just the ca_cert in the vault stanza and see if that works (you still need all three in the cert method stanza).

datadude816 · 2021-02-16T21:55:27Z

The error with the second config is
"client certificate must be supplied" backoff=2.182675565"

datadude816 · 2021-02-16T22:07:02Z

This config does not work:

`vault {
tls_disable = false
address = "https://active.vault.service.consul.domain.us:8200"
ca_cert = "/etc/pki/ca-trust/source/anchors/CA.pem"

}
pid_file = "/var/vault/.pidfile"
exit_after_auth = false
log_level = "trace"
auto_auth {
method "cert" {
name = "p5520"
mount_path = "auth/cert/certs/test0"
client_key = "/etc/pki/tls/certs/machine.key"
client_cert = "/etc/pki/tls/certs/machine.pem"
ca_cert = "/etc/pki/ca-trust/source/anchors/CA.pem"
}

sink "file" {
config = {
path = "/var/vault/token"
}
}
}
`

results in the same error:
2021-02-16T14:56:38.805-0700 [INFO] auth.handler: authenticating
2021-02-16T14:56:38.809-0700 [ERROR] auth.handler: error authenticating: error="Error making API request.

URL: PUT https://active.vault.service.consul.domain.us:8200/v1/auth/cert/certs/test0/login
Code: 400. Errors:

client certificate must be supplied" backoff=2.565912036

calvn · 2021-02-16T23:05:13Z

Thanks for running that! I'm scanning through the codebase, but can't quite figure out why providing the info in the method stanza won't work. Can you run agent with -log-level=trace and see if "deriving auth client to use" shows as an output somewhere in that run (also noticed that agent doesn't accept log_level in the config)?

datadude816 · 2021-02-16T23:12:36Z

Yes it does...here is the complete output..
sudo -u vault /usr/local/bin/vault agent -config=/etc/vault.d/vault.hcl -log-level=trace
==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                 Cgo: disabled
           Log Level: trace
             Version: Vault v1.6.2
         Version Sha: be65a227ef2e80f8588b3b13584b5c0d9238c1d7

2021-02-16T16:08:47.944-0700 [INFO] sink.file: creating file sink
2021-02-16T16:08:47.944-0700 [TRACE] sink.file: enter write_token: path=/var/vault/token
2021-02-16T16:08:47.945-0700 [TRACE] sink.file: exit write_token: path=/var/vault/token
2021-02-16T16:08:47.945-0700 [INFO] sink.file: file sink configured: path=/var/vault/token mode=-rw-r-----
2021-02-16T16:08:47.945-0700 [INFO] template.server: starting template server
2021-02-16T16:08:47.945-0700 [INFO] template.server: no templates found
2021-02-16T16:08:47.945-0700 [INFO] sink.server: starting sink server
2021-02-16T16:08:47.945-0700 [INFO] auth.handler: starting auth handler
2021-02-16T16:08:47.945-0700 [INFO] auth.handler: authenticating
2021-02-16T16:08:47.945-0700 [TRACE] auth.cert: beginning authentication
2021-02-16T16:08:47.945-0700 [TRACE] auth.cert: deriving auth client to use
2021-02-16T16:08:47.986-0700 [ERROR] auth.handler: error authenticating: error="Error making API request.

URL: PUT https://active.vault.service.consul.domain.us:8200/v1/auth/cert/certs/test0/login
Code: 400. Errors:

client certificate must be supplied" backoff=2.428856411

datadude816 · 2021-02-16T23:15:08Z

Yes, correct the log level in the config file has no effect.

calvn · 2021-02-16T23:20:20Z

Thanks for the quick response. I'll have to circle back to this, but it looks like you are able to proceed by providing the TLS information in the vault stanza.

datadude816 · 2021-02-16T23:23:13Z

with testing maybe but with the security issue of using it in the vault stanza I can't go any further. because yes, I am planning on using cache I was just simplifying the config as far down as I could to figure out the issue.

calvn · 2021-02-16T23:40:12Z

I see. Thanks for bringing this to our attention!

calvn · 2021-02-16T23:55:19Z

One more question -- What does the config on the Vault server look like? I'm particularly interested on whether tls_require_and_verify_client_cert was set, but sharing the full config would give a clearer picture (minus any sensitive info).

datadude816 · 2021-02-17T00:06:01Z

vault.hcl (server)

api_addr = "https://10.10.1.142:8200" ui = true disable_mlock = true cluster_addr = "https://10.10.1.142:8201" listener "tcp" { tls_disable = 0 address = "0.0.0.0:8200" cluster_address = "0.0.0.0:8201" tls_cert_file = "/etc/pki/tls/certs/machine.pem" tls_key_file = "/etc/pki/tls/certs/machine.key" } storage "raft" { path = "/opt/vault/raft" node_id = "vbnnode2" retry_join { leader_api_addr = "https://10.10.1.141:8200" leader_ca_cert_file = "/etc/pki/ca-trust/source/anchors/CA.pem" leader_client_cert_file = "/etc/pki/tls/certs/machine.pem" leader_client_key_file = "/etc/pki/tls/certs/machine.key" } retry_join { leader_api_addr = "https://10.10.2.141:8200" leader_ca_cert_file = "/etc/pki/ca-trust/source/anchors/CA.pem" leader_client_cert_file = "/etc/pki/tls/certs/machine.pem" leader_client_key_file = "/etc/pki/tls/certs/machine.key" } retry_join { leader_api_addr = "https://10.10.2.142:8200" leader_ca_cert_file = "/etc/pki/ca-trust/source/anchors/CA.pem" leader_client_cert_file = "/etc/pki/tls/certs/machine.pem" leader_client_key_file = "/etc/pki/tls/certs/machine.key" } retry_join { leader_api_addr = "https://10.10.3.141:8200" leader_ca_cert_file = "/etc/pki/ca-trust/source/anchors/CA.pem" leader_client_cert_file = "/etc/pki/tls/certs/machine.pem" leader_client_key_file = "/etc/pki/tls/certs/machine.key" } } service_registration "consul" { address = "127.0.0.1:8500" }

calvn · 2021-05-12T23:27:06Z

@datadude816 the tag on #11576 resulted in this issue being automatically closed when it got merged. Can you give this another try with a build from master and see if the issue persists. Alternatively, you can wait until the Vault 1.7.2 release which should come out soon. Feel free to re-open if the PR doesn't address this issue!

HridoyRoy added auth/cert Authentication - certificates ecosystem agent pod/bridge labels Feb 9, 2021

calvn mentioned this issue May 11, 2021

agent/cert: properly return the cached client on AuthClient #11576

Merged

calvn closed this as completed in #11576 May 12, 2021

calvn added this to the 1.7.2 milestone May 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vault agent certificate authentication error #10835

Vault agent certificate authentication error #10835

datadude816 commented Feb 3, 2021

datadude816 commented Feb 16, 2021

calvn commented Feb 16, 2021

datadude816 commented Feb 16, 2021

calvn commented Feb 16, 2021

datadude816 commented Feb 16, 2021

calvn commented Feb 16, 2021

calvn commented Feb 16, 2021

datadude816 commented Feb 16, 2021

datadude816 commented Feb 16, 2021

calvn commented Feb 16, 2021

datadude816 commented Feb 16, 2021

datadude816 commented Feb 16, 2021

calvn commented Feb 16, 2021

datadude816 commented Feb 16, 2021

calvn commented Feb 16, 2021

calvn commented Feb 16, 2021

datadude816 commented Feb 17, 2021

calvn commented May 12, 2021

Vault agent certificate authentication error #10835

Vault agent certificate authentication error #10835

Comments

datadude816 commented Feb 3, 2021

datadude816 commented Feb 16, 2021

calvn commented Feb 16, 2021

datadude816 commented Feb 16, 2021

calvn commented Feb 16, 2021

datadude816 commented Feb 16, 2021

calvn commented Feb 16, 2021

calvn commented Feb 16, 2021

datadude816 commented Feb 16, 2021

datadude816 commented Feb 16, 2021

calvn commented Feb 16, 2021

datadude816 commented Feb 16, 2021

datadude816 commented Feb 16, 2021

calvn commented Feb 16, 2021

datadude816 commented Feb 16, 2021

calvn commented Feb 16, 2021

calvn commented Feb 16, 2021

datadude816 commented Feb 17, 2021

calvn commented May 12, 2021