vault login fail after unsealing with CLI.

[jjone36]

Describe the bug
After starting a vault server following this guide, I got the error while authenticating as the initial root token.

$ vault login
Token (will be hidden): 

Error authenticating: empty response from lookup-self

all the steps before the command was the same with the guide,
Even after unsealing and exporting the root token and the vault api address, I failed to the command vault login
And I couldn't find the .vault-token file which is automatically created when testing with the dev mode.
(It worked successfully when executing as dev server tho.)

To Reproduce
Steps to reproduce the behavior:

1. Follows the steps in this guide with the config file as below.
2. Run the command vault login

Expected behavior
A clear and concise description of what you expected to happen.

Environment:

• Vault Server Version (retrieve with vault status):

$ vault status
Key                    Value
---                    -----
Seal Type              shamir
Initialized            true
Sealed                 false
Total Shares           5
Threshold              3
Version                1.14.0
Build Date             2023-06-19T11:40:23Z
Storage Type           mysql
Cluster Name           vault-cluster-abeaa9a4
Cluster ID             cb145506-e1c3-7746-b1e8-6ec6fac6acf1
HA Enabled             true
HA Cluster             https://vault_server_fqdn:8201
HA Mode                standby
Active Node Address    http://localhost:8200

• Vault CLI Version (retrieve with vault version): 1.14.0
• Server Operating System/Architecture: linux

Vault server configuration file(s):

storage "mysql" {
  address = "mysql_server_fqdn:3306"
  database = "VAULT"
  username = "username"
  password = "passwod"
  ha_enabled = "true"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = "true"
}

disable_mlock = true

api_addr = "http://localhost:8200"
cluster_addr = "https://vault_server_fqdn:8201"
enable_response_header_hostname = true
enable_response_header_raft_node_id = true

ui = true
log_level="info"

Additional context
And also, I can't get the response with cluster address with the following error message.

curl -X 'GET' -H 'accept: application/json' -H 'X-Vault-Token: ' http://{vault_server_fqdn}:8201/v1/sys/health
curl: (52) Empty reply from server

One possible reason I'm thinking is because of https. (I'm not sure)
Is not available using http for a cluster server address?

Thank you for your help in advance.

[maxb]

Error authenticating: empty response from lookup-self

This indicates that the Vault client library has received an unexpectedly "empty" response from the Vault server - but it is not very good at passing on good diagnostics when this occurs. You should repeat the equivalent operation using curl to get better diagnostics:

curl -H "X-Vault-Token: TOKEN_VALUE_HERE" "$VAULT_ADDR/v1/auth/token/lookup-self"

---

---

curl -X 'GET' -H 'accept: application/json' -H 'X-Vault-Token: ' http://{vault_server_fqdn}:8201/v1/sys/health

This command is incorrect: you are attempting to send a Vault API request to port 8201 (the internal use only clustering port) instead of port 8200 (the API).

[jjone36]

@maxb thank you so much for your kindness.

I tried the command in your comment and got no respose.
And I checked the log after the request and here is the log I found.
Could you please give me any tips to fix this error?

$ curl -X 'GET' -H 'accept: application/json' -H 'X-Vault-Token: $ROOT_TOKEN'  http://$VAULT_ADDR:8200/v1/auth/token/lookup-self | jq
(no respose)
$ cat vault.log
2023-07-19T10:37:56.492+0900 [ERROR] core: error during forwarded RPC request: error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: remote error: tls: internal error\""
2023-07-19T10:37:56.492+0900 [ERROR] core: forward request error: error="error during forwarding RPC request"

This command is incorrect: you are attempting to send a Vault API request to port 8201 (the internal use only clustering port) instead of port 8200 (the API).

Thanks. Now I understood how HA works in Vault so this was a wrong request ;)

[maxb]

$ curl -X 'GET' -H 'accept: application/json' -H 'X-Vault-Token: $ROOT_TOKEN' http://$VAULT_ADDR:8200/v1/auth/token/lookup-self | jq
(no respose)

This use of $VAULT_ADDR looks wrong. The Vault CLI expects VAULT_ADDR to be set to a full URL. So in this case http://hostname:8200.

Including | jq on the end is potentially masking errors. Also you may want to add a verbose (-v) option to curl.

2023-07-19T10:37:56.492+0900 [ERROR] core: error during forwarded RPC request: error="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: remote error: tls: internal error""

This error is unexpected and mysterious. Something is going wrong when the standby node attempts to connect to the internal cluster port on the active node, to forward the request. It is not possible to conclude anything more detailed from the information shown so far. You would have to explain your HA setup in considerably greater detail, and it would probably be useful if you stopped obscuring the hostnames involved, as well.

[jjone36]

@maxb

After removing | jq on the end of the commend, I got the expected response.
And as adding -v option with curl got the proper result. Thanks a lot for your help. :)

Please let me ask you one more question about how to configure HA and vault agent servers.
If I installed two Vault nodes as described here, and if I set LB on these servers, what address should I use for VAULT_CLUSTER_ADDR to each server?
Would it be okay to use LB address for VAULT_CLUSTER_ADDR as below?

• A
  • VAULT_ADDR: https://a.vault.mycompany.com:8200
  • VAULT_CLUSTER_ADDR: https://lb.vault.mycompany.com:8201
• B
  • VAULT_ADDR: https://b.vault.mycompany.com:8200
  • VAULT_CLUSTER_ADDR: https://lb.vault.mycompany.com:8201

And If I install and run Vault agent, should I install it on the same servers where my Vault is running, which are A and B?

[maxb]

• VAULT_CLUSTER_ADDR: https://lb.vault.mycompany.com:8201

No, this would not be OK, because the purpose of the Vault cluster address is direct communication from one Vault node to another. Actually, this probably explains your tls: internal error error - your load balancer is not (and cannot) replicate the specialized TLS configuration used by the Vault port 8201 listener. You should use the individual node hostnames.

And If I install and run Vault agent, should I install it on the same servers where my Vault is running, which are A and B?

That completely depends on what you intend to use Vault agent for, but probably not, as there is usually no cause to use Vault agent on the same servers as Vault server.

[maxb]

Additionally, VAULT_CLUSTER_ADDR overrides cluster_addr in the Vault configuration file - you should pick one place to control this setting, and stick to it, rather than have something in your configuration file which is actually overridden at runtime by an environment variable.

[stevendpclark]

Thanks @maxb for answering @jjone36's questions.

@jjone36 I'll close out this issue as I believe there is no bug here for us to address. Please feel free to re-open if I am incorrect.