0.11.5 vault fails to load and merge storage and listener hcl config files correctly

[eedwards-sk]

Firstly, it appears that config override doesn't actually allow you to merge sub keys.

e.g. if I have default.hcl with:

foo "bar" {
  baz = "alpha"
}

and I create an override.hcl file with:

foo "bar" {
  bar = "beta"
}

it appears to actually replace the entire value of foo "bar" rather than merge into the object

e.g. the expected result from above would be:

foo "bar" {
  baz = "alpha"
  bar = "beta"
}

instead, the actual result is:

foo "bar" {
  bar = "beta"
}

this is relevant given the below issue

---

Given

(using a setup created mostly from hashicorp's terraform-aws-vault project)

/opt/vault/config:

-rw-r--r-- 1 vault vault  473 Nov 28 00:19 default.hcl
-rw-r--r-- 1 vault vault   49 Nov 27 23:28 override-dogstatsd.hcl
-rw-r--r-- 1 vault vault  337 Nov 28 00:16 override-listener.hcl
-rw-r--r-- 1 vault vault  316 Nov 28 00:17 override-storage.hcl

default.hcl:

ui = true

listener "tcp" {
  address         = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_cert_file   = "/opt/snapkitchen/vault-pki/server-fullchain.pem"
  tls_key_file    = "/opt/snapkitchen/vault-pki/server-key.pem"
}


storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault/"
  scheme  = "http"
  service = "vault"
}

# HA settings
cluster_addr  = "https://172.31.66.26:8201"
api_addr      = "https://172.31.66.26:8200"

override-dogstatsd.hcl:

telemetry {
 dogstatsd_addr = "localhost:8125"
}

override-listener.hcl:
(note how I have to repeat the contents of the listener config from default.hcl file rather than just adding what I want to merge in)

listener "tcp" {
  address         = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_cert_file   = "/opt/snapkitchen/vault-pki/server-fullchain.pem"
  tls_key_file    = "/opt/snapkitchen/vault-pki/server-key.pem"
  tls_require_and_verify_client_cert = "true"
  tls_client_ca_file = "/opt/snapkitchen/vault-pki/ca/ca-chain.pem"
}

override-storage.hcl:
(note how I have to repeat the contents of the storage config from default.hcl file rather than just adding what I want to merge in)

storage "consul" {
  address = "127.0.0.1:8501"
  path    = "vault/"
  scheme  = "https"
  service = "vault"
  tls_ca_file    = "/opt/snapkitchen/consul-vault-pki/ca/ca-chain.pem"
  tls_cert_file  = "/opt/snapkitchen/consul-vault-pki/vault.pem"
  tls_key_file   = "/opt/snapkitchen/consul-vault-pki/vault-key.pem"
}

Expected Results

vault runs and connects to backend using specified overridden configuration

Actual Results

vault runs and fails to connect to backend as it uses the values from default.hcl and does not override them per the values in override-storage.hcl:

WARNING! Unable to read migration status.
2018-11-28T00:34:52.122Z [DEBUG] storage.consul: config path set: path=vault/
2018-11-28T00:34:52.123Z [DEBUG] storage.consul: config disable_registration set: disable_registration=false
2018-11-28T00:34:52.123Z [DEBUG] storage.consul: config service set: service=vault
2018-11-28T00:34:52.123Z [DEBUG] storage.consul: config service_tags set: service_tags=
2018-11-28T00:34:52.123Z [DEBUG] storage.consul: config service_address set: service_address=<nil>
2018-11-28T00:34:52.123Z [DEBUG] storage.consul: config address set: address=127.0.0.1:8500
2018-11-28T00:34:52.123Z [DEBUG] storage.consul: config scheme set: scheme=http
2018-11-28T00:34:52.124Z [WARN]  migration check error: error="Get http://127.0.0.1:8500/v1/kv/vault/core/migration: dial tcp 127.0.0.1:8500: connect: connection refused"

Further Problems

If I manually remove the storage configuration section from default.hcl, then it will fail to run at all:

Error initializing listener of type tcp: listen tcp4 0.0.0.0:8200: bind: address already in use

if I ALSO manually remove the listener config from default.hcl, then it finally runs, but it says that api_addr is unset, even though it's explicitly set in default.hcl as above

==> Vault server configuration:

             Api Address: https://172.31.66.26:8200
                     Cgo: disabled
         Cluster Address: https://172.31.66.26:8201
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
               Log Level: debug
                   Mlock: supported: true, enabled: true
                 Storage: consul (HA available)
                 Version: Vault v0.11.5
             Version Sha: a59ffa4a0f09bbf198241fe6793a96722789b639

==> Vault server started! Log data will stream in below:

2018-11-28T00:34:05.970Z [DEBUG] storage.consul: config path set: path=vault/
2018-11-28T00:34:05.970Z [DEBUG] storage.consul: config disable_registration set: disable_registration=false
2018-11-28T00:34:05.970Z [DEBUG] storage.consul: config service set: service=vault
2018-11-28T00:34:05.970Z [DEBUG] storage.consul: config service_tags set: service_tags=
2018-11-28T00:34:05.970Z [DEBUG] storage.consul: config service_address set: service_address=<nil>
2018-11-28T00:34:05.970Z [DEBUG] storage.consul: config address set: address=127.0.0.1:8501
2018-11-28T00:34:05.970Z [DEBUG] storage.consul: config scheme set: scheme=https
2018-11-28T00:34:05.970Z [DEBUG] storage.consul: configured TLS
2018-11-28T00:34:05.982Z [WARN]  no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set
2018-11-28T00:34:05.984Z [DEBUG] storage.cache: creating LRU cache: size=0
2018-11-28T00:34:06.012Z [DEBUG] cluster listener addresses synthesized: cluster_addresses=[0.0.0.0:8201]
^C==> Vault shutdown triggered
2018-11-28T00:34:07.071Z [DEBUG] core: shutdown called
2018-11-28T00:34:07.071Z [INFO]  storage.consul: shutting down consul backend

Sorry if this is confusing, but basically configuration files don't seem to load properly at all right now.

[danieljgraves]

I'm also experiencing this issue, and I can't find any posts anywhere to explain how i'm supposed to supply a replacement backend without causing a merge conflict when Vault scans the config directory and doesn't ignore the default storage. My use case is that i'm trying to use S3 as my backend, so I have created a storage-backend.hcl file that just has my S3 configuration, that's it. But I have found through trial and error that before I start vault, I have to manually go to the default.hcl file and delete the default storage stanza section completely, and THEN start Vault, which will grab the remaining default.hcl configs and then move onto my storage-backend.hcl file to load the storage backend successfully. I wish Vault would just ignore the entire stanza that's being reproduced in override files.

[danieljgraves]

What i've done in the mean time to get the expected behavior is INSTEAD of calling the entire configuration /opt/vautl/config/ and load the default.hcl file and my storage-backend.hcl file, i'm only running this command: /usr/local/bin/vault server -config=/opt/vault/config/storage-backend.hcl and Vault will launch but complain about missing some "default" configurations, but it will generate what it think's it is missing. This has allowed me to let Vault fill in the "dynamic" pieces that I need, while i'm defining the rest of my parameters in my storage-backend.hcl file. I hope this helps others.

This is what my "storage-backend.hcl" file looks like now:

ui = true
storage "s3" {
  bucket     = "abc123vaultdemo"
}

listener "tcp" {
  address         = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_cert_file   = "/opt/vault/tls/vault.crt.pem"
  tls_key_file    = "/opt/vault/tls/vault.key.pem"
  tls_disable_client_certs = true
}

[ahaynssen]

Saw the version tag on issue, @catsby I seem to still have this same issue still in 1.7.1

[hsimon-hashicorp]

Hi folks, this issue has been open for quite some time. Is this still a valid issue in newer versions of Vault? As of today, the current version is 1.16. Thanks!