node_meta keys are not available on Consul server nodes

[shlem952]

Overview of the Issue

We are trying to add the node_meta key to the servers in an existing cluster, and after a reboot / restart of the process it is unavailable (.Node.Meta = null).
But if we launch a new cluster with a -node-meta key, the keys are available.

Reproduction Steps

1. Run consul cluster 3+ servers node
   Configuration file consul<id>.json

{
    "acl": {
      "default_policy": "deny",
      "down_policy": "extend-cache",
      "enabled": true
    },
    "addresses": {
      "dns": "0.0.0.0",
      "http": "0.0.0.0"
    },
    "data_dir": "/var/lib/consul",
    "datacenter": "test-dc",
    "disable_anonymous_signature": true,
    "disable_host_node_id": true,
    "disable_remote_exec": true,
    "disable_update_check": true,
    "dns_config": {
      "service_ttl": {
        "*": "15s"
      },
      "soa": {
        "min_ttl": 30
      }
    },
    "domain": "test-domain",
    "encrypt": "my_secret_key",
    "gossip_lan": {
      "suspicion_mult": 6
    },
    "log_file": "/var/log/consul/consul-sd.log",
    "log_rotate_max_files": 30,
    "node_name": "consul<id>",
    "ports": {
      "dns": 8600,
      "http": 8500
    },
    "rejoin_after_leave": true,
    "server": true,
    "start_join": [
      "consul01",
      "consul02",
      "consul03"
    ],
    "ui": true,
    "verify_incoming": false,
    "verify_outgoing": false
  }

docker-compose.yaml

version: '3'
services:
  consul01: &consul-server
    image: consul:1.7.2
    hostname: consul01
    networks:
      - consul-demo
    volumes:
      - <work_dir>/config/consul01.json:/consul/config/consul01.json
    command: "agent --config-dir /consul/config"
  consul02:
    <<: *consul-server
    hostname: consul02
    volumes:
      - <work_dir>/config/consul02.json:/consul/config/consul02.json
  consul03:
    <<: *consul-server
    hostname: consul03
    volumes:
      - <work_dir>/config/consul03.json:/consul/config/consul03.json
    ports:
      - "8400:8400"
      - "8500:8500"
      - "8600:8600"
      - "8600:8600/udp"
    command: "agent -bootstrap --config-dir /consul/config"
networks:
  consul-demo:

2. Bootstrap ACL & check that the node meta is missing

consul acl bootstrap
curl -s --header "X-Consul-Token: <token>" 127.0.0.1:8500/v1/catalog/nodes | jq

3. Add node-meta for existing process
   Configuration file consul<id>.json

...
    "node_meta": {
        "key": "value"
    },
...

4. Reload process

/bin/kill -HUP <process id>

5. check if node-meta appeared

curl -s --header "X-Consul-Token: <token>" 127.0.0.1:8500/v1/catalog/nodes | jq

Operating system and Environment details

OS: Debian Stretch
Deployment: 5 server nodes and ~ 800 client nodes
Version: 1.7.2

[idrennanvmware]

@shlem952 it looks like your node_meta is missing [ and ]

here's what ours looks like

"node_meta": [{ "consul_version": "1.8.0","some_value":"hi","more_meta":"stuffhere" }]

[bofm]

@idrennanvmware do you mean you have node_meta of an array type in your /etc/consul.d/<something>.<json|hcl> and it works fine?

[idrennanvmware]

Correct. This is what our json snippet looked like

{
   <SNIP>,
   "node_meta":[
      {
         "key1":"value1",
         "key2":"value2",
         "key3":"value3"
      }
   ]
}

and this is what our HCL looks like (we recently moved over)

node_meta = [{ "key1"="value1","key2"="value2","key3"="value3" }]

[bofm]

Will try the variant with an object inside a list. But this is odd. The examples in the docs show that node_meta must be an object type. Moreover, node_meta of an object type works well on Consul agents (but it doesn't on servers).

[shlem952]

@idrennanvmware we tried to do as you suggest, but it still does not work.

[idrennanvmware]

Just getting back to this - sorry for the delay - I really am confused why this isn't working for you. Our clients and servers have the same configuration around this. I'm going to post a chunk of our config (same on client and servers) and see if you can spot a difference.

This one is in JSON format

{
   "enable_central_service_config":true,
   "acl":{
      "enabled":true,
      "default_policy":"deny",
      "tokens":{
         "agent":"snip"
      },
      "enable_token_persistence":true,
      "down_policy":"extend-cache"
   },
   "encrypt":"snip",
   "encrypt_verify_incoming":true,
   "encrypt_verify_outgoing":true,
   "telemetry":{
      "dogstatsd_addr":"127.0.0.1:8125",
      "disable_hostname":true
   },
   "data_dir":"/opt/consul",
   "client_addr":"0.0.0.0",
   "datacenter":"snip",
   "primary_datacenter":"snip",
   "bind_addr":"snip",
   "bootstrap_expect":3,
   "ui":true,
   "server":true,
   "retry_join":[
      "snip",
      "snip"
   ],
   "enable_local_script_checks":true,
   "log_file":"/opt/consul/consul.log",
   "log_level":"INFO",
   "log_rotate_max_files":5,
   "log_rotate_bytes":10000000,
   "disable_update_check":true,
   "dns_config":{
      "service_ttl":{
         "*":"5s"
      }
   },
   "recursors":[
      "8.8.8.8"
   ],
   "ports":{
      "grpc":8502
   },
   "connect":{
      "enabled":true
   },
   "node_meta":[
      {
         "key0":"true",
         "key1":"true",
         "key2":"true",
         "key3":"true",
         "key4":"true",
         "key5":"true",
         "key6":"true",
         "key7":"true"
      }
   ]
}

[neurostream]

I can duplicate this issue if I'm bootstrapping with the /v1/acl/bootstrap interface to get my initial master token (SecretID returned by the first and only request to that API endpoint when initializing the cluster ) - even if enable_token_persistence=true.

( @shlem952 : your config looks like mine when I encounter this problem, in that I don't set tokens in the config file - but, instead, use the bootstrap API ...is that what you're doing as well? )

Even if I supply the master token returned by /v1/acl/bootstrap, the node_meta query still comes back blank. I think setting the master token in the node's agent configuration also does some other magic setup that has to be done separately if using the bootstrap API - such as setting up some default agent permission behaviors and maybe even a restart of the agent ( not sure a "consul reload" works).

With this bootstrap API, node agent's acl.tokens.default might be undefined and may need additional policies ( like node "read" permission, at least) to read node_meta values. If I set acl.tokens.default to use a token with enough permissions, it works.

I really love the bootstrap API for automation ( especially if calling it from a Vault install in order to swallow the Consul master token into Vault upon init - thus initializing both Vault and Consul securely at the same time); so I think I just need to learn the additional steps required to run the bootstrap and create a policy and token for the agent's acl.tokens.default.