Environment fingerprinters allow bogus values

[jcalonso]

Nomad version

1.0.1

Operating system and Environment details

Ubuntu 18.04 baremetal (non cloud)

Issue

When the environment fingerprinters try to obtain the network details, it is possible to end up with some bogus values as described in:

https://discuss.hashicorp.com/t/network-fingerprinting-calling-my-home-router/19023

In this specific case the values were populated with a html page from the home router probably because it was retuning a 200 status code on non existing pages and confusing the client thinking it had a successful request and setting the body of the request as the value.

Reproduction steps

• Run Nomad in a non cloud environment with the env_fingerprinters on (default).
• A router address that return 200 on any request with some unwanted body
• Logs and UI will show the bogus body printed out.

Nomad Client logs (if appropriate)

Dec 18 09:03:45 nomadnode01 systemd[1]: Started Nomad Client.
Dec 18 09:03:45 nomadnode01 nomad[6979]: ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation.
Dec 18 09:03:45 nomadnode01 nomad[6979]: ==> Loaded configuration from /etc/nomad.d/client.hcl, /etc/nomad.d/server.hcl
Dec 18 09:03:45 nomadnode01 nomad[6979]: ==> Starting Nomad agent...
Dec 18 09:03:46 nomadnode01 nomad[6979]: ==> Nomad agent configuration:
Dec 18 09:03:46 nomadnode01 nomad[6979]:        Advertise Addrs: HTTP: 192.168.1.11:4646; RPC: 192.168.1.11:4647; Serf: 192.168.1.11:4648
Dec 18 09:03:46 nomadnode01 nomad[6979]:             Bind Addrs: HTTP: 192.168.1.11:4646; RPC: 192.168.1.11:4647; Serf: 192.168.1.11:4648
Dec 18 09:03:46 nomadnode01 nomad[6979]:                 Client: true
Dec 18 09:03:46 nomadnode01 nomad[6979]:              Log Level: DEBUG
Dec 18 09:03:46 nomadnode01 nomad[6979]:                 Region: global (DC: dc01)
Dec 18 09:03:46 nomadnode01 nomad[6979]:                 Server: true
Dec 18 09:03:46 nomadnode01 nomad[6979]:                Version: 1.0.1
Dec 18 09:03:46 nomadnode01 nomad[6979]: ==> Nomad agent started! Log data will stream in below:
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.775Z [WARN]  agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/var/lib/nomad/storage/server/plugins
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.776Z [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/var/lib/nomad/storage/server/plugins
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.776Z [INFO]  agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.776Z [INFO]  agent: detected plugin: name=java type=driver plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.776Z [INFO]  agent: detected plugin: name=docker type=driver plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.776Z [INFO]  agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.776Z [INFO]  agent: detected plugin: name=exec type=driver plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.776Z [INFO]  agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.794Z [INFO]  nomad.raft: restored from snapshot: id=7-8351-1608257027253
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.811Z [INFO]  nomad.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:192.168.1.11:4647 Address:192.168.1.11:4647}]"
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:46.469Z [INFO]  client.gc: marking allocation for GC: alloc_id=5042c56f-fa29-467d-e6ad-7be5e42e4a07
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.811Z [INFO]  nomad.raft: entering follower state: follower="Node at 192.168.1.11:4647 [Follower]" leader=
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.811Z [INFO]  nomad: serf: EventMemberJoin: client.global 192.168.1.11
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.811Z [INFO]  nomad: starting scheduling worker(s): num_workers=4 schedulers=[service, batch, system, _core]
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.811Z [WARN]  nomad: serf: Failed to re-join any previously known node
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.811Z [INFO]  client: using state directory: state_dir=/var/lib/nomad/storage/server/client
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.812Z [INFO]  nomad: adding server: server="client.global (Addr: 192.168.1.11:4647) (DC: dc01)"
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.812Z [INFO]  client: using alloc directory: alloc_dir=/var/lib/nomad/storage/server/alloc
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.813Z [DEBUG] client.fingerprint_mgr: built-in fingerprints: fingerprinters=[arch, bridge, cgroup, cni, consul, cpu, host, memory, network, nomad, signal, storage, vault, env_azure, env_aws, env_gce]
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.813Z [INFO]  client.fingerprint_mgr.cgroup: cgroups are available
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.813Z [DEBUG] client.fingerprint_mgr: CNI config dir is not set or does not exist, skipping: cni_config_dir=
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.813Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=cgroup period=15s
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.815Z [INFO]  client.fingerprint_mgr.consul: consul agent is available
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.815Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=consul period=15s
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.815Z [DEBUG] client.fingerprint_mgr.cpu: detected cpu frequency: MHz=3900
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.815Z [DEBUG] client.fingerprint_mgr.cpu: detected core count: cores=4
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.817Z [DEBUG] client.fingerprint_mgr.network: link speed detected: interface=enp0s31f6 mbits=100
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.817Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=enp0s31f6 IP=192.168.1.11
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.817Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=enp0s31f6 IP=2806:102e:8:2e65::9
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.817Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=enp0s31f6 IP=2806:102e:8:2e65:e2d5:5eff:fe00:8c97
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.817Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=enp0s31f6 IP=fdac:6175:1a7e:dd00:e2d5:5eff:fe00:8c97
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.818Z [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/sbin/ethtool device=lo
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.818Z [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/lo/speed
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.818Z [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: mbits=1000
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.819Z [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/sbin/ethtool device=docker0
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.819Z [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/docker0/speed
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.819Z [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: mbits=1000
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.823Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=vault period=15s
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.858Z [DEBUG] consul.sync: sync complete: registered_services=3 deregistered_services=0 registered_checks=3 deregistered_checks=0
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:46.025Z [DEBUG] client.fingerprint_mgr.env_aws: read an empty value: attribute="network/interfaces/macs/<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
Dec 18 09:03:46 nomadnode01 nomad[6979]: <html xmlns="http://www.w3.org/1999/xhtml">
Dec 18 09:03:46 nomadnode01 nomad[6979]: <head>
Dec 18 09:03:46 nomadnode01 nomad[6979]: <meta content="text/html; charset=utf-8" http-equiv="Content-Type" />
Dec 18 09:03:46 nomadnode01 nomad[6979]: <meta content="no-cache" http-equiv="Pragma" />
Dec 18 09:03:46 nomadnode01 nomad[6979]: <title>Waiting...</title>
Dec 18 09:03:46 nomadnode01 nomad[6979]: <script type="text/javascript">
Dec 18 09:03:46 nomadnode01 nomad[6979]: var pageName = '/';
Dec 18 09:03:46 nomadnode01 nomad[6979]: top.location.replace(pageName);
Dec 18 09:03:46 nomadnode01 nomad[6979]: </script>
Dec 18 09:03:46 nomadnode01 nomad[6979]: </head>
Dec 18 09:03:46 nomadnode01 nomad[6979]: <body> </body>
Dec 18 09:03:46 nomadnode01 nomad[6979]: </html>/ipv6s"

[tgross]

Thanks for opening this @jcalonso. Dropping the workaround from Discuss here for reference:

options = {
  "fingerprint.denylist" = "env_aws,env_gce,env_azure"
 }

[superboum]

Just a message to say that this strange behavior affected us. We were using the ${attr.unique.network.ip-address} value to configure our PostgreSQL cluster with Stolon, and of course, it failed majestically to start it when Nomad passed the HTML of the router instead of a valid IP address.

For the context, our setup could be categorized as an "homelab setup" on a consumer network, so I am aware that we are not the main target of Hashicorp.

I applied the fix proposed above and it worked for us.