connection refused to Consul UI unless -client option is public address.

[petemounce]

I am running consul 0.4.1 in EC2, on Amazon Linux ami-607bd917 (eu-west-1, t2.micro, naked; no yum updates), and I cannot reach the consul UI externally from the instance. My cluster bootstraps fine.

$ curl http://10.10.10.98:8500/ui/dist/
curl: (7) Failed to connect to 10.10.10.98 port 8500: Connection refused

$ curl http://127.0.0.1:8500/ui/dist/
# correct HTML response

I am using an upstart job like:

description "Consul agent"

start on runlevel [2345]
stop on runlevel [!2345]

respawn

script
  # Make sure to use all our CPUs, because Consul can block a scheduler thread
  export GOMAXPROCS=`nproc`

  # Get the public IP
  BIND=`ifconfig eth0 | grep "inet addr" | awk '{ print substr($2,6) }'`

  export CLUSTER_SIZE=$(aws autoscaling describe-auto-scaling-groups | jq -c -M -r '.AutoScalingGroups[] | {name: .AutoScalingGroupName, desired_capacity: .DesiredCapacity}' | grep je-consul-qa13-all.*ConsulCluster | jq -c -M -r '.desired_capacity' | tr -d "\n")
  exec /usr/local/bin/consul agent \
    -bootstrap-expect $CLUSTER_SIZE \
    -config-dir="/etc/consul.d" \
    -bind=$BIND \
    >>/var/log/consul.log 2>&1
end script

and /etc/consul.d/consul.json as

{
"server": true,
"rejoin_after_leave": true,
"enable_syslog": true,
"syslog_facility": "local5",
"data_dir": "/var/consul/data",
"ui_dir": "/var/consul/ui",
"datacenter": "eu-west-1",
"statsd_addr": "<some IP>:8125",
"recursor": "<some IP>"
}

$ netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State
tcp        0      0 10.10.10.98:tmi             *:*                         LISTEN
tcp        0      0 10.10.10.98:amberon         *:*                         LISTEN
tcp        0      0 10.10.10.98:8302            *:*                         LISTEN
tcp        0      0 localhost:cvd               *:*                         LISTEN
tcp        0      0 localhost:fmtp              *:*                         LISTEN
tcp        0      0 *:ssh                       *:*                         LISTEN
tcp        0      0 localhost:asterix           *:*                         LISTEN
tcp        0      0 10.10.10.98:amberon         10.233.2.251:35382          TIME_WAIT
tcp        0      0 10.10.10.98:39957           10.233.2.251:amberon        TIME_WAIT
tcp        0      0 10.10.10.98:tmi             10.233.2.251:40657          ESTABLISHED
tcp        0      0 10.10.10.98:amberon         10.233.0.36:56857           TIME_WAIT
tcp        0    512 10.10.10.98:ssh             10.10.30.248:63183          ESTABLISHED
tcp        0      0 10.10.10.98:tmi             10.233.2.251:40654          ESTABLISHED
tcp        0      0 10.10.10.98:57182           10.233.2.251:tmi            ESTABLISHED
tcp        0      0 *:ssh                       *:*                         LISTEN
udp        0      0 localhost:asterix           *:*
udp        0      0 10.10.10.98:51700           10.233.0.10:8125            ESTABLISHED
udp        0      0 *:bootpc                    *:*
udp        0      0 10.10.10.98:amberon         *:*
udp        0      0 10.10.10.98:8302            *:*
udp        0      0 10.10.10.98:ntp             *:*
udp        0      0 localhost:ntp               *:*
udp        0      0 *:ntp                       *:*
udp        0      0 fe80::403:8eff:fe95:36c:ntp *:*
udp        0      0 localhost:ntp               *:*
udp        0      0 *:ntp                       *:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path
unix  2      [ ACC ]     STREAM     LISTENING     10585  /tmp/tmux-500/default
unix  7      [ ]         DGRAM                    8720   /dev/log
unix  2      [ ACC ]     STREAM     LISTENING     8799   /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     7627   @/com/ubuntu/upstart
unix  2      [ ACC ]     SEQPACKET  LISTENING     7832   @/org/kernel/udev/udevd
unix  3      [ ]         STREAM     CONNECTED     10578
unix  2      [ ]         DGRAM                    11277
unix  2      [ ]         DGRAM                    10519
unix  3      [ ]         STREAM     CONNECTED     10579
unix  3      [ ]         DGRAM                    7841
unix  3      [ ]         STREAM     CONNECTED     10584
unix  3      [ ]         STREAM     CONNECTED     8803
unix  3      [ ]         STREAM     CONNECTED     8802
unix  3      [ ]         STREAM     CONNECTED     8807
unix  3      [ ]         STREAM     CONNECTED     8808   /var/run/dbus/system_bus_socket
unix  2      [ ]         DGRAM                    10217
unix  3      [ ]         STREAM     CONNECTED     10523
unix  3      [ ]         STREAM     CONNECTED     10522
unix  3      [ ]         DGRAM                    7840
unix  3      [ ]         STREAM     CONNECTED     10581
unix  3      [ ]         STREAM     CONNECTED     10582
unix  2      [ ]         DGRAM                    9444
unix  2      [ ]         DGRAM                    9739
unix  3      [ ]         STREAM     CONNECTED     10583

I can work around this by, in my upstart, invoking consul agent .... -client=$BIND ... but then that forces me to specify the -rpc-addr=<external IP> on each consul command (in the server cluster, anyhow) thereafter; rather not do that since it complicates things a bit.

I can also work around this by running another consul cluster of agents whose only job is to join the cluster and host the UI - but I'd rather not do that, since it seems like it shouldn't be necessary and will cost more in terms of $ and management.

Is there a way to make this work without my workarounds?

[armon]

You need to configure the HTTP address to listen on a public IP. By default it uses loopback. The -client flag does this, but also changes the IP of all the listeners (RPC, HTTP, DNS). You can do more fine grained configuration with a configuration file however.

[petemounce]

That works perfectly; thanks!

[saulshanabrook]

@petemounce Could you share exactly what you did to make it work? I am also having the same problem.

[petemounce]

@saulshanabrook I used the configuration file to set the http interface to listen on the NIC instead of the default loopback address.

...
  "addresses" : {
    "http": "10.10.10.257" // obviously made up ;)
  }
...

Here's the consul agent upstart task from my cloudformation:

"/etc/init/consul.conf": {
  "content": {
    "Fn::Join": [
      "",
      [
        "description \"Consul agent\"\n",
        "\n",
        "start on runlevel [2345]\n",
        "stop on runlevel [!2345]\n",
        "\n",
        "respawn\n",
        "\n",
        "script\n",
        "  # Make sure to use all our CPUs, because Consul can block a scheduler thread\n",
        "  export GOMAXPROCS=`nproc`\n",
        "\n",
        "  # Get the public IP\n",
        "  BIND=`ifconfig eth0 | grep \"inet addr\" | awk '{ print substr($2,6) }'`\n",
        "\n",
        "  export CLUSTER_SIZE=$(aws autoscaling describe-auto-scaling-groups | jq -c -M -r '.AutoScalingGroups[] | {name: .AutoScalingGroupName, desired_capacity: .DesiredCapacity}' | grep je-",
        {
          "Ref": "FeatureName"
        },
        "-",
        {
          "Ref": "EnvironmentName"
        },
        "-",
        {
          "Ref": "Tenant"
        },
        ".*ConsulCluster | jq -c -M -r '.desired_capacity' | tr -d \"\\n\")\n",
        "  exec /usr/local/bin/consul agent \\\n",
        "    -bootstrap-expect $CLUSTER_SIZE \\\n",
        "    -config-dir=\"/etc/consul.d\" \\\n",
        "    -bind=$BIND \\\n",
        "    >>/var/log/consul.log 2>&1\n",
        "end script\n",
        "\n"
      ]
    ]
  }
}

...

"022_configure_http_api_interface": {
  "command": "BIND=`ifconfig eth0 | grep \"inet addr\" | awk '{ print substr($2,6) }'` && sed -i \"s/http\\\": \\\"\\\"/http\\\": \\\"${BIND}\\\"/\" /etc/consul.d/consul.json"
},

[saulshanabrook]

@petemounce thank you!

Would $BIND be 10.10.10.257 as well in this case?

With it set up like that I still get this:

[centos@ip-10-128-1-235 ~]$ curl 10.128.1.235:8500/v1/catalog/nodes
curl: (7) Failed connect to 10.128.1.235:8500; Connection refused

But works with localhost

[centos@ip-10-128-1-235 ~]$ curl localhost:8500/v1/catalog/nodes
[{"Node":"10.128.1.115","Address":"10.128.1.115"},{"Node":"10.128.1.194","Address":"10.128.1.194"},{"Node":"10.128.1.198","Address":"10.128.1.198"},{"Node":"10.128.1.218","Address":"10.128.1.218"},{"Node":"10.128.1.235","Address":"10.128.1.235"},{"Node":"10.128.1.33","Address":"10.128.1.33"},{"Node":"10.128.1.57","Address":"10.128.1.57"},{"Node":"10.128.1.73","Address":"10.128.1.73"},{"Node":"10.128.1.80","Address":"10.128.1.80"}]

This is my config:

[centos@ip-10-128-1-235 ~]$ sudo cat /etc/consul/consul.json
{
  "datacenter": "tag_Service_controller",
  "addresses" : {
    "http": "10.128.1.235"
  },
  "bind_addr": "10.128.1.235",
  "node_name": "10.128.1.235",
  "rejoin_after_leave": true,
  "domain": "consul",
  "retry_join": [ "10.128.1.80", "10.128.1.235", "10.128.1.57" ],
  "server": true,
  "bootstrap_expect": 3,
  "verify_incoming": false,
  "verify_outgoing": false,
  "data_dir": "/var/lib/consul",
  "ui_dir": "/usr/share/consul-ui",
  "disable_remote_exec": false
}

[petemounce]

Yes, think so. My pleasure :-)

Sent from my phone. Please excuse typos and brevity, but never text speak.
On 11 May 2015 23:04, "Saul Shanabrook" notifications@github.com wrote:

@petemounce https://github.com/petemounce thank you! Would $BIND be
"10.10.10.257" as well in this case?

—
Reply to this email directly or view it on GitHub
#599 (comment).

[liaokailin]

[root@localhost ~]# consul agent -h | grep client
-client=127.0.0.1 Sets the address to bind for client access.

we can see : the default ip is 127.0.0.1 ,modify this val change client visit ip

eg:

consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul/ -ui-dir /root/software/consul-ui/ -client=192.168.23.154

[ivasilyev-servicetitan-com]

Hello.

I'm trying to solve similar task.
I need to connect to consul agent outside the machine using HTTP Api interface (port 8500).
The difference is - I want to connect to consul agent running in client mode, not server mode.

Is it possible?

Cause I tried different options with my consul.json config.
It's
{
"advertise_addr": "10.3.0.1",
"bind_addr": "10.3.0.1",
"addresses" : { "http": "10.3.0.1 0.0.0.0" },
"service": {"name": "myService"},
"client_addr": "0.0.0.0",
"datacenter": "westus",
"node_id": "51f55f0e-f215-44c6-95c6-3e776b9f7f0b",
"retry_join": ["10.2.10.19", "10.2.10.20", "10.2.10.21"],
"server": false,
"ui": true,
"log_level": "info",
"raft_protocol": 3,
"ports": { "dns": 53 },
"recursors": ["168.63.129.16"],
"telemetry": {
"dogstatsd_addr": "127.0.0.1:8125"
}
}
And it doesn't connect outside.
It succesfully opens http://10.3.0.1:8500/v1/kv/myKey
and http://127.0.0.1:8500/v1/kv/myKey from 10.3.0.1 machine.

But outside the machine http://10.3.0.1:8500/v1/kv/myKey can be opened.

I checked I can open http://10.2.10.19:8500/v1/kv/myKey (which consul server ip).

So is it possible to connect to consul client agent using HTTP API?