Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

connection refused to Consul UI unless -client option is public address. #599

Closed
petemounce opened this issue Jan 13, 2015 · 8 comments
Closed

Comments

@petemounce
Copy link

I am running consul 0.4.1 in EC2, on Amazon Linux ami-607bd917 (eu-west-1, t2.micro, naked; no yum updates), and I cannot reach the consul UI externally from the instance. My cluster bootstraps fine.

$ curl http://10.10.10.98:8500/ui/dist/
curl: (7) Failed to connect to 10.10.10.98 port 8500: Connection refused

$ curl http://127.0.0.1:8500/ui/dist/
# correct HTML response

I am using an upstart job like:

description "Consul agent"

start on runlevel [2345]
stop on runlevel [!2345]

respawn

script
  # Make sure to use all our CPUs, because Consul can block a scheduler thread
  export GOMAXPROCS=`nproc`

  # Get the public IP
  BIND=`ifconfig eth0 | grep "inet addr" | awk '{ print substr($2,6) }'`

  export CLUSTER_SIZE=$(aws autoscaling describe-auto-scaling-groups | jq -c -M -r '.AutoScalingGroups[] | {name: .AutoScalingGroupName, desired_capacity: .DesiredCapacity}' | grep je-consul-qa13-all.*ConsulCluster | jq -c -M -r '.desired_capacity' | tr -d "\n")
  exec /usr/local/bin/consul agent \
    -bootstrap-expect $CLUSTER_SIZE \
    -config-dir="/etc/consul.d" \
    -bind=$BIND \
    >>/var/log/consul.log 2>&1
end script

and /etc/consul.d/consul.json as

{
"server": true,
"rejoin_after_leave": true,
"enable_syslog": true,
"syslog_facility": "local5",
"data_dir": "/var/consul/data",
"ui_dir": "/var/consul/ui",
"datacenter": "eu-west-1",
"statsd_addr": "<some IP>:8125",
"recursor": "<some IP>"
}
$ netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State
tcp        0      0 10.10.10.98:tmi             *:*                         LISTEN
tcp        0      0 10.10.10.98:amberon         *:*                         LISTEN
tcp        0      0 10.10.10.98:8302            *:*                         LISTEN
tcp        0      0 localhost:cvd               *:*                         LISTEN
tcp        0      0 localhost:fmtp              *:*                         LISTEN
tcp        0      0 *:ssh                       *:*                         LISTEN
tcp        0      0 localhost:asterix           *:*                         LISTEN
tcp        0      0 10.10.10.98:amberon         10.233.2.251:35382          TIME_WAIT
tcp        0      0 10.10.10.98:39957           10.233.2.251:amberon        TIME_WAIT
tcp        0      0 10.10.10.98:tmi             10.233.2.251:40657          ESTABLISHED
tcp        0      0 10.10.10.98:amberon         10.233.0.36:56857           TIME_WAIT
tcp        0    512 10.10.10.98:ssh             10.10.30.248:63183          ESTABLISHED
tcp        0      0 10.10.10.98:tmi             10.233.2.251:40654          ESTABLISHED
tcp        0      0 10.10.10.98:57182           10.233.2.251:tmi            ESTABLISHED
tcp        0      0 *:ssh                       *:*                         LISTEN
udp        0      0 localhost:asterix           *:*
udp        0      0 10.10.10.98:51700           10.233.0.10:8125            ESTABLISHED
udp        0      0 *:bootpc                    *:*
udp        0      0 10.10.10.98:amberon         *:*
udp        0      0 10.10.10.98:8302            *:*
udp        0      0 10.10.10.98:ntp             *:*
udp        0      0 localhost:ntp               *:*
udp        0      0 *:ntp                       *:*
udp        0      0 fe80::403:8eff:fe95:36c:ntp *:*
udp        0      0 localhost:ntp               *:*
udp        0      0 *:ntp                       *:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path
unix  2      [ ACC ]     STREAM     LISTENING     10585  /tmp/tmux-500/default
unix  7      [ ]         DGRAM                    8720   /dev/log
unix  2      [ ACC ]     STREAM     LISTENING     8799   /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     7627   @/com/ubuntu/upstart
unix  2      [ ACC ]     SEQPACKET  LISTENING     7832   @/org/kernel/udev/udevd
unix  3      [ ]         STREAM     CONNECTED     10578
unix  2      [ ]         DGRAM                    11277
unix  2      [ ]         DGRAM                    10519
unix  3      [ ]         STREAM     CONNECTED     10579
unix  3      [ ]         DGRAM                    7841
unix  3      [ ]         STREAM     CONNECTED     10584
unix  3      [ ]         STREAM     CONNECTED     8803
unix  3      [ ]         STREAM     CONNECTED     8802
unix  3      [ ]         STREAM     CONNECTED     8807
unix  3      [ ]         STREAM     CONNECTED     8808   /var/run/dbus/system_bus_socket
unix  2      [ ]         DGRAM                    10217
unix  3      [ ]         STREAM     CONNECTED     10523
unix  3      [ ]         STREAM     CONNECTED     10522
unix  3      [ ]         DGRAM                    7840
unix  3      [ ]         STREAM     CONNECTED     10581
unix  3      [ ]         STREAM     CONNECTED     10582
unix  2      [ ]         DGRAM                    9444
unix  2      [ ]         DGRAM                    9739
unix  3      [ ]         STREAM     CONNECTED     10583

I can work around this by, in my upstart, invoking consul agent .... -client=$BIND ... but then that forces me to specify the -rpc-addr=<external IP> on each consul command (in the server cluster, anyhow) thereafter; rather not do that since it complicates things a bit.

I can also work around this by running another consul cluster of agents whose only job is to join the cluster and host the UI - but I'd rather not do that, since it seems like it shouldn't be necessary and will cost more in terms of $ and management.

Is there a way to make this work without my workarounds?

@armon
Copy link
Member

armon commented Jan 14, 2015

You need to configure the HTTP address to listen on a public IP. By default it uses loopback. The -client flag does this, but also changes the IP of all the listeners (RPC, HTTP, DNS). You can do more fine grained configuration with a configuration file however.

@petemounce
Copy link
Author

That works perfectly; thanks!

@saulshanabrook
Copy link

@petemounce Could you share exactly what you did to make it work? I am also having the same problem.

@petemounce
Copy link
Author

@saulshanabrook I used the configuration file to set the http interface to listen on the NIC instead of the default loopback address.

...
  "addresses" : {
    "http": "10.10.10.257" // obviously made up ;)
  }
...

Here's the consul agent upstart task from my cloudformation:

"/etc/init/consul.conf": {
  "content": {
    "Fn::Join": [
      "",
      [
        "description \"Consul agent\"\n",
        "\n",
        "start on runlevel [2345]\n",
        "stop on runlevel [!2345]\n",
        "\n",
        "respawn\n",
        "\n",
        "script\n",
        "  # Make sure to use all our CPUs, because Consul can block a scheduler thread\n",
        "  export GOMAXPROCS=`nproc`\n",
        "\n",
        "  # Get the public IP\n",
        "  BIND=`ifconfig eth0 | grep \"inet addr\" | awk '{ print substr($2,6) }'`\n",
        "\n",
        "  export CLUSTER_SIZE=$(aws autoscaling describe-auto-scaling-groups | jq -c -M -r '.AutoScalingGroups[] | {name: .AutoScalingGroupName, desired_capacity: .DesiredCapacity}' | grep je-",
        {
          "Ref": "FeatureName"
        },
        "-",
        {
          "Ref": "EnvironmentName"
        },
        "-",
        {
          "Ref": "Tenant"
        },
        ".*ConsulCluster | jq -c -M -r '.desired_capacity' | tr -d \"\\n\")\n",
        "  exec /usr/local/bin/consul agent \\\n",
        "    -bootstrap-expect $CLUSTER_SIZE \\\n",
        "    -config-dir=\"/etc/consul.d\" \\\n",
        "    -bind=$BIND \\\n",
        "    >>/var/log/consul.log 2>&1\n",
        "end script\n",
        "\n"
      ]
    ]
  }
}

...

"022_configure_http_api_interface": {
  "command": "BIND=`ifconfig eth0 | grep \"inet addr\" | awk '{ print substr($2,6) }'` && sed -i \"s/http\\\": \\\"\\\"/http\\\": \\\"${BIND}\\\"/\" /etc/consul.d/consul.json"
},

@saulshanabrook
Copy link

@petemounce thank you!

Would $BIND be 10.10.10.257 as well in this case?

With it set up like that I still get this:

[centos@ip-10-128-1-235 ~]$ curl 10.128.1.235:8500/v1/catalog/nodes
curl: (7) Failed connect to 10.128.1.235:8500; Connection refused

But works with localhost

[centos@ip-10-128-1-235 ~]$ curl localhost:8500/v1/catalog/nodes
[{"Node":"10.128.1.115","Address":"10.128.1.115"},{"Node":"10.128.1.194","Address":"10.128.1.194"},{"Node":"10.128.1.198","Address":"10.128.1.198"},{"Node":"10.128.1.218","Address":"10.128.1.218"},{"Node":"10.128.1.235","Address":"10.128.1.235"},{"Node":"10.128.1.33","Address":"10.128.1.33"},{"Node":"10.128.1.57","Address":"10.128.1.57"},{"Node":"10.128.1.73","Address":"10.128.1.73"},{"Node":"10.128.1.80","Address":"10.128.1.80"}]

This is my config:

[centos@ip-10-128-1-235 ~]$ sudo cat /etc/consul/consul.json
{
  "datacenter": "tag_Service_controller",
  "addresses" : {
    "http": "10.128.1.235"
  },
  "bind_addr": "10.128.1.235",
  "node_name": "10.128.1.235",
  "rejoin_after_leave": true,
  "domain": "consul",
  "retry_join": [ "10.128.1.80", "10.128.1.235", "10.128.1.57" ],
  "server": true,
  "bootstrap_expect": 3,
  "verify_incoming": false,
  "verify_outgoing": false,
  "data_dir": "/var/lib/consul",
  "ui_dir": "/usr/share/consul-ui",
  "disable_remote_exec": false
}

@petemounce
Copy link
Author

Yes, think so. My pleasure :-)

Sent from my phone. Please excuse typos and brevity, but never text speak.
On 11 May 2015 23:04, "Saul Shanabrook" notifications@github.com wrote:

@petemounce https://github.com/petemounce thank you! Would $BIND be
"10.10.10.257" as well in this case?


Reply to this email directly or view it on GitHub
#599 (comment).

@liaokailin
Copy link

[root@localhost ~]# consul agent -h | grep client
-client=127.0.0.1 Sets the address to bind for client access.

we can see : the default ip is 127.0.0.1 ,modify this val change client visit ip

eg:

consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul/ -ui-dir /root/software/consul-ui/ -client=192.168.23.154

@ivasilyev-servicetitan-com

Hello.

I'm trying to solve similar task.
I need to connect to consul agent outside the machine using HTTP Api interface (port 8500).
The difference is - I want to connect to consul agent running in client mode, not server mode.

Is it possible?

Cause I tried different options with my consul.json config.
It's
{
"advertise_addr": "10.3.0.1",
"bind_addr": "10.3.0.1",
"addresses" : { "http": "10.3.0.1 0.0.0.0" },
"service": {"name": "myService"},
"client_addr": "0.0.0.0",
"datacenter": "westus",
"node_id": "51f55f0e-f215-44c6-95c6-3e776b9f7f0b",
"retry_join": ["10.2.10.19", "10.2.10.20", "10.2.10.21"],
"server": false,
"ui": true,
"log_level": "info",
"raft_protocol": 3,
"ports": { "dns": 53 },
"recursors": ["168.63.129.16"],
"telemetry": {
"dogstatsd_addr": "127.0.0.1:8125"
}
}
And it doesn't connect outside.
It succesfully opens http://10.3.0.1:8500/v1/kv/myKey
and http://127.0.0.1:8500/v1/kv/myKey from 10.3.0.1 machine.

But outside the machine http://10.3.0.1:8500/v1/kv/myKey can be opened.

I checked I can open http://10.2.10.19:8500/v1/kv/myKey (which consul server ip).

So is it possible to connect to consul client agent using HTTP API?

duckhan pushed a commit to duckhan/consul that referenced this issue Oct 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants