Failed connect to 127.0.0.1:4001 (vmware)

[drnic]

I booted up coreos via vagrant/vmware instructions, ssh'd in, and followed along with http://coreos.com/docs/using-coreos/

$ curl -L http://127.0.0.1:4001/v1/keys/message -d value="Hello world"
curl: (7) Failed connect to 127.0.0.1:4001; Connection refused

The tail of my running processes is:

  248 ?        Ss     0:00 /usr/lib/systemd/systemd-journald
  270 ?        S      0:00 [kworker/0:3]
  274 ?        Ss     0:00 /usr/lib/systemd/systemd-udevd
  331 ?        S      0:00 [jbd2/sda9-8]
  332 ?        S<     0:00 [ext4-dio-unwrit]
  349 ?        S      0:00 [jbd2/sda6-8]
  350 ?        S<     0:00 [ext4-dio-unwrit]
  382 ?        Ss     0:00 /sbin/dhcpcd -q --nobackground
  384 ?        Ss     0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
  386 ?        Ss     0:00 /bin/bash /usr/sbin/update_engine_reboot_manager
  389 ?        Ss     0:00 /usr/lib/systemd/systemd-logind
  392 ?        Ssl    0:00 /usr/sbin/update_engine -foreground -logtostderr -no_connection_manager
  395 ?        Ssl    0:00 /usr/bin/docker -d -D
  397 ?        S      0:00 dbus-monitor --system type=signal,interface='org.chromium.UpdateEngineInterface',member='StatusUpdate'
  399 ?        S      0:00 /bin/bash /usr/sbin/update_engine_reboot_manager
  402 tty1     Ss+    0:00 /sbin/agetty --noclear tty1
  404 ttyS0    Ss+    0:00 /sbin/agetty --keep-baud ttyS0 115200 38400 9600
  616 ?        Ss     0:00 sshd: core [priv]   
  618 ?        S      0:00 sshd: core@notty    
  735 ?        Ss     0:00 sshd: core [priv]   
  737 ?        R      0:00 sshd: core@pts/0    
  738 pts/0    Ss     0:00 -bash
  746 pts/0    R+     0:00 ps ax

There is no etcd process. Is there a step missing in docs or a bug in the vagrant up scripts?

[philips]

@drnic What does journalctl -u etcd say?

You can start etcd again with systemctl start etcd.

Thanks.

[drnic]

$ journalctl -u etcd
-- Logs begin at Mon 2013-09-23 12:53:52 UTC, end at Mon 2013-09-23 12:58:52 UTC. --
Sep 23 12:53:55 localhost etcd-bootstrap[464]: Detected environment "vmware", just starting solo master...
Sep 23 12:53:55 localhost etcd-bootstrap[464]: [etcd] 12:53:55.554878 INFO Wrote node configuration to '/var/lib/etcd/info'
Sep 23 12:53:55 localhost etcd-bootstrap[464]: [etcd] 12:53:55.625561 INFO etcd server [default-name:http://127.0.0.1:4001]
Sep 23 12:53:55 localhost etcd-bootstrap[464]: [etcd] 12:53:55.627543 INFO raft server [default-name:127.0.0.1:7001]
Sep 23 12:54:10 localhost systemd[1]: Stopping etcd...
Sep 23 12:54:10 localhost systemd[1]: etcd.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 23 12:54:10 localhost systemd[1]: Stopped etcd.
Sep 23 12:54:10 localhost systemd[1]: Unit etcd.service entered failed state.

Restarting works:

core@localhost ~ $ sudo systemctl start etcd
core@localhost ~ $ curl -L http://127.0.0.1:4001/v1/keys/message -d value="Hello world"
{"action":"SET","key":"/message","value":"Hello world","newKey":true,"index":3}

[philips]

Interesting. I will look into that.

[philips]

@drnic Is this resolved in the latest release?

[drnic]

11 days ago I assume it was. I was following the instructions/docs.

[philips]

@drnic Ok, I will close it out for now. Let me know if it comes back!

Thanks, Brandon

[gmbuell]

I'm experiencing this issue whenever I specify a private network in vagrant. For example:

config.vm.network "private_network", ip: "10.0.0.2"

etcd fails to start but then runs after being started manually.

As a potentially related issue, if I try and run two separate coreos boxes inside the same vagrant configuration, even after starting etcd manually values set on one machine do not propagate to the other
Vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.define "server1" do |server|
    server.vm.box = "coreos"
    server.vm.box_url =
      "http://storage.core-os.net/coreos/amd64-generic/dev-channel/coreos_production_vagrant.box"
    server.vm.network "private_network", ip: "10.0.0.2"
  end
  config.vm.define "server2" do |server|
    server.vm.box = "coreos"
    server.vm.box_url =
      "http://storage.core-os.net/coreos/amd64-generic/dev-channel/coreos_production_vagrant.box"
    server.vm.network "private_network", ip: "10.0.0.3"
  end
end

Note that if not setting private network settings manually, etcd starts correctly but does not propagate between multiple machines either.

This is on Windows with Vagrant 1.3.5 and VirtualBox-4.3.2-90405-Win

[relaxdiego]

I'm having the same experience as @gmbuell. I commented out the lines that configure shared folders and etcd is working fine again.

#   config.vm.network "private_network", ip: "172.12.8.150"
#   config.vm.synced_folder ".", "/home/core/share", id: "core", :nfs => true,  :mount_options   => ['nolock,vers=3,udp']  

Running:

• Vagrant 1.3.5
• VirtualBox 4.3.2 r90405
• OS X Mavericks

[relaxdiego]

I automated @philips workaround in the Vagrantfile:

  config.vm.network "private_network", ip: "172.12.8.150"
  config.vm.synced_folder ".", "/home/core/share", id: "core", :nfs => true,  :mount_options   => ['nolock,vers=3,udp']  

  # Workaround for bug https://github.com/coreos/docs/issues/21
  config.vm.provision 'shell', inline: 'sudo systemctl start etcd'

[hyperscientist]

I believe this issue should be reopened as I'm experiencing the same problem.

After setting up synced_folder (in Vagrantfile) I'm getting reboot loop of:

Jan 17 10:31:59 localhost systemd[1]: Starting etcd...
Jan 17 10:31:59 localhost systemd[1]: Started etcd.
Jan 17 10:32:00 localhost etcd-bootstrap[354]: Detected environment "oracle", just starting solo master...
Jan 17 10:32:00 localhost etcd-bootstrap[354]: [deprecated] use -bind-addr, not -cl[deprecated] use -data-dir, not -d[deprecated] use -name, not -n[etcd] Jan 17 10:32:00.222 INFO      | etcd se
Jan 17 10:32:00 localhost etcd-bootstrap[354]: [etcd] Jan 17 10:32:00.223 INFO      | raft server [name localhost, listen on :7001, advertised url http://127.0.0.1:7001]
Jan 17 11:03:21 localhost systemd[1]: Stopping etcd...
Jan 17 11:03:21 localhost systemd[1]: etcd.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 17 11:03:21 localhost systemd[1]: Stopped etcd.
Jan 17 11:03:21 localhost systemd[1]: Unit etcd.service entered failed state.

I'm using up to date: OS X + VirtualBox + Vagrant + coreos-vagrant

// I get that there are workarounds and I'm fine with them for now. I only wanted to give you guys a shout that the problem still exist.

[blairham]

Just an update, this should now be fixed:

hashicorp/vagrant#2852

I've verified it's working properly now.

[ceefour]

I'm also getting this issue :

Host: Ubuntu 13.10 64-bit
vagrant 1:1.5.4
virtualbox 4.3.0-89960Ubunturaring
CoreOS 291.0.0

Before restarting etcd :

core@core-01 ~ $ curl -L http://127.0.0.1:4001/v1/keys/message -d value="Hello world"
curl: (7) Failed connect to 127.0.0.1:4001; Connection refused

core@core-01 ~ $ journalctl -u etcd
-- Logs begin at Wed 2014-04-23 12:17:01 UTC, end at Wed 2014-04-23 12:38:25 UTC. --

After restart etcd :

core@core-01 ~ $ sudo systemctl start etcd

core@core-01 ~ $ curl -L http://127.0.0.1:4001/v1/keys/message -d value="Hello world"
{"action":"set","key":"/message","value":"Hello world","newKey":true,"index":2}

core@core-01 ~ $ journalctl -u etcd
-- Logs begin at Wed 2014-04-23 12:17:01 UTC, end at Wed 2014-04-23 12:40:37 UTC. --
Apr 23 12:40:36 core-01 systemd[1]: Starting etcd...
Apr 23 12:40:36 core-01 systemd[1]: Started etcd.
Apr 23 12:40:37 core-01 etcd[3309]: [etcd] Apr 23 12:40:37.009 INFO      | default: state changed from 'stopped' to 'follower'.
Apr 23 12:40:37 core-01 etcd[3309]: [etcd] Apr 23 12:40:37.010 INFO      | default: state changed from 'follower' to 'leader'.
Apr 23 12:40:37 core-01 etcd[3309]: [etcd] Apr 23 12:40:37.011 INFO      | default: leader changed from '' to 'default'.
Apr 23 12:40:37 core-01 etcd[3309]: [etcd] Apr 23 12:40:37.013 INFO      | etcd server [name default, listen on [::]:4001, advertised url http://127.0.0.1:4001]
Apr 23 12:40:37 core-01 etcd[3309]: [etcd] Apr 23 12:40:37.015 INFO      | peer server [name default, listen on [::]:7001, advertised url http://127.0.0.1:7001]

Vagrantfile is exactly as in here : https://github.com/coreos/coreos-vagrant/tree/f0f64398e1c12d0736661200fc60a0115914e9ab

[ceefour]

I think this issue should be reopened...

[chriscowley]

I also feel this should be re-opened. I have a simple install under libvirt. My libvirt xml file resembles that in the example and my user-data file is:

#cloud-config

hostname: coreos0
ssh_authorized_keys:
 - ssh-rsa ... <blah, blah,blah>
coreos:
  etcd:
    name: coreos0
    discovery: https://discovery.etcd.io/<token>
    addr: <dhcp-assigned-ip>:4001
    peer-addr: <dhcp-assigned-ip>:7001

I have also tried that with the addr and peer-addr set to 127.0.0.1. In both cases journalctl tells me that etcd is started.

core@coreos0 ~ $ curl -L http://127.0.0.1:4001/v1/keys/message -d value="Hello world"
curl: (7) Failed connect to 127.0.0.1:4001; Connection refused

Strikes me that there is something fundamentally wrong here, either with CoreOS itself, or the docs.