The connection to the server 172.17.4.101:443 was refused

[dixudx]

I followed the doc Kubernetes Installation with Vagrant & CoreOS to deploy a Kubernetes env. After the deployment using vagrant up, I can't access the cluster.

vagrant$ export KUBECONFIG="${KUBECONFIG}:$(pwd)/kubeconfig"
vagrant$ kubectl config use-context vagrant-multi
switched to context "vagrant-multi".
vagrant$ kubectl get nodes
The connection to the server 172.17.4.101:443 was refused - did you specify the right host or port?

After ssh into c1,

vagrant$ vagrant ssh c1
CoreOS alpha (1045.0.0)
Last login: Fri May 13 07:24:04 2016 from 10.0.2.2
Update Strategy: No Reboots
Failed Units: 2
  coreos-cloudinit-412108170.service
  update-engine.service
core@c1 ~ $ journalctl -u kubelet
-- No entries --
core@c1 ~ $ systemctl list-units
  UNIT                                                                                     LOAD      ACTIVE     SUB          JOB   DESCRIPTION
  boot.automount                                                                           loaded    active     waiting            Boot partition Automount Point
  proc-sys-fs-binfmt_misc.automount                                                        loaded    active     waiting            Arbitrary Executable File Formats File System Automount Point
  sys-devices-pci0000:00-0000:00:01.1-ata1-host0-target0:0:0-0:0:0:0-block-sda-sda1.device loaded    active     plugged            VBOX_HARDDISK EFI-SYSTEM
  sys-devices-pci0000:00-0000:00:01.1-ata1-host0-target0:0:0-0:0:0:0-block-sda-sda2.device loaded    active     plugged            VBOX_HARDDISK BIOS-BOOT
  sys-devices-pci0000:00-0000:00:01.1-ata1-host0-target0:0:0-0:0:0:0-block-sda-sda3.device loaded    active     plugged            VBOX_HARDDISK USR-A
  sys-devices-pci0000:00-0000:00:01.1-ata1-host0-target0:0:0-0:0:0:0-block-sda-sda4.device loaded    active     plugged            VBOX_HARDDISK USR-B
  sys-devices-pci0000:00-0000:00:01.1-ata1-host0-target0:0:0-0:0:0:0-block-sda-sda6.device loaded    active     plugged            VBOX_HARDDISK OEM
  sys-devices-pci0000:00-0000:00:01.1-ata1-host0-target0:0:0-0:0:0:0-block-sda-sda7.device loaded    active     plugged            VBOX_HARDDISK OEM-CONFIG
  sys-devices-pci0000:00-0000:00:01.1-ata1-host0-target0:0:0-0:0:0:0-block-sda-sda9.device loaded    active     plugged            VBOX_HARDDISK ROOT
  sys-devices-pci0000:00-0000:00:01.1-ata1-host0-target0:0:0-0:0:0:0-block-sda.device      loaded    active     plugged            VBOX_HARDDISK
  sys-devices-pci0000:00-0000:00:03.0-virtio0-net-eth0.device                              loaded    active     plugged            Virtio network device
  sys-devices-pci0000:00-0000:00:08.0-virtio1-net-eth1.device                              loaded    active     plugged            Virtio network device
  sys-devices-platform-serial8250-tty-ttyS0.device                                         loaded    active     plugged            /sys/devices/platform/serial8250/tty/ttyS0
  sys-devices-platform-serial8250-tty-ttyS1.device                                         loaded    active     plugged            /sys/devices/platform/serial8250/tty/ttyS1
  sys-devices-platform-serial8250-tty-ttyS2.device                                         loaded    active     plugged            /sys/devices/platform/serial8250/tty/ttyS2
  sys-devices-platform-serial8250-tty-ttyS3.device                                         loaded    active     plugged            /sys/devices/platform/serial8250/tty/ttyS3
  sys-devices-virtual-tty-ttyprintk.device                                                 loaded    active     plugged            /sys/devices/virtual/tty/ttyprintk
  sys-subsystem-net-devices-eth0.device                                                    loaded    active     plugged            Virtio network device
  sys-subsystem-net-devices-eth1.device                                                    loaded    active     plugged            Virtio network device
  -.mount                                                                                  loaded    active     mounted            /
  boot.mount                                                                               loaded    active     mounted            Boot partition
  dev-hugepages.mount                                                                      loaded    active     mounted            Huge Pages File System
  dev-mqueue.mount                                                                         loaded    active     mounted            POSIX Message Queue File System
  media.mount                                                                              loaded    active     mounted            External Media Directory
  run-user-500.mount                                                                       loaded    active     mounted            /run/user/500
  sys-kernel-debug.mount                                                                   loaded    active     mounted            Debug File System
  tmp.mount                                                                                loaded    active     mounted            Temporary Directory
  usr-share-oem.mount                                                                      loaded    active     mounted            /usr/share/oem
  usr.mount                                                                                loaded    active     mounted            /usr
  coreos-cloudinit-vagrant-user.path                                                       loaded    active     running            coreos-cloudinit-vagrant-user.path
  motdgen.path                                                                             loaded    active     waiting            Watch for update engine configuration changes
  systemd-ask-password-console.path                                                        loaded    active     waiting            Dispatch Password Requests to Console Directory Watch
  systemd-ask-password-wall.path                                                           loaded    active     waiting            Forward Password Requests to Wall Directory Watch
  user-cloudinit@var-lib-coreos\x2dinstall-user_data.path                                  loaded    active     waiting            Watch for a cloud-config at /var/lib/coreos-install/user_data
  init.scope                                                                               loaded    active     running            System and Service Manager
  session-2.scope                                                                          loaded    active     running            Session 2 of user core
  audit-rules.service                                                                      loaded    active     exited             Load Security Auditing Rules
● coreos-cloudinit-412108170.service                                                       not-found failed     failed             coreos-cloudinit-412108170.service
  coreos-setup-environment.service                                                         loaded    active     exited             Modifies /etc/environment for CoreOS
  dbus.service                                                                             loaded    active     running            D-Bus System Message Bus
  early-docker.service                                                                     loaded    active     running            Early Docker Application Container Engine
  flanneld.service                                                                         loaded    activating start        start Network fabric for containers
  getty@tty1.service                                                                       loaded    active     running            Getty on tty1
  kmod-static-nodes.service                                                                loaded    active     exited             Create list of required static device nodes for the current kernel
  ldconfig.service                                                                         loaded    active     exited             Rebuild Dynamic Linker Cache
  locksmithd.service                                                                       loaded    activating auto-restart       Cluster reboot manager
  sshd-keygen.service                                                                      loaded    active     exited             Generate sshd host keys
  sshd@1-10.0.2.15:22-10.0.2.2:62272.service                                               loaded    active     running            OpenSSH per-connection server daemon (10.0.2.2:62272)
  system-cloudinit@usr-share-oem-cloud\x2dconfig.yml.service                               loaded    active     exited             Load cloud-config from /usr/share/oem/cloud-config.yml
  system-cloudinit@var-tmp-hostname.yml.service                                            loaded    active     exited             Load cloud-config from /var/tmp/hostname.yml
  system-cloudinit@var-tmp-networks.yml.service                                            loaded    active     exited             Load cloud-config from /var/tmp/networks.yml
  systemd-hwdb-update.service                                                              loaded    active     exited             Rebuild Hardware Database
  systemd-journal-catalog-update.service                                                   loaded    active     exited             Rebuild Journal Catalog
  systemd-journal-flush.service                                                            loaded    active     exited             Flush Journal to Persistent Storage
  systemd-journald.service                                                                 loaded    active     running            Journal Service
  systemd-logind.service                                                                   loaded    active     running            Login Service
  systemd-networkd.service                                                                 loaded    active     running            Network Service
  systemd-random-seed.service                                                              loaded    active     exited             Load/Save Random Seed
  systemd-resolved.service                                                                 loaded    active     running            Network Name Resolution
  systemd-sysctl.service                                                                   loaded    active     exited             Apply Kernel Variables
  systemd-sysusers.service                                                                 loaded    active     exited             Create System Users
  systemd-timesyncd.service                                                                loaded    active     running            Network Time Synchronization
  systemd-tmpfiles-setup-dev.service                                                       loaded    active     exited             Create Static Device Nodes in /dev
  systemd-tmpfiles-setup.service                                                           loaded    active     exited             Create Volatile Files and Directories
  systemd-udev-settle.service                                                              loaded    active     exited             udev Wait for Complete Device Initialization
  systemd-udev-trigger.service                                                             loaded    active     exited             udev Coldplug all Devices
  systemd-udevd.service                                                                    loaded    active     running            udev Kernel Device Manager
  systemd-update-done.service                                                              loaded    active     exited             Update is Completed
  systemd-update-utmp.service                                                              loaded    active     exited             Update UTMP about System Boot/Shutdown
  systemd-user-sessions.service                                                            loaded    active     exited             Permit User Sessions
  systemd-vconsole-setup.service                                                           loaded    active     exited             Setup Virtual Console
● update-engine.service                                                                    masked    failed     failed             update-engine.service
  user-cloudinit@var-lib-coreos\x2dvagrant-vagrantfile\x2duser\x2ddata.service             loaded    active     exited             Load cloud-config from /var/lib/coreos-vagrant/vagrantfile-user-data
  -.slice                                                                                  loaded    active     active             Root Slice
  system-addon\x2dconfig.slice                                                             loaded    active     active             system-addon\x2dconfig.slice
  system-addon\x2drun.slice                                                                loaded    active     active             system-addon\x2drun.slice
  system-getty.slice                                                                       loaded    active     active             system-getty.slice
  system-sshd.slice                                                                        loaded    active     active             system-sshd.slice
  system-system\x2dcloudinit.slice                                                         loaded    active     active             system-system\x2dcloudinit.slice
  system-user\x2dcloudinit.slice                                                           loaded    active     active             system-user\x2dcloudinit.slice
  system.slice                                                                             loaded    active     active             System Slice
  user-500.slice                                                                           loaded    active     active             User Slice of core
  user.slice                                                                               loaded    active     active             User and Session Slice
  dbus.socket                                                                              loaded    active     running            D-Bus System Message Bus Socket
  docker.socket                                                                            loaded    active     listening          Docker Socket for the API
  early-docker.socket                                                                      loaded    active     running            Early Docker Socket for the API
  rkt-metadata.socket                                                                      loaded    active     listening          rkt metadata service socket
  sshd.socket                                                                              loaded    active     listening          OpenSSH Server Socket
  systemd-coredump.socket                                                                  loaded    active     listening          Process Core Dump Socket
  systemd-initctl.socket                                                                   loaded    active     listening          /dev/initctl Compatibility Named Pipe
  systemd-journald-audit.socket                                                            loaded    active     running            Journal Audit Socket
  systemd-journald-dev-log.socket                                                          loaded    active     running            Journal Socket (/dev/log)
  systemd-journald.socket                                                                  loaded    active     running            Journal Socket
  systemd-networkd.socket                                                                  loaded    active     running            Network Service Netlink Socket
  systemd-udevd-control.socket                                                             loaded    active     running            udev Control Socket
  systemd-udevd-kernel.socket                                                              loaded    active     running            udev Kernel Socket
  basic.target                                                                             loaded    active     active             Basic System
  cryptsetup.target                                                                        loaded    active     active             Encrypted Volumes
  getty.target                                                                             loaded    active     active             Login Prompts
  local-fs-pre.target                                                                      loaded    active     active             Local File Systems (Pre)
  local-fs.target                                                                          loaded    active     active             Local File Systems
  multi-user.target                                                                        loaded    active     active             Multi-User System
  network.target                                                                           loaded    active     active             Network
  paths.target                                                                             loaded    active     active             Paths
  remote-fs.target                                                                         loaded    active     active             Remote File Systems
  slices.target                                                                            loaded    active     active             Slices
  sockets.target                                                                           loaded    active     active             Sockets
  swap.target                                                                              loaded    active     active             Swap
  sysinit.target                                                                           loaded    active     active             System Initialization
  system-config.target                                                                     loaded    active     active             Load system-provided cloud configs
  time-sync.target                                                                         loaded    active     active             System Time Synchronized
  timers.target                                                                            loaded    active     active             Timers
  user-config.target                                                                       loaded    active     active             Load user-provided cloud configs
  logrotate.timer                                                                          loaded    active     waiting            Daily Log Rotation
  rkt-gc.timer                                                                             loaded    active     waiting            Periodic Garbage Collection for rkt
  systemd-tmpfiles-clean.timer                                                             loaded    active     waiting            Daily Cleanup of Temporary Directories

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
JOB    = Pending job for the unit.

116 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
core@c1 ~ $ systemctl list-units | grep kube
core@c1 ~ $ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.2.15  netmask 255.255.255.0  broadcast 10.0.2.255
        inet6 fe80::a00:27ff:fe89:25dc  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:89:25:dc  txqueuelen 1000  (Ethernet)
        RX packets 1357  bytes 143116 (139.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1163  bytes 132458 (129.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.4.101  netmask 255.255.255.0  broadcast 172.17.4.255
        inet6 fe80::a00:27ff:fec9:fa05  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:c9:fa:05  txqueuelen 1000  (Ethernet)
        RX packets 28  bytes 5948 (5.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 41  bytes 4489 (4.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 6  bytes 498 (498.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6  bytes 498 (498.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

core@c1 ~ $ netstat -tulpn
(No info could be read for "-p": geteuid()=500 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp6       0      0 :::22                   :::*                    LISTEN      -                   
udp        0      0 10.0.2.15:68            0.0.0.0:*                           -                   
udp6       0      0 fe80::a00:27ff:fe89:546 :::*  

No service is running on port 443.

And I've already updated the git repo to the latest commit 03f86ac

[philips]

@dixudx It can take some time for the services to come up. Can you get the log of the kubelet? journalctl -u kubelet?

[arealmaas]

Getting the same error when following the tutorial.

journalctl -u kubelet:

May 13 14:10:11 localhost kubelet-wrapper[1713]: image: using image from file /usr/share/rkt/stage1-fly.aci
May 13 14:10:11 localhost kubelet-wrapper[1713]: run: open /usr/share/rkt/stage1-fly.aci.asc: no such file or directory
May 13 14:10:11 localhost systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
May 13 14:10:11 localhost systemd[1]: kubelet.service: Unit entered failed state.
May 13 14:10:11 localhost systemd[1]: kubelet.service: Failed with result 'exit-code'.
May 13 14:10:21 localhost systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.

[danielfm]

@dixudx: I had the same problem today even though my cluster configuration (running in AWS) was working fine. It seems like a problem with the latest CoreOS alpha AMI (CoreOS-alpha-1045.0.0, ami-46ff772a). Downgrading to previous AMI (CoreOS-alpha-1032.1.0, ami-8d74fde1) did the trick for me.

Since you are using the same CoreOS version I was, maybe you are experiencing the same problem. Try using an earlier version.

[arealmaas]

Running this in vagrant, same issue with the alpha-1032.1.0

[aaronlevy]

This is due to an issue in the latest CoreOS alpha: coreos/bugs#1282

@AlmaasAre You are seeing the same error output in v1032.1.0?

Can you confirm os version by running cat /etc/os-release

[dixudx]

@aaronlevy

vagrant$ vagrant ssh c1
CoreOS alpha (1045.0.0)
Last login: Fri May 13 07:24:04 2016 from 10.0.2.2
Update Strategy: No Reboots
Failed Units: 2
  coreos-cloudinit-412108170.service
  update-engine.service

The version of CoreOS is CoreOS alpha (1045.0.0)

[dixudx]

@philips I destroyed all the instances and rebuilt them. The same errors still occurred.

vagrant$ vagrant status
Current machine states:

e1                        running (virtualbox)
e2                        running (virtualbox)
e3                        running (virtualbox)
c1                        running (virtualbox)
c2                        running (virtualbox)
w1                        running (virtualbox)
w2                        running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

vagrant$ vagrant ssh c1
CoreOS alpha (1045.0.0)
Last login: Sat May 14 06:06:59 2016 from 10.0.2.2
Failed Units: 2
  coreos-cloudinit-562055073.service
  update-engine.service
core@c1 ~ $ journalctl -u kubelet
-- No entries --
core@c1 ~ $ journalctl -f -u kubelet
-- Logs begin at Sat 2016-05-14 06:04:53 UTC. --
^C
core@c1 /etc/systemd $ systemctl list-unit-files | grep kubelet
kubelet.service                        disabled
core@c1 /etc/systemd $ ls
bootchart.conf  coredump.conf  journal-remote.conf  journal-upload.conf  journald.conf  logind.conf  network  resolved.conf  system  system.conf  timesyncd.conf  user  user.conf

Actually kubelet is not enabled. There is no doubt no service will run on port 443 and the connection is refused.

[micahnz]

@aaronlevy I have the similar with alpha-1032.1.0

CoreOS alpha (1032.1.0)
Last login: Sat May 14 12:30:47 2016 from 10.0.2.2
Failed Units: 1
  coreos-cloudinit-539161959.service

coreos-cloudinit-539161959.service                                 loaded failed failed

Edit: I am using Vagrant and the problem is the same with 1029.0.0 & 1032.0.0

[micahnz]

My failure was due to launching from windows and dos cariage return screwing up the install scripts, I ran dos2unix on the files and had no failed services on vagrant up. However none of my services actually started but I think that is unrelated to this issue.

[wangkuiyi]

beta channel (1010.3.0) works for me. alpha doesn't.

core@localhost ~ $ cat /etc/os-release 
NAME=CoreOS
ID=coreos
VERSION=1010.3.0
VERSION_ID=1010.3.0
BUILD_ID=2016-05-05-0730
PRETTY_NAME="CoreOS 1010.3.0 (MoreOS)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

[arealmaas]

@dixudx Works with 1032.1.0 now, just took a while to boot up :)

[dixudx]

The same issue still occurred on latest version 1047.0.0.

vagrant$ vagrant status
Current machine states:

e1                        running (virtualbox)
e2                        running (virtualbox)
e3                        running (virtualbox)
c1                        running (virtualbox)
c2                        running (virtualbox)
w1                        running (virtualbox)
w2                        running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.
vagrant$ export KUBECONFIG="${KUBECONFIG}:$(pwd)/kubeconfig"
vagrant$ kubectl config use-context vagrant-multi
switched to context "vagrant-multi".
vagrant$ kubectl get nodes
The connection to the server 172.17.4.101:443 was refused - did you specify the right host or port?

vagrant$ vagrant ssh c1
CoreOS alpha (1047.0.0)
Last login: Sun May 15 13:12:03 2016 from 10.0.2.2
Failed Units: 2
  coreos-cloudinit-192295069.service
  update-engine.service
core@c1 ~ $ journalctl -u kubelet
-- No entries --
core@c1 ~ $ cat /etc/os-release
NAME=CoreOS
ID=coreos
VERSION=1047.0.0
VERSION_ID=1047.0.0
BUILD_ID=2016-05-14-0118
PRETTY_NAME="CoreOS 1047.0.0 (MoreOS)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"
core@c1 ~ $ journalctl -f -u kubelet
-- Logs begin at Sun 2016-05-15 13:09:40 UTC. --
^C
core@c1 ~ $ systemctl list-unit-files | grep kubelet
kubelet.service                        disabled

[dixudx]

@AlmaasAre what do you mean taking a while to boot up on version 1032.1.0. Did you perform some manual operations. The services of CoreOS are managed by systemd.

core@c1 ~ $ systemctl list-unit-files | grep kubelet
kubelet.service                        disabled

From the above log, the service kubelet is disabled, not even in activating status, which means service kubelet is not currently governed or triggered by other services.

[arealmaas]

@dixudx Sorry, the kubelet service took a while to start. Suddenly it started working again. Might've been a restart of the VM that solved it not sure. Works with 1032.1.0 now

[msens]

Same issue here:

$ vagrant ssh c1
CoreOS alpha (1045.0.0)
Last login: Thu May 19 14:00:54 2016 from 10.0.2.2
Update Strategy: No Reboots
Failed Units: 1
  update-engine.service

core@c1 ~ $ cat /etc/os-release
NAME=CoreOS
ID=coreos
VERSION=1045.0.0
VERSION_ID=1045.0.0
BUILD_ID=2016-05-11-2111
PRETTY_NAME="CoreOS 1045.0.0 (MoreOS)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

core@c1 ~ $ journalctl -u kubelet
..
..
May 19 14:04:19 c1 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
May 19 14:04:19 c1 systemd[1]: Stopped kubelet.service.
May 19 14:04:19 c1 systemd[1]: Starting kubelet.service...
May 19 14:04:19 c1 systemd[1]: Started kubelet.service.
May 19 14:04:19 c1 kubelet-wrapper[2433]: image: using image from file /usr/share/rkt/stage1-fly.aci
May 19 14:04:19 c1 kubelet-wrapper[2433]: run: open /usr/share/rkt/stage1-fly.aci.asc: no such file or directory
May 19 14:04:19 c1 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
May 19 14:04:19 c1 systemd[1]: kubelet.service: Unit entered failed state.
May 19 14:04:19 c1 systemd[1]: kubelet.service: Failed with result 'exit-code'.
May 19 14:04:30 c1 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
May 19 14:04:30 c1 systemd[1]: Stopped kubelet.service.
May 19 14:04:30 c1 systemd[1]: Starting kubelet.service...
May 19 14:04:30 c1 systemd[1]: Started kubelet.service.
May 19 14:04:30 c1 kubelet-wrapper[2464]: image: using image from file /usr/share/rkt/stage1-fly.aci
May 19 14:04:30 c1 kubelet-wrapper[2464]: run: open /usr/share/rkt/stage1-fly.aci.asc: no such file or directory
May 19 14:04:30 c1 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
May 19 14:04:30 c1 systemd[1]: kubelet.service: Unit entered failed state.
May 19 14:04:30 c1 systemd[1]: kubelet.service: Failed with result 'exit-code'.

core@c1 ~ $ systemctl | grep kube
  kubelet.service                                                                          loaded    activating auto-restart kubelet.service

core@c1 ~ $ netstat -tulpn
(No info could be read for "-p": geteuid()=500 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 :::22                   :::*                    LISTEN      -
udp        0      0 0.0.0.0:8472            0.0.0.0:*                           -
udp        0      0 10.0.2.15:68            0.0.0.0:*                           -
udp6       0      0 fe80::a00:27ff:fe89:546 :::*                                -

core@c1 ~ $ systemctl list-unit-files
UNIT FILE                              STATE
boot.automount                         static
proc-sys-fs-binfmt_misc.automount      static
org.freedesktop.hostname1.busname      static
org.freedesktop.import1.busname        static
org.freedesktop.locale1.busname        static
org.freedesktop.login1.busname         static
org.freedesktop.machine1.busname       static
org.freedesktop.network1.busname       static
org.freedesktop.resolve1.busname       static
org.freedesktop.systemd1.busname       static
org.freedesktop.timedate1.busname      static
dev-disk-by\x2dlabel-OEM.device        static
boot.mount                             static
dev-hugepages.mount                    static
dev-mqueue.mount                       static
media-configdrive.mount                static
media-configvirtfs.mount               static
media-ovfenv.mount                     static
media.mount                            static
proc-fs-nfsd.mount                     static
proc-sys-fs-binfmt_misc.mount          static
proc-xen.mount                         static
sys-fs-fuse-connections.mount          static
sys-kernel-config.mount                static
sys-kernel-debug.mount                 static
tmp.mount                              static
usr-share-oem.mount                    static
var-lib-machines.mount                 static
var-lib-nfs-rpc_pipefs.mount           static
coreos-cloudinit-vagrant-user.path     static
motdgen.path                           static
systemd-ask-password-console.path      static
systemd-ask-password-wall.path         static
user-cloudinit@.path                   static
addon-config@.service                  static
addon-run@.service                     static
audit-rules.service                    disabled
auth-rpcgss-module.service             static
autovt@.service                        disabled
blk-availability.service               disabled
calico-node.service                    disabled
clean-ca-certificates.service          static
console-getty.service                  disabled
console-shell.service                  disabled
container-getty@.service               static
coreos-cloudinit-vagrant-mkdir.service static
coreos-metadata-sshkeys@.service       disabled
coreos-metadata.service                disabled
coreos-setup-environment.service       disabled
coreos-tmpfiles.service                static
dbus-org.freedesktop.hostname1.service static
dbus-org.freedesktop.import1.service   static
dbus-org.freedesktop.locale1.service   static
dbus-org.freedesktop.login1.service    static
dbus-org.freedesktop.machine1.service  static
dbus-org.freedesktop.network1.service  disabled
dbus-org.freedesktop.resolve1.service  disabled
dbus-org.freedesktop.timedate1.service static
dbus.service                           static
debug-shell.service                    disabled
dhcpcd.service                         disabled
dm-event.service                       disabled
docker.service                         disabled
early-docker.service                   disabled
emergency.service                      static
etcd.service                           static
etcd2.service                          disabled
extend-filesystems.service             static
flanneld.service                       enabled
fleet.service                          disabled
fstrim.service                         static
getty@.service                         disabled
git-daemon@.service                    static
initrd-cleanup.service                 static
initrd-parse-etc.service               static
initrd-switch-root.service             static
initrd-udevadm-cleanup-db.service      static
ip6tables-restore.service              disabled
ip6tables-store.service                disabled
ip6tables.service                      indirect
iptables-restore.service               disabled
iptables-store.service                 disabled
iptables.service                       indirect
iscsid-initiatorname.service           static
iscsid.service                         disabled
issuegen.service                       static
kmod-static-nodes.service              static
kubelet.service                        enabled
ldconfig.service                       static
locksmithd.service                     disabled
logrotate.service                      static
lvm2-lvmetad.service                   disabled
lvm2-monitor.service                   disabled
lvm2-pvscan@.service                   static
mcstrans.service                       disabled
mdadm-grow-continue@.service           static
mdadm-last-resort@.service             static
mdmon@.service                         static
mdmonitor.service                      static
motdgen.service                        static
multipathd.service                     disabled
nfs-idmapd.service                     static
nfs-mountd.service                     static
nfs-server.service                     disabled
nfs-utils.service                      static
nfsd.service                           disabled
ntpd.service                           disabled
ntpdate.service                        disabled
polkit.service                         static
pwcheck.service                        disabled
quotaon.service                        static
realmd.service                         static
remount-root.service                   static
remount-usr.service                    static
rescue.service                         static
rkt-gc.service                         static
rkt-metadata.service                   disabled
rngd.service                           disabled
rpc-idmapd.service                     static
rpc-mountd.service                     static
rpc-statd-notify.service               static
rpc-statd.service                      static
rpcbind.service                        disabled
rsyncd.service                         disabled
saslauthd.service                      disabled
serial-getty@.service                  disabled
setup-nsswitch.service                 static
sntp.service                           disabled
ssh-key-proc-cmdline.service           disabled
sshd-keygen.service                    static
sshd.service                           disabled
sshd@.service                          static
sssd.service                           disabled
system-cloudinit@.service              static
systemd-ask-password-console.service   static
systemd-ask-password-wall.service      static
systemd-backlight@.service             static
systemd-binfmt.service                 static
systemd-bootchart.service              disabled
systemd-bus-proxyd.service             static
systemd-coredump@.service              static
systemd-exit.service                   static
systemd-firstboot.service              static
systemd-fsck-root.service              static
systemd-fsck@.service                  static
systemd-halt.service                   static
systemd-hibernate-resume@.service      static
systemd-hibernate.service              static
systemd-hostnamed.service              static
systemd-hwdb-update.service            static
systemd-hybrid-sleep.service           static
systemd-importd.service                static
systemd-initctl.service                static
systemd-journal-catalog-update.service static
systemd-journal-flush.service          static
systemd-journal-gatewayd.service       indirect
systemd-journal-remote.service         indirect
systemd-journal-upload.service         disabled
systemd-journald.service               static
systemd-kexec.service                  static
systemd-localed.service                static
systemd-logind.service                 static
systemd-machine-id-commit.service      static
systemd-machined.service               static
systemd-modules-load.service           static
systemd-networkd-wait-online.service   disabled
systemd-networkd.service               disabled
systemd-nspawn@.service                disabled
systemd-poweroff.service               static
systemd-quotacheck.service             static
systemd-random-seed.service            static
systemd-reboot.service                 static
systemd-remount-fs.service             static
systemd-resolved.service               disabled
systemd-rfkill.service                 static
systemd-suspend.service                static
systemd-sysctl.service                 static
systemd-sysusers.service               static
systemd-timedated.service              static
systemd-timesyncd.service              disabled
systemd-tmpfiles-clean.service         static
systemd-tmpfiles-setup-dev.service     static
systemd-tmpfiles-setup.service         static
systemd-udev-settle.service            static
systemd-udev-trigger.service           static
systemd-udevd.service                  static
systemd-update-done.service            static
systemd-update-utmp-runlevel.service   static
systemd-update-utmp.service            static
systemd-user-sessions.service          static
systemd-vconsole-setup.service         static
tcsd.service                           disabled
tpmd.service                           disabled
update-ca-certificates.service         static
update-engine-stub.service             disabled
update-engine.service                  masked
user-cloudinit-proc-cmdline.service    static
user-cloudinit@.service                static
user-config-ovfenv.service             static
user-configdrive.service               static
user-configvirtfs.service              static
user@.service                          static
uuidd.service                          indirect
virtfs@.service                        static
xenserver-pv-version.service           static
-.slice                                static
machine.slice                          static
system.slice                           static
user.slice                             static
dbus.socket                            static
dm-event.socket                        disabled
docker.socket                          disabled
early-docker.socket                    disabled
fleet.socket                           static
git-daemon.socket                      disabled
iscsid.socket                          disabled
lvm2-lvmetad.socket                    disabled
multipathd.socket                      disabled
rkt-metadata.socket                    disabled
sshd.socket                            disabled
syslog.socket                          static
systemd-bus-proxyd.socket              static
systemd-coredump.socket                static
systemd-initctl.socket                 static
systemd-journal-gatewayd.socket        disabled
systemd-journal-remote.socket          disabled
systemd-journald-audit.socket          static
systemd-journald-dev-log.socket        static
systemd-journald.socket                static
systemd-networkd.socket                disabled
systemd-rfkill.socket                  static
systemd-udevd-control.socket           static
systemd-udevd-kernel.socket            static
uuidd.socket                           disabled
basic.target                           static
bluetooth.target                       static
busnames.target                        static
cryptsetup-pre.target                  static
cryptsetup.target                      static
ctrl-alt-del.target                    disabled
default.target                         static
early-docker.target                    static
emergency.target                       static
exit.target                            disabled
final.target                           static
getty.target                           static
graphical.target                       static
halt.target                            disabled
hibernate.target                       static
hybrid-sleep.target                    static
initrd-fs.target                       static
initrd-root-fs.target                  static
initrd-switch-root.target              static
initrd.target                          static
kexec.target                           disabled
local-fs-pre.target                    static
local-fs.target                        static
machines.target                        disabled
multi-user.target                      static
network-online.target                  static
network-pre.target                     static
network.target                         static
nfs-client.target                      disabled
nss-lookup.target                      static
nss-user-lookup.target                 static
paths.target                           static
poweroff.target                        disabled
printer.target                         static
reboot.target                          disabled
remote-fs-pre.target                   static
remote-fs.target                       disabled
rescue.target                          disabled
rpcbind.target                         static
shutdown.target                        static
sigpwr.target                          static
sleep.target                           static
slices.target                          static
smartcard.target                       static
sockets.target                         static
sound.target                           static
suspend.target                         static
swap.target                            static
sysinit.target                         static
system-config.target                   static
system-update.target                   static
time-sync.target                       static
timers.target                          static
umount.target                          static
user-config.target                     static
fstrim.timer                           disabled
logrotate.timer                        static
mdadm-last-resort@.timer               static
rkt-gc.timer                           disabled
systemd-tmpfiles-clean.timer           static
update-engine-stub.timer               disabled

295 unit files listed.

[msens]

Also with newer box 1053.2.0, still an issue:

vagrant ssh c1
CoreOS alpha (1053.2.0)
Last login: Tue May 24 17:24:05 2016 from 10.0.2.2
Failed Units: 1
  update-engine.service

[wmangano]

i tryed to vagrant up with 1010.5.0 and 1053.2.0 .... no luck.

localbox$ kubectl get nodes
The connection to the server 172.17.4.101:443 was refused - did you specify the right host or port?

cannot find the error. journal -f shows a loop while downloading/trying to start flannel.

[aaronlevy]

This seems to be touching on a few different issues. I'll try and touch on them, but if you are still having an issue, please open a new ticket (at this point this issue covers a few different issues).

Originally, there was an error where the rkt-fly stage1 could not be found (coreos/bugs#1282) -- this should be resolved in all alpha/beta/stable channels.

the "update-engine.service" failed unit is actually because we disable auto-updates to slightly improve the user-experience of not having a server reboot immediately on first boot. But nothing is broken.

@wmangano would you mind opening a new issue if you are still having problems? It might have to do with flannel not being able to contact etcd -- but if it's still and issue we can dig further.

[cdaringe]

for those who just stumble onto this issue, the downloads do often take a while as discussed above. to confirm that your downloads are still in transit, you can:

• vagrant ssh c1 (or other nodes in your cluster)
• look at the kubelets log, journalctl -u kubelet. look at the bottom for a download progress update!

[MattFriedman]

@cdaringe

Just spent a bunch of time hunting this down as well.

Could some output be put onto stdout/console so that this isn't such a mystery to the uninitiated?

I think it would save a lot of people some trouble and hunting. I think it would be very helpful.

Thanks,
Matt

[aaronlevy]

Another option might just to put a blocking loop at the end of the vagrant script which waits for the api-server to be available (and outputs something like "waiting for api..."

[aaronlevy]

Opened #806 to track this

[linuxman1]

I think a lot of user are using coreos-kubernetes so why nobody there would update the configs to help us?
kubelet service is not enabled by default so I started it like,
systemctl list-unit-files | grep kubelet
kubelet.service disabled
sudo systemctl start kubelet
sudo systemctl enable kubelet
Then I had issue says,
coreos-cloudinit[905]: 2018/08/29 22:36:54 Failed to apply cloud-config: Unit etcd2.service not found.
Aug 29 22:51:25 w1 flannel-wrapper[1588]: E0829 22:51:25.555657 1588 main.go:349] Couldn't fetch network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 172.17.4.51:2379: getsockopt: connection refused
I searched and found this,
coreos/bugs#2395
They say both etcd2 and fleet are deprecated on CoreOs.
Docs says Docker is used by default, and I didn't update to use Rockets, but in reality Rockets are working instead of Docker by default!
core@w2 ~ $ rkt list
UUID APP IMAGE NAME STATE CREATED STARTED NETWORKS
28ec0fc9 hyperkube quay.io/coreos/hyperkube:v1.7.3_coreos.0 running 27 minutes ago 27 minutes ago
8997c8bb hyperkube quay.io/coreos/hyperkube:v1.7.3_coreos.0 exited garbage 1 hour ago 1 hour ago
dd3086fa hyperkube quay.io/coreos/hyperkube:v1.7.3_coreos.0 exited garbage 53 minutes ago 53 minutes ago
ec7268f7 flannel quay.io/coreos/flannel:v0.10.0 running 1 minute ago 1 minute ago
But seems it listens only on 127.0.0.1, so other workers can't connect to it,
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN -
Aug 29 22:24:38 w3 flannel-wrapper[1322]: E0829 22:24:38.741764 1322 main.go:349] Couldn't fetch network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 172.17.4.51:2379: getsockopt: connection refused