Flanneld timing out while downloading image #1833

Closed
F21 opened this Issue Feb 26, 2017 · 1 comment

Comments

Projects
None yet
4 participants
@F21

F21 commented Feb 26, 2017

Issue Report

Bug

Container Linux Version

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1325.1.0
VERSION_ID=1325.1.0
BUILD_ID=2017-02-23-0035
PRETTY_NAME="Container Linux by CoreOS 1325.1.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

Environment

Vagrant and VirtualBox 5.1.14

Expected Behavior

Flanneld should not timeout while downloading the image

Actual Behavior

The Flanneld image does not download completely before timeout out (due to a slower connection). This causes flannel-docker-opts.service to fail.

 $ journalctl -u  flannel-docker-opts.service
-- Logs begin at Sun 2017-02-26 04:21:43 UTC, end at Sun 2017-02-26 05:13:08 UTC. --
Feb 26 04:22:43 c1 systemd[1]: Starting flannel docker export service - Network fabric for containers (System Application Container)...
Feb 26 04:22:43 c1 rkt[1728]: rm: unable to resolve UUID from file: open /var/lib/coreos/flannel-wrapper2.uuid: no such file or directory
Feb 26 04:22:43 c1 rkt[1728]: rm: failed to remove one or more pods
Feb 26 04:22:43 c1 flannel-wrapper[1758]: + exec /usr/bin/rkt run --uuid-file-save=/var/lib/coreos/flannel-wrapper2.uuid --trust-keys-from-https --net=host --volume run-flannel,kind=host,source=/run/flannel,readOnly=false --volume etc-ss
Feb 26 04:22:45 c1 flannel-wrapper[1758]: image: keys already exist for prefix "quay.io/coreos/flannel", not fetching again
Feb 26 04:22:47 c1 flannel-wrapper[1758]: Downloading signature:  0 B/473 B
Feb 26 04:22:47 c1 flannel-wrapper[1758]: Downloading signature:  473 B/473 B
Feb 26 04:22:47 c1 flannel-wrapper[1758]: Downloading signature:  473 B/473 B
Feb 26 04:22:49 c1 flannel-wrapper[1758]: Downloading ACI:  0 B/8.58 MB
Feb 26 04:22:49 c1 flannel-wrapper[1758]: Downloading ACI:  16.3 KB/8.58 MB
Feb 26 04:22:50 c1 flannel-wrapper[1758]: Downloading ACI:  138 KB/8.58 MB
Feb 26 04:22:51 c1 flannel-wrapper[1758]: Downloading ACI:  399 KB/8.58 MB
Feb 26 04:22:52 c1 flannel-wrapper[1758]: Downloading ACI:  799 KB/8.58 MB
Feb 26 04:22:53 c1 flannel-wrapper[1758]: Downloading ACI:  1.15 MB/8.58 MB
Feb 26 04:22:54 c1 flannel-wrapper[1758]: Downloading ACI:  1.32 MB/8.58 MB
Feb 26 04:22:55 c1 flannel-wrapper[1758]: Downloading ACI:  1.65 MB/8.58 MB
Feb 26 04:22:56 c1 flannel-wrapper[1758]: Downloading ACI:  1.95 MB/8.58 MB
Feb 26 04:22:57 c1 flannel-wrapper[1758]: Downloading ACI:  2.24 MB/8.58 MB
Feb 26 04:22:58 c1 flannel-wrapper[1758]: Downloading ACI:  2.56 MB/8.58 MB
Feb 26 04:22:59 c1 flannel-wrapper[1758]: Downloading ACI:  2.87 MB/8.58 MB
Feb 26 04:23:00 c1 flannel-wrapper[1758]: Downloading ACI:  3.18 MB/8.58 MB
Feb 26 04:23:01 c1 flannel-wrapper[1758]: Downloading ACI:  3.51 MB/8.58 MB
Feb 26 04:23:02 c1 flannel-wrapper[1758]: Downloading ACI:  3.81 MB/8.58 MB
Feb 26 04:23:03 c1 flannel-wrapper[1758]: Downloading ACI:  4.05 MB/8.58 MB
Feb 26 04:23:04 c1 flannel-wrapper[1758]: Downloading ACI:  4.32 MB/8.58 MB
Feb 26 04:23:06 c1 flannel-wrapper[1758]: Downloading ACI:  4.51 MB/8.58 MB
Feb 26 04:23:07 c1 flannel-wrapper[1758]: Downloading ACI:  4.73 MB/8.58 MB
Feb 26 04:23:08 c1 flannel-wrapper[1758]: Downloading ACI:  4.92 MB/8.58 MB
Feb 26 04:23:09 c1 flannel-wrapper[1758]: Downloading ACI:  5.08 MB/8.58 MB
Feb 26 04:23:10 c1 flannel-wrapper[1758]: Downloading ACI:  5.2 MB/8.58 MB
Feb 26 04:23:11 c1 flannel-wrapper[1758]: Downloading ACI:  5.31 MB/8.58 MB
Feb 26 04:23:12 c1 flannel-wrapper[1758]: Downloading ACI:  5.41 MB/8.58 MB
Feb 26 04:23:14 c1 flannel-wrapper[1758]: Downloading ACI:  5.52 MB/8.58 MB
Feb 26 04:23:15 c1 flannel-wrapper[1758]: Downloading ACI:  5.62 MB/8.58 MB
Feb 26 04:23:16 c1 flannel-wrapper[1758]: Downloading ACI:  5.73 MB/8.58 MB
Feb 26 04:23:17 c1 flannel-wrapper[1758]: Downloading ACI:  5.83 MB/8.58 MB
Feb 26 04:23:18 c1 flannel-wrapper[1758]: Downloading ACI:  5.92 MB/8.58 MB
Feb 26 04:23:19 c1 flannel-wrapper[1758]: Downloading ACI:  5.99 MB/8.58 MB
Feb 26 04:23:20 c1 flannel-wrapper[1758]: Downloading ACI:  6.06 MB/8.58 MB
Feb 26 04:23:21 c1 flannel-wrapper[1758]: Downloading ACI:  6.14 MB/8.58 MB
Feb 26 04:23:22 c1 flannel-wrapper[1758]: Downloading ACI:  6.18 MB/8.58 MB
Feb 26 04:23:24 c1 flannel-wrapper[1758]: Downloading ACI:  6.21 MB/8.58 MB
Feb 26 04:23:25 c1 flannel-wrapper[1758]: Downloading ACI:  6.23 MB/8.58 MB
Feb 26 04:23:26 c1 flannel-wrapper[1758]: Downloading ACI:  6.27 MB/8.58 MB
Feb 26 04:23:27 c1 flannel-wrapper[1758]: Downloading ACI:  6.32 MB/8.58 MB
Feb 26 04:23:29 c1 flannel-wrapper[1758]: Downloading ACI:  6.37 MB/8.58 MB
Feb 26 04:23:30 c1 flannel-wrapper[1758]: Downloading ACI:  6.44 MB/8.58 MB
Feb 26 04:23:31 c1 flannel-wrapper[1758]: Downloading ACI:  6.53 MB/8.58 MB
Feb 26 04:23:32 c1 flannel-wrapper[1758]: Downloading ACI:  6.63 MB/8.58 MB
Feb 26 04:23:33 c1 flannel-wrapper[1758]: Downloading ACI:  6.79 MB/8.58 MB
Feb 26 04:23:34 c1 flannel-wrapper[1758]: Downloading ACI:  7.03 MB/8.58 MB
Feb 26 04:23:35 c1 flannel-wrapper[1758]: Downloading ACI:  7.21 MB/8.58 MB
Feb 26 04:23:36 c1 flannel-wrapper[1758]: Downloading ACI:  7.35 MB/8.58 MB
Feb 26 04:23:37 c1 flannel-wrapper[1758]: Downloading ACI:  7.52 MB/8.58 MB
Feb 26 04:23:38 c1 flannel-wrapper[1758]: Downloading ACI:  7.66 MB/8.58 MB
Feb 26 04:23:39 c1 flannel-wrapper[1758]: Downloading ACI:  7.8 MB/8.58 MB
Feb 26 04:23:41 c1 flannel-wrapper[1758]: Downloading ACI:  7.92 MB/8.58 MB
Feb 26 04:23:42 c1 flannel-wrapper[1758]: Downloading ACI:  8.04 MB/8.58 MB
Feb 26 04:23:43 c1 flannel-wrapper[1758]: Downloading ACI:  8.18 MB/8.58 MB
Feb 26 04:23:43 c1 systemd[1]: flannel-docker-opts.service: Start operation timed out. Terminating.
Feb 26 04:23:43 c1 systemd[1]: Failed to start flannel docker export service - Network fabric for containers (System Application Container).
Feb 26 04:23:43 c1 systemd[1]: flannel-docker-opts.service: Unit entered failed state.
Feb 26 04:23:43 c1 systemd[1]: flannel-docker-opts.service: Failed with result 'timeout'.

This is because flannel-docker-opts.service has TimeoutStartSec=60 set. See https://github.com/coreos/coreos-overlay/blob/master/app-admin/flannel-wrapper/files/flannel-docker-opts.service

Reproduction Steps

  1. Run vagrant up to start a multi-node vagrant kubernetes cluster from the coreos-kubernetes project.

euank added a commit to euank/coreos-overlay that referenced this issue Apr 27, 2017

app-admin/flannel-wrapper: update timeouts
See coreos/bugs#1833 where this was requested.

This has become more important with the more recent flannel releases
including kubernetes and becoming larger as well.

The value of 5 minutes is arbitrarily chosen as a reasonable increase
over the default 90s.

euank added a commit to euank/coreos-overlay that referenced this issue Apr 27, 2017

app-admin/flannel-wrapper: update timeouts
See coreos/bugs#1833 where this was requested.

This has become more important with the more recent flannel releases
including kubernetes and becoming larger as well.

The value of 5 minutes is arbitrarily chosen as a reasonable increase
over the default 90s.
@euank

This comment has been minimized.

Show comment
Hide comment
@euank

euank Apr 28, 2017

Member

I've removed the timeout from the docker-opts service and bumped the flanneld service up to 5 minutes.

The next alpha release should pick that change up and that should be a large enough default for most/all environments.

Member

euank commented Apr 28, 2017

I've removed the timeout from the docker-opts service and bumped the flanneld service up to 5 minutes.

The next alpha release should pick that change up and that should be a large enough default for most/all environments.

@bgilbert bgilbert closed this May 10, 2017

ChrisMcKenzie added a commit to ChrisMcKenzie/coreos-overlay that referenced this issue Dec 9, 2017

app-admin/flannel-wrapper: update timeouts
See coreos/bugs#1833 where this was requested.

This has become more important with the more recent flannel releases
including kubernetes and becoming larger as well.

The value of 5 minutes is arbitrarily chosen as a reasonable increase
over the default 90s.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment