Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flanneld timing out while downloading image #1833

Closed
F21 opened this issue Feb 26, 2017 · 1 comment
Closed

Flanneld timing out while downloading image #1833

F21 opened this issue Feb 26, 2017 · 1 comment

Comments

@F21
Copy link

@F21 F21 commented Feb 26, 2017

Issue Report

Bug

Container Linux Version

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1325.1.0
VERSION_ID=1325.1.0
BUILD_ID=2017-02-23-0035
PRETTY_NAME="Container Linux by CoreOS 1325.1.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

Environment

Vagrant and VirtualBox 5.1.14

Expected Behavior

Flanneld should not timeout while downloading the image

Actual Behavior

The Flanneld image does not download completely before timeout out (due to a slower connection). This causes flannel-docker-opts.service to fail.

 $ journalctl -u  flannel-docker-opts.service
-- Logs begin at Sun 2017-02-26 04:21:43 UTC, end at Sun 2017-02-26 05:13:08 UTC. --
Feb 26 04:22:43 c1 systemd[1]: Starting flannel docker export service - Network fabric for containers (System Application Container)...
Feb 26 04:22:43 c1 rkt[1728]: rm: unable to resolve UUID from file: open /var/lib/coreos/flannel-wrapper2.uuid: no such file or directory
Feb 26 04:22:43 c1 rkt[1728]: rm: failed to remove one or more pods
Feb 26 04:22:43 c1 flannel-wrapper[1758]: + exec /usr/bin/rkt run --uuid-file-save=/var/lib/coreos/flannel-wrapper2.uuid --trust-keys-from-https --net=host --volume run-flannel,kind=host,source=/run/flannel,readOnly=false --volume etc-ss
Feb 26 04:22:45 c1 flannel-wrapper[1758]: image: keys already exist for prefix "quay.io/coreos/flannel", not fetching again
Feb 26 04:22:47 c1 flannel-wrapper[1758]: Downloading signature:  0 B/473 B
Feb 26 04:22:47 c1 flannel-wrapper[1758]: Downloading signature:  473 B/473 B
Feb 26 04:22:47 c1 flannel-wrapper[1758]: Downloading signature:  473 B/473 B
Feb 26 04:22:49 c1 flannel-wrapper[1758]: Downloading ACI:  0 B/8.58 MB
Feb 26 04:22:49 c1 flannel-wrapper[1758]: Downloading ACI:  16.3 KB/8.58 MB
Feb 26 04:22:50 c1 flannel-wrapper[1758]: Downloading ACI:  138 KB/8.58 MB
Feb 26 04:22:51 c1 flannel-wrapper[1758]: Downloading ACI:  399 KB/8.58 MB
Feb 26 04:22:52 c1 flannel-wrapper[1758]: Downloading ACI:  799 KB/8.58 MB
Feb 26 04:22:53 c1 flannel-wrapper[1758]: Downloading ACI:  1.15 MB/8.58 MB
Feb 26 04:22:54 c1 flannel-wrapper[1758]: Downloading ACI:  1.32 MB/8.58 MB
Feb 26 04:22:55 c1 flannel-wrapper[1758]: Downloading ACI:  1.65 MB/8.58 MB
Feb 26 04:22:56 c1 flannel-wrapper[1758]: Downloading ACI:  1.95 MB/8.58 MB
Feb 26 04:22:57 c1 flannel-wrapper[1758]: Downloading ACI:  2.24 MB/8.58 MB
Feb 26 04:22:58 c1 flannel-wrapper[1758]: Downloading ACI:  2.56 MB/8.58 MB
Feb 26 04:22:59 c1 flannel-wrapper[1758]: Downloading ACI:  2.87 MB/8.58 MB
Feb 26 04:23:00 c1 flannel-wrapper[1758]: Downloading ACI:  3.18 MB/8.58 MB
Feb 26 04:23:01 c1 flannel-wrapper[1758]: Downloading ACI:  3.51 MB/8.58 MB
Feb 26 04:23:02 c1 flannel-wrapper[1758]: Downloading ACI:  3.81 MB/8.58 MB
Feb 26 04:23:03 c1 flannel-wrapper[1758]: Downloading ACI:  4.05 MB/8.58 MB
Feb 26 04:23:04 c1 flannel-wrapper[1758]: Downloading ACI:  4.32 MB/8.58 MB
Feb 26 04:23:06 c1 flannel-wrapper[1758]: Downloading ACI:  4.51 MB/8.58 MB
Feb 26 04:23:07 c1 flannel-wrapper[1758]: Downloading ACI:  4.73 MB/8.58 MB
Feb 26 04:23:08 c1 flannel-wrapper[1758]: Downloading ACI:  4.92 MB/8.58 MB
Feb 26 04:23:09 c1 flannel-wrapper[1758]: Downloading ACI:  5.08 MB/8.58 MB
Feb 26 04:23:10 c1 flannel-wrapper[1758]: Downloading ACI:  5.2 MB/8.58 MB
Feb 26 04:23:11 c1 flannel-wrapper[1758]: Downloading ACI:  5.31 MB/8.58 MB
Feb 26 04:23:12 c1 flannel-wrapper[1758]: Downloading ACI:  5.41 MB/8.58 MB
Feb 26 04:23:14 c1 flannel-wrapper[1758]: Downloading ACI:  5.52 MB/8.58 MB
Feb 26 04:23:15 c1 flannel-wrapper[1758]: Downloading ACI:  5.62 MB/8.58 MB
Feb 26 04:23:16 c1 flannel-wrapper[1758]: Downloading ACI:  5.73 MB/8.58 MB
Feb 26 04:23:17 c1 flannel-wrapper[1758]: Downloading ACI:  5.83 MB/8.58 MB
Feb 26 04:23:18 c1 flannel-wrapper[1758]: Downloading ACI:  5.92 MB/8.58 MB
Feb 26 04:23:19 c1 flannel-wrapper[1758]: Downloading ACI:  5.99 MB/8.58 MB
Feb 26 04:23:20 c1 flannel-wrapper[1758]: Downloading ACI:  6.06 MB/8.58 MB
Feb 26 04:23:21 c1 flannel-wrapper[1758]: Downloading ACI:  6.14 MB/8.58 MB
Feb 26 04:23:22 c1 flannel-wrapper[1758]: Downloading ACI:  6.18 MB/8.58 MB
Feb 26 04:23:24 c1 flannel-wrapper[1758]: Downloading ACI:  6.21 MB/8.58 MB
Feb 26 04:23:25 c1 flannel-wrapper[1758]: Downloading ACI:  6.23 MB/8.58 MB
Feb 26 04:23:26 c1 flannel-wrapper[1758]: Downloading ACI:  6.27 MB/8.58 MB
Feb 26 04:23:27 c1 flannel-wrapper[1758]: Downloading ACI:  6.32 MB/8.58 MB
Feb 26 04:23:29 c1 flannel-wrapper[1758]: Downloading ACI:  6.37 MB/8.58 MB
Feb 26 04:23:30 c1 flannel-wrapper[1758]: Downloading ACI:  6.44 MB/8.58 MB
Feb 26 04:23:31 c1 flannel-wrapper[1758]: Downloading ACI:  6.53 MB/8.58 MB
Feb 26 04:23:32 c1 flannel-wrapper[1758]: Downloading ACI:  6.63 MB/8.58 MB
Feb 26 04:23:33 c1 flannel-wrapper[1758]: Downloading ACI:  6.79 MB/8.58 MB
Feb 26 04:23:34 c1 flannel-wrapper[1758]: Downloading ACI:  7.03 MB/8.58 MB
Feb 26 04:23:35 c1 flannel-wrapper[1758]: Downloading ACI:  7.21 MB/8.58 MB
Feb 26 04:23:36 c1 flannel-wrapper[1758]: Downloading ACI:  7.35 MB/8.58 MB
Feb 26 04:23:37 c1 flannel-wrapper[1758]: Downloading ACI:  7.52 MB/8.58 MB
Feb 26 04:23:38 c1 flannel-wrapper[1758]: Downloading ACI:  7.66 MB/8.58 MB
Feb 26 04:23:39 c1 flannel-wrapper[1758]: Downloading ACI:  7.8 MB/8.58 MB
Feb 26 04:23:41 c1 flannel-wrapper[1758]: Downloading ACI:  7.92 MB/8.58 MB
Feb 26 04:23:42 c1 flannel-wrapper[1758]: Downloading ACI:  8.04 MB/8.58 MB
Feb 26 04:23:43 c1 flannel-wrapper[1758]: Downloading ACI:  8.18 MB/8.58 MB
Feb 26 04:23:43 c1 systemd[1]: flannel-docker-opts.service: Start operation timed out. Terminating.
Feb 26 04:23:43 c1 systemd[1]: Failed to start flannel docker export service - Network fabric for containers (System Application Container).
Feb 26 04:23:43 c1 systemd[1]: flannel-docker-opts.service: Unit entered failed state.
Feb 26 04:23:43 c1 systemd[1]: flannel-docker-opts.service: Failed with result 'timeout'.

This is because flannel-docker-opts.service has TimeoutStartSec=60 set. See https://github.com/coreos/coreos-overlay/blob/master/app-admin/flannel-wrapper/files/flannel-docker-opts.service

Reproduction Steps

  1. Run vagrant up to start a multi-node vagrant kubernetes cluster from the coreos-kubernetes project.
euank added a commit to euank/coreos-overlay that referenced this issue Apr 27, 2017
See coreos/bugs#1833 where this was requested.

This has become more important with the more recent flannel releases
including kubernetes and becoming larger as well.

The value of 5 minutes is arbitrarily chosen as a reasonable increase
over the default 90s.
euank added a commit to euank/coreos-overlay that referenced this issue Apr 27, 2017
See coreos/bugs#1833 where this was requested.

This has become more important with the more recent flannel releases
including kubernetes and becoming larger as well.

The value of 5 minutes is arbitrarily chosen as a reasonable increase
over the default 90s.
@euank
Copy link
Contributor

@euank euank commented Apr 28, 2017

I've removed the timeout from the docker-opts service and bumped the flanneld service up to 5 minutes.

The next alpha release should pick that change up and that should be a large enough default for most/all environments.

@bgilbert bgilbert closed this May 10, 2017
ChrisMcKenzie added a commit to ChrisMcKenzie/coreos-overlay that referenced this issue Dec 9, 2017
See coreos/bugs#1833 where this was requested.

This has become more important with the more recent flannel releases
including kubernetes and becoming larger as well.

The value of 5 minutes is arbitrarily chosen as a reasonable increase
over the default 90s.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.