Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LCOW: nightly build regression has broken apt-get install in docker build #39227

Closed
Iristyle opened this issue May 16, 2019 · 2 comments
Closed

LCOW: nightly build regression has broken apt-get install in docker build #39227

Iristyle opened this issue May 16, 2019 · 2 comments
Labels
area/builder area/lcow Issues and PR's related to the experimental LCOW feature kind/experimental

Comments

@Iristyle
Copy link

Iristyle commented May 16, 2019 •
edited

Description

Putting a placeholder here for now until more info can be collected. Something in between the 2019-04-28 nightly and 2019-05-15 nightly breaks builds of Ubuntu based containers in LCOW with the infamous Go networking error docker : net/http: request canceled (Client.Timeout exceeded while reading body) discussed in issues:

docker/for-win#611
#23910

Reverting the daemon back to the previous known good working nightly has resolved this issue for now.

Steps to reproduce the issue:

  1. Run a docker build over a Dockerfile with a step like apt-get install

Describe the results you received:

Step 12/28 : RUN apt-get update &&     apt-get install -y --no-install-recommends wget=1.17.1-1ubuntu1 ca-certificates net-tools apt-transport-https &&     wget --quiet --output-document puppet-apt-key https://artifactory.delivery.puppetlabs.net/artifactory/api/gpg/key/public &&     apt-key add puppet-apt-key &&     echo "deb https://artifactory.delivery.puppetlabs.net/artifactory/debian_enterprise__local xenial ${PE_VERSION}" > /etc/apt/sources.list.d/artifactory.list &&     apt-get update &&     apt-get install -y --no-install-recommends pe-console-services="${version}-1xenial" &&     rm /etc/apt/sources.list.d/artifactory.list &&     rm puppet-apt-key &&    apt-get clean &&     rm -rf /var/lib/apt/lists/* &&     touch /etc/puppetlabs/console-services/rbac-certificate-whitelist &&     mkdir -p /etc/puppetlabs/puppet/ssl/certs &&     mkdir -p /opt/puppetlabs/server/data/console-services/certs &&     mkdir -p /etc/puppetlabs/console-services/conf.d
 ---> Running in 4680915a9f18
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:3 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [827 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Get:6 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1558 kB]
Get:7 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [12.7 kB]
Get:8 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [553 kB]
Get:9 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [6113 B]
Get:10 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages [14.1 kB]
Get:11 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [9827 kB]
##[error]docker : net/http: request canceled (Client.Timeout exceeded while reading body)

The failures were repeatable and occur at different points in the package installs.

Additional information you deem important (e.g. issue happens only occasionally):

Issue is reproducible.

LCOW image is based on linuxkit/lcow@d5dfdbc

I did try adding an override of 8.8.8.8 for dns to daemon.json as discussed in some of the similar issues. That did not resolve the problem.

For erroring configuration

Output of docker version:

Client:
 Version:           master-dockerproject-2019-05-15
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        4c605bdc
 Built:             Wed May 15 23:48:54 2019
 OS/Arch:           windows/amd64
 Experimental:      false

Server:
 Engine:
  Version:          master-dockerproject-2019-05-15
  API version:      1.41 (minimum version 1.24)
  Go version:       go1.12.5
  Git commit:       34b5672
  Built:            Thu May 16 00:00:13 2019
  OS/Arch:          windows/amd64
  Experimental:     true

Output of docker info:

Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0-beta2)
  buildx: Build with BuildKit (Docker Inc., v0.2.0-32-g715d38f-tp)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 261
 Server Version: master-dockerproject-2019-05-15
 Storage Driver: windowsfilter (windows) lcow (linux)
  Windows: 
  LCOW: 
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics l2bridge l2tunnel nat null overlay transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: hyperv
 Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
 Operating System: Windows 10 Enterprise Version 1809 (OS Build 17763.503)
 OSType: windows
 Architecture: x86_64
 CPUs: 2
 Total Memory: 16GiB
 Name: ci-lcow-prod-1
 ID: 0ac02c9d-aaba-42f4-8749-5a64af3068d8
 Docker Root Dir: C:\ProgramData\docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

For working configuration

Output of docker version:

Client:
 Version:           master-dockerproject-2019-04-28
 API version:       1.40
 Go version:        go1.12.4
 Git commit:        55879706
 Built:             Sun Apr 28 23:48:06 2019
 OS/Arch:           windows/amd64
 Experimental:      false

Server:
 Engine:
  Version:          master-dockerproject-2019-04-28
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.12.4
  Git commit:       0c07dad
  Built:            Sun Apr 28 23:59:25 2019
  OS/Arch:          windows/amd64
  Experimental:     true

Output of docker info:

Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0-beta2)
  buildx: Build with BuildKit (Docker Inc., v0.2.0-6-g509c4b6-tp)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 271
 Server Version: master-dockerproject-2019-04-28
 Storage Driver: windowsfilter (windows) lcow (linux)
  Windows:
  LCOW:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics l2bridge l2tunnel nat null overlay transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: hyperv
 Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
 Operating System: Windows 10 Enterprise Version 1809 (OS Build 17763.503)
 OSType: windows
 Architecture: x86_64
 CPUs: 2
 Total Memory: 16GiB
 Name: ci-lcow-prod-1
 ID: 0ac02c9d-aaba-42f4-8749-5a64af3068d8
 Docker Root Dir: C:\ProgramData\docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

This is an LCOW CI node

/cc @jhowardmsft
@Iristyle Iristyle changed the title LCOW: nightly builds have broken apt-get install in docker build LCOW: nightly build regression has broken apt-get install in docker build May 16, 2019
@Iristyle
Copy link
Author

Iristyle commented May 17, 2019 •
edited

I tried to figure out which commit made things go sideways by using git bisect and following the build procedure defined in https://github.com/moby/moby/blob/master/Dockerfile.windows

I produced binaries for a number of the SHAs during the bisect (including the failing nightly SHA 34b5672) and none of them failed:

34b5672
34cc5c2
f4b9142
de6df46
3042254
ee39952

Strangely, every dockerd.exe produced was exactly the same size (71584 KB / 73,302,016 bytes), yet they all showed the specified SHA I used to build with docker run --name binaries -e DOCKER_GITCOMMIT=$DOCKER_GITCOMMIT nativebuildimage hack\make.ps1 -Binary when I ran docker version - for instance:

Client:
 Version:      17.06.2-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   cec0b72
 Built:        Tue Sep  5 19:57:19 2017
 OS/Arch:      windows/amd64

Server:
 Version:      0.0.0-dev
 API version:  1.41 (minimum version 1.24)
 Go version:   go1.12.5
 Git commit:   34b56728ed
 Built:        05/17/2019 00:17:29
 OS/Arch:      windows/amd64
 Experimental: true

The nightly build of dockerd.exe from the zip file at https://master.dockerproject.com/windows/x86_64/docker.zip is a totally different size - 59455 KB. Not sure if that's a result of being cross-compiled on Linux or if there's something that isn't behaving properly with the builds I'm producing. For certain the file sizes seem suspect.

UPDATE: It's possible I was producing incorrect builds if there were git subrepos I was supposed to check out while performing the bisect.

@Iristyle Iristyle mentioned this issue Jun 19, 2019
Open
@Iristyle
Copy link
Author

More recent nightlies from 7-31-2019 seem to have removed these problems.

I haven't verified the problems I've gotten with rebuilding things, but that's a separate issue anyhow.

@thaJeztah thaJeztah added area/lcow Issues and PR's related to the experimental LCOW feature kind/experimental labels Aug 30, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/builder area/lcow Issues and PR's related to the experimental LCOW feature kind/experimental
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Iristyle @thaJeztah @GordonTheTurtle