Error on build: double free or corruption (out) SIGABRT: abort PC=0x7f7464f01e97 m=0 sigcode=18446744073709551610 signal arrived during cgo execution

[Sarke]

Description

See error below.

Steps to reproduce the issue:

1. sudo docker build -t site-watcher .

Describe the results you received:

double free or corruption (out)
SIGABRT: abort
PC=0x7f3e05499e97 m=0 sigcode=18446744073709551610
signal arrived during cgo execution

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x4afd50, 0xc42004fcc0, 0xc42004fce8)
	/usr/lib/go-1.8/src/runtime/cgocall.go:131 +0xe2 fp=0xc42004fc90 sp=0xc42004fc50
github.com/docker/docker-credential-helpers/secretservice._Cfunc_free(0xc6a920)
	github.com/docker/docker-credential-helpers/secretservice/_obj/_cgo_gotypes.go:111 +0x41 fp=0xc42004fcc0 sp=0xc42004fc90
github.com/docker/docker-credential-helpers/secretservice.Secretservice.List.func5(0xc6a920)
	/build/golang-github-docker-docker-credential-helpers-cMhSy1/golang-github-docker-docker-credential-helpers-0.5.0/obj-x86_64-linux-gnu/src/github.com/docker/docker-credential-helpers/secretservice/secretservice_linux.go:96 +0x60 fp=0xc42004fcf8 sp=0xc42004fcc0
github.com/docker/docker-credential-helpers/secretservice.Secretservice.List(0x0, 0x756060, 0xc4200123c0)
	/build/golang-github-docker-docker-credential-helpers-cMhSy1/golang-github-docker-docker-credential-helpers-0.5.0/obj-x86_64-linux-gnu/src/github.com/docker/docker-credential-helpers/secretservice/secretservice_linux.go:97 +0x217 fp=0xc42004fda0 sp=0xc42004fcf8
github.com/docker/docker-credential-helpers/secretservice.(*Secretservice).List(0x77e548, 0xc42004fe88, 0x410022, 0xc420012320)
	<autogenerated>:4 +0x46 fp=0xc42004fde0 sp=0xc42004fda0
github.com/docker/docker-credential-helpers/credentials.List(0x756ba0, 0x77e548, 0x7560e0, 0xc42000e018, 0x0, 0x10)
	/build/golang-github-docker-docker-credential-helpers-cMhSy1/golang-github-docker-docker-credential-helpers-0.5.0/obj-x86_64-linux-gnu/src/github.com/docker/docker-credential-helpers/credentials/credentials.go:145 +0x3e fp=0xc42004fe68 sp=0xc42004fde0
github.com/docker/docker-credential-helpers/credentials.HandleCommand(0x756ba0, 0x77e548, 0x7ffd71137855, 0x4, 0x7560a0, 0xc42000e010, 0x7560e0, 0xc42000e018, 0x40e398, 0x4d35c0)
	/build/golang-github-docker-docker-credential-helpers-cMhSy1/golang-github-docker-docker-credential-helpers-0.5.0/obj-x86_64-linux-gnu/src/github.com/docker/docker-credential-helpers/credentials/credentials.go:60 +0x16d fp=0xc42004fed8 sp=0xc42004fe68
github.com/docker/docker-credential-helpers/credentials.Serve(0x756ba0, 0x77e548)
	/build/golang-github-docker-docker-credential-helpers-cMhSy1/golang-github-docker-docker-credential-helpers-0.5.0/obj-x86_64-linux-gnu/src/github.com/docker/docker-credential-helpers/credentials/credentials.go:41 +0x1cb fp=0xc42004ff58 sp=0xc42004fed8
main.main()
	/build/golang-github-docker-docker-credential-helpers-cMhSy1/golang-github-docker-docker-credential-helpers-0.5.0/secretservice/cmd/main_linux.go:9 +0x4f fp=0xc42004ff88 sp=0xc42004ff58
runtime.main()
	/usr/lib/go-1.8/src/runtime/proc.go:185 +0x20a fp=0xc42004ffe0 sp=0xc42004ff88
runtime.goexit()
	/usr/lib/go-1.8/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42004ffe8 sp=0xc42004ffe0

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/usr/lib/go-1.8/src/runtime/asm_amd64.s:2197 +0x1

rax    0x0
rbx    0x7ffd711368c0
rcx    0x7f3e05499e97
rdx    0x0
rdi    0x2
rsi    0x7ffd71136650
rbp    0x7ffd711369c0
rsp    0x7ffd71136650
r8     0x0
r9     0x7ffd71136650
r10    0x8
r11    0x246
r12    0x7ffd711368c0
r13    0x1000
r14    0x0
r15    0x30
rip    0x7f3e05499e97
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
Sending build context to Docker daemon  18.43kB
Step 1/8 : FROM node:10-slim
 ---> c33688dea543

Rest of build proceeds normally.

Describe the results you expected:
Not seeing the error info.

Additional information you deem important (e.g. issue happens only occasionally):
It takes about 20 seconds before the output above appears. It happens every time, and I've tried with different projects.

Output of docker version:

Docker version 18.06.1-ce, build e68fc7a

same thing happened with an earlier version

Docker version 17.12.1-ce, build 7390fc6

Output of docker info:

Containers: 2
 Running: 1
 Paused: 0
 Stopped: 1
Images: 15
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-33-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.35GiB
Name: peter-office
ID: Y2JB:LSMD:DZPK:E5UG:T6ZW:4SD6:FYVT:ZFDO:KKEV:3ADM:2MHP:KNGX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
physical

[Sarke]

This appears to be the same issue: docker/docker-credential-helpers#103

[Sarke]

@aidanh010 found the problem:
docker/docker-credential-helpers#103 (comment)

$ sudo apt remove golang-docker-credential-helpers
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  docker-compose golang-docker-credential-helpers python-docker
  python-dockerpycreds
0 upgraded, 0 newly installed, 4 to remove and 0 not upgraded.
After this operation, 2,438 kB disk space will be freed.
Do you want to continue? [Y/n] 
(Reading database ... 276129 files and directories currently installed.)
Removing docker-compose (1.17.1-2) ...
Removing python-docker (2.5.1-1) ...
Removing python-dockerpycreds (0.2.1-1) ...
Removing golang-docker-credential-helpers (0.5.0-2) ...
Processing triggers for man-db (2.8.3-2) ...

Build runs fine after that.

Shame it removes docker-compose as well.

[thaJeztah]

@Sarke did you install your version of docker from the official docker package repositories, or from the distro (Ubuntu) repository? The golang-docker-credential-helpers package is not part of the official docker packages, so wondering if your docker version is also built by the distro packagers, and something is funky with those builds

[Sarke]

@thaJeztah for docker I used the official https://download.docker.com/linux/ubuntu bionic repo, as per the documentation.

However, for docker-compose, which depends on golang-docker-credential-helpers, it comes from the official Ubuntu repo.

$ apt-cache showpkg golang-docker-credential-helpers

Package: golang-docker-credential-helpers
Versions: 
0.5.0-2 (/var/lib/apt/lists/mirror.it.ubc.ca_ubuntu_dists_bionic_universe_binary-amd64_Packages) (/var/lib/dpkg/status)
 Description Language: 
                 File: /var/lib/apt/lists/mirror.it.ubc.ca_ubuntu_dists_bionic_universe_binary-amd64_Packages
                  MD5: 644749d414f966cf290307b53116aed8
 Description Language: 
                 File: /var/lib/apt/lists/mirror.it.ubc.ca_ubuntu_dists_bionic_universe_binary-i386_Packages
                  MD5: 644749d414f966cf290307b53116aed8
 Description Language: en
                 File: /var/lib/apt/lists/mirror.it.ubc.ca_ubuntu_dists_bionic_universe_i18n_Translation-en
                  MD5: 644749d414f966cf290307b53116aed8


Reverse Depends: 
  python-dockerpycreds,golang-docker-credential-helpers
  golang-docker-credential-helpers:i386,golang-docker-credential-helpers
  python3-dockerpycreds,golang-docker-credential-helpers
Dependencies: 
0.5.0-2 - libc6 (2 2.14) libglib2.0-0 (2 2.28.0) libsecret-1-0 (2 0.7) golang-docker-credential-helpers:i386 (32 (null)) 
Provides: 
0.5.0-2 - 
Reverse Provides: 

$ dpkg -s golang-docker-credential-helpers

Package: golang-docker-credential-helpers
Status: install ok installed
Priority: extra
Section: devel
Installed-Size: 1435
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Source: golang-github-docker-docker-credential-helpers
Version: 0.5.0-2
Depends: libc6 (>= 2.14), libglib2.0-0 (>= 2.28.0), libsecret-1-0 (>= 0.7)
Description: Use native stores to safeguard Docker credentials
 The docker-credential-helpers package is a suite of programs to use
 the built-in OS native stores to keep Docker credentials safe.
 .
 This package contains the command-line executables.
Original-Maintainer: Debian Go Packaging Team <pkg-go-maintainers@lists.alioth.debian.org>
Built-Using: golang-1.8 (= 1.8.1-1ubuntu3)
Homepage: https://github.com/docker/docker-credential-helpers

[Sarke]

Installing docker-compose using pip seems to work fine, and doesn't cause the golang-docker-credential-helpers deb to be installed. So all is well, but it's a workaround.

$ pip install docker-compose

Collecting docker-compose
  Downloading https://files.pythonhosted.org/packages/67/03/b833b571595e05c933d3af3685be3b27b1166c415d005b3eadaa5be80d25/docker_compose-1.22.0-py2.py3-none-any.whl (126kB)
    100% |████████████████████████████████| 133kB 2.0MB/s 
Collecting six<2,>=1.3.0 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting backports.ssl-match-hostname>=3.5; python_version < "3.5" (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/76/21/2dc61178a2038a5cb35d14b61467c6ac632791ed05131dda72c20e7b9e23/backports.ssl_match_hostname-3.5.0.1.tar.gz
Collecting docopt<0.7,>=0.6.1 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz
Collecting docker<4.0,>=3.4.1 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/06/0b/ce97eb31058eddaef316973b8299b737ebab0bcb5798fd5a3225d53b4455/docker-3.5.0-py2.py3-none-any.whl (125kB)
    100% |████████████████████████████████| 133kB 2.0MB/s 
Collecting ipaddress>=1.0.16; python_version < "3.3" (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
Collecting PyYAML<4,>=3.10 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz (270kB)
    100% |████████████████████████████████| 276kB 2.0MB/s 
Collecting dockerpty<0.5,>=0.4.1 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/8d/ee/e9ecce4c32204a6738e0a5d5883d3413794d7498fe8b06f44becc028d3ba/dockerpty-0.4.1.tar.gz
Collecting requests!=2.11.0,!=2.12.2,!=2.18.0,<2.19,>=2.6.1 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/49/df/50aa1999ab9bde74656c2919d9c0c085fd2b3775fd3eca826012bef76d8c/requests-2.18.4-py2.py3-none-any.whl (88kB)
    100% |████████████████████████████████| 92kB 2.5MB/s 
Collecting enum34<2,>=1.0.4; python_version < "3.4" (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Collecting cached-property<2,>=1.2.0 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/3b/86/85c1be2e8db9e13ef9a350aecd6dea292bd612fa288c2f40d035bb750ded/cached_property-1.5.1-py2.py3-none-any.whl
Collecting jsonschema<3,>=2.5.1 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/77/de/47e35a97b2b05c2fadbec67d44cfcdcd09b8086951b331d82de90d2912da/jsonschema-2.6.0-py2.py3-none-any.whl
Collecting texttable<0.10,>=0.9.0 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/02/e1/2565e6b842de7945af0555167d33acfc8a615584ef7abd30d1eae00a4d80/texttable-0.9.1.tar.gz
Collecting websocket-client<1.0,>=0.32.0 (from docker-compose)
  Downloading https://files.pythonhosted.org/packages/14/d4/6a8cd4e7f67da465108c7cc0a307a1c5da7e2cdf497330b682069b1d4758/websocket_client-0.53.0-py2.py3-none-any.whl (198kB)
    100% |████████████████████████████████| 204kB 1.8MB/s 
Collecting docker-pycreds>=0.3.0 (from docker<4.0,>=3.4.1->docker-compose)
  Downloading https://files.pythonhosted.org/packages/ea/bf/7e70aeebc40407fbdb96fa9f79fc8e4722ea889a99378303e3bcc73f4ab5/docker_pycreds-0.3.0-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests!=2.11.0,!=2.12.2,!=2.18.0,<2.19,>=2.6.1->docker-compose)
  Downloading https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 1.6MB/s 
Collecting urllib3<1.23,>=1.21.1 (from requests!=2.11.0,!=2.12.2,!=2.18.0,<2.19,>=2.6.1->docker-compose)
  Downloading https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl (132kB)
    100% |████████████████████████████████| 133kB 1.7MB/s 
Collecting certifi>=2017.4.17 (from requests!=2.11.0,!=2.12.2,!=2.18.0,<2.19,>=2.6.1->docker-compose)
  Downloading https://files.pythonhosted.org/packages/df/f7/04fee6ac349e915b82171f8e23cee63644d83663b34c539f7a09aed18f9e/certifi-2018.8.24-py2.py3-none-any.whl (147kB)
    100% |████████████████████████████████| 153kB 2.7MB/s 
Collecting chardet<3.1.0,>=3.0.2 (from requests!=2.11.0,!=2.12.2,!=2.18.0,<2.19,>=2.6.1->docker-compose)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 2.6MB/s 
Collecting functools32; python_version == "2.7" (from jsonschema<3,>=2.5.1->docker-compose)
  Downloading https://files.pythonhosted.org/packages/c5/60/6ac26ad05857c601308d8fb9e87fa36d0ebf889423f47c3502ef034365db/functools32-3.2.3-2.tar.gz
Building wheels for collected packages: backports.ssl-match-hostname, docopt, PyYAML, dockerpty, texttable, functools32
  Running setup.py bdist_wheel for backports.ssl-match-hostname ... done
  Stored in directory: /home/peter/.cache/pip/wheels/99/7e/f7/a88a9bcf7a3bd6b12cf6a74eee8c89746aaa02f71ab7b33939
  Running setup.py bdist_wheel for docopt ... done
  Stored in directory: /home/peter/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e
  Running setup.py bdist_wheel for PyYAML ... done
  Stored in directory: /home/peter/.cache/pip/wheels/ad/da/0c/74eb680767247273e2cf2723482cb9c924fe70af57c334513f
  Running setup.py bdist_wheel for dockerpty ... done
  Stored in directory: /home/peter/.cache/pip/wheels/e5/1e/86/bd0a97a0907c6c654af654d5875d1d4383dd1f575f77cee4aa
  Running setup.py bdist_wheel for texttable ... done
  Stored in directory: /home/peter/.cache/pip/wheels/99/1e/2b/8452d3a48dad98632787556a0f2f90d56703b39cdf7d142dd1
  Running setup.py bdist_wheel for functools32 ... done
  Stored in directory: /home/peter/.cache/pip/wheels/b5/18/32/77a1030457155606ba5e3ec3a8a57132b1a04b1c4f765177b2
Successfully built backports.ssl-match-hostname docopt PyYAML dockerpty texttable functools32
Installing collected packages: six, backports.ssl-match-hostname, docopt, docker-pycreds, ipaddress, idna, urllib3, certifi, chardet, requests, websocket-client, docker, PyYAML, dockerpty, enum34, cached-property, functools32, jsonschema, texttable, docker-compose
Successfully installed PyYAML-3.13 backports.ssl-match-hostname-3.5.0.1 cached-property-1.5.1 certifi-2018.8.24 chardet-3.0.4 docker-3.5.0 docker-compose-1.22.0 docker-pycreds-0.3.0 dockerpty-0.4.1 docopt-0.6.2 enum34-1.1.6 functools32-3.2.3.post2 idna-2.6 ipaddress-1.0.22 jsonschema-2.6.0 requests-2.18.4 six-1.11.0 texttable-0.9.1 urllib3-1.22 websocket-client-0.53.0

$ docker-compose --version

docker-compose version 1.22.0, build f46880f

[aidanbh]

Filed a bug with Ubuntu a few weeks ago, still waiting for response: https://bugs.launchpad.net/ubuntu/+source/docker-compose/+bug/1792824

[thaJeztah]

Looks like a fix for this was merged in the docker-credentials-helpers repository, but is not yet in the v0.6.1 release; docker/docker-credential-helpers#29

[thaJeztah]

docker/docker-credential-helpers#29 was merged, and is part of v0.6.2 of the credential helpers