Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Out of memory while 7GB available #41369

Open
ondrejspilkaABB opened this issue Aug 18, 2020 · 0 comments
Open

Out of memory while 7GB available #41369

ondrejspilkaABB opened this issue Aug 18, 2020 · 0 comments

Comments

@ondrejspilkaABB
Copy link

ondrejspilkaABB commented Aug 18, 2020 •
edited

Description

Docker CE as well as containers running on Ubuntu 18.04 LTS started to expose "out of memory" errors.
But system reports 7GB free to use (lsw as well as htop).

Steps to reproduce the issue:

  1. Installed on industrial PC, i7, 16GBRAM, SSD 1TB
  2. running 3 containers, 1st two is custom Python containers, 2nd is timescale-postgis11
  3. docker as well as containers randomly exhibits "out of memory"
    Describe the results you received:
    Normal operation as on identical system running in Azure VM

Describe the results you expected:
Out of memory random errors

Additional information you deem important (e.g. issue happens only occasionally):
One of the failed containers we installed was Swarmpit.
But also other containers exhibit the same behaviour, like PostgreSQL

This is sample attempt to see Swarmpit log, but dont blame Swarmpit, other commands or containers randomly expose the same.

sudo docker service logs swarmpit_app
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f347b6aaf47 m=9 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f347b6aaf47
stack: frame={sp:0x7f34713cd830, fp:0x0} stack=[0x7f3470bce288,0x7f34713cde88)
00007f34713cd730:  000000c0000c8300  000000c000092380 
00007f34713cd740:  000000c0002f87b0  000000c0002c9b80 
00007f34713cd750:  000000c0002c9a40  000000c0002f8240 
00007f34713cd760:  000000c0002f8270  000000c00029c6c0 
00007f34713cd770:  000000c00029c700  000000c0002c92c0 
00007f34713cd780:  000000c0002c9400  000000c0002c9540 
00007f34713cd790:  000000c0002c97c0  000000c0002c9900 
00007f34713cd7a0:  000000c0000c81b0  000000c0000c81e0 
00007f34713cd7b0:  000000c0002f87b0  000000c0002c9b80 
00007f34713cd7c0:  000000c0002c9a40  000000c0002f8240 
00007f34713cd7d0:  000000c0002f8270  000000c00029c6c0 
00007f34713cd7e0:  000000c00029c700  000000c0002c92c0 
00007f34713cd7f0:  000000c0002c9400  000000c0002c9540 
00007f34713cd800:  000000c0002c97c0  000000c0002c9900 
00007f34713cd810:  000000c0002f84e0  000000c00029c7c0 
00007f34713cd820:  0000000000000000  0000000000000000 
00007f34713cd830: <0000000000000000  0000000000000000 
00007f34713cd840:  0000000000000001  0000000000000000 
00007f34713cd850:  0000000000000000  0000000000000000 
00007f34713cd860:  0000000000000000  0000000000000000 
00007f34713cd870:  0000000000000000  0000000000000002 
00007f34713cd880:  0000000000000002  800000000000000e 
00007f34713cd890:  0000000000000000  0000000000000000 
00007f34713cd8a0:  0000000000000000  0000000000000000 
00007f34713cd8b0:  fffffffe7fffffff  ffffffffffffffff 
00007f34713cd8c0:  ffffffffffffffff  ffffffffffffffff 
00007f34713cd8d0:  ffffffffffffffff  ffffffffffffffff 
00007f34713cd8e0:  ffffffffffffffff  ffffffffffffffff 
00007f34713cd8f0:  ffffffffffffffff  ffffffffffffffff 
00007f34713cd900:  ffffffffffffffff  ffffffffffffffff 
00007f34713cd910:  ffffffffffffffff  ffffffffffffffff 
00007f34713cd920:  ffffffffffffffff  ffffffffffffffff 
runtime: unknown pc 0x7f347b6aaf47
stack: frame={sp:0x7f34713cd830, fp:0x0} stack=[0x7f3470bce288,0x7f34713cde88)
00007f34713cd730:  000000c0000c8300  000000c000092380 
00007f34713cd740:  000000c0002f87b0  000000c0002c9b80 
00007f34713cd750:  000000c0002c9a40  000000c0002f8240 
00007f34713cd760:  000000c0002f8270  000000c00029c6c0 
00007f34713cd770:  000000c00029c700  000000c0002c92c0 
00007f34713cd780:  000000c0002c9400  000000c0002c9540 
00007f34713cd790:  000000c0002c97c0  000000c0002c9900 
00007f34713cd7a0:  000000c0000c81b0  000000c0000c81e0 
00007f34713cd7b0:  000000c0002f87b0  000000c0002c9b80 
00007f34713cd7c0:  000000c0002c9a40  000000c0002f8240 
00007f34713cd7d0:  000000c0002f8270  000000c00029c6c0 
00007f34713cd7e0:  000000c00029c700  000000c0002c92c0 
00007f34713cd7f0:  000000c0002c9400  000000c0002c9540 
00007f34713cd800:  000000c0002c97c0  000000c0002c9900 
00007f34713cd810:  000000c0002f84e0  000000c00029c7c0 
00007f34713cd820:  0000000000000000  0000000000000000 
00007f34713cd830: <0000000000000000  0000000000000000 
00007f34713cd840:  0000000000000001  0000000000000000 
00007f34713cd850:  0000000000000000  0000000000000000 
00007f34713cd860:  0000000000000000  0000000000000000 
00007f34713cd870:  0000000000000000  0000000000000002 
00007f34713cd880:  0000000000000002  800000000000000e 
00007f34713cd890:  0000000000000000  0000000000000000 
00007f34713cd8a0:  0000000000000000  0000000000000000 
00007f34713cd8b0:  fffffffe7fffffff  ffffffffffffffff 
00007f34713cd8c0:  ffffffffffffffff  ffffffffffffffff 
00007f34713cd8d0:  ffffffffffffffff  ffffffffffffffff 
00007f34713cd8e0:  ffffffffffffffff  ffffffffffffffff 
00007f34713cd8f0:  ffffffffffffffff  ffffffffffffffff 
00007f34713cd900:  ffffffffffffffff  ffffffffffffffff 
00007f34713cd910:  ffffffffffffffff  ffffffffffffffff 
00007f34713cd920:  ffffffffffffffff  ffffffffffffffff 

goroutine 1 [semacquire, locked to thread]:
regexp/syntax.(*compiler).inst(...)
	/usr/local/go/src/regexp/syntax/compile.go:173
regexp/syntax.(*compiler).cap(...)
	/usr/local/go/src/regexp/syntax/compile.go:188
regexp/syntax.(*compiler).compile(0xc0004f7c28, 0xc000610070, 0x390000001c)
	/usr/local/go/src/regexp/syntax/compile.go:139 +0x16d8
regexp/syntax.(*compiler).compile(0xc0004f7c28, 0xc000610150, 0xc0004f58a8)
	/usr/local/go/src/regexp/syntax/compile.go:156 +0xb2b
regexp/syntax.(*compiler).compile(0xc0004f7c28, 0xc0005d1e30, 0x0)
	/usr/local/go/src/regexp/syntax/compile.go:138 +0x14f7
regexp/syntax.(*compiler).compile(0xc0004f7c28, 0xc0006101c0, 0x0)
	/usr/local/go/src/regexp/syntax/compile.go:146 +0xdb1
regexp/syntax.(*compiler).compile(0xc0004f7c28, 0xc0006103f0, 0xc0004f5fe0)
	/usr/local/go/src/regexp/syntax/compile.go:154 +0xb03
regexp/syntax.(*compiler).compile(0xc0004f7c28, 0xc0005d1dc0, 0x1a0000000d)
	/usr/local/go/src/regexp/syntax/compile.go:138 +0x14f7
regexp/syntax.(*compiler).compile(0xc0004f7c28, 0xc000610460, 0x1)
	/usr/local/go/src/regexp/syntax/compile.go:146 +0xdb1
regexp/syntax.(*compiler).compile(0xc0004f7c28, 0xc000610a80, 0xc0004f6718)
	/usr/local/go/src/regexp/syntax/compile.go:154 +0xb03
regexp/syntax.(*compiler).compile(0xc0004f7c28, 0xc0005d1d50, 0x2a00000013)
	/usr/local/go/src/regexp/syntax/compile.go:138 +0x14f7
regexp/syntax.(*compiler).compile(0xc0004f7c28, 0xc000610af0, 0xc0004f6be8)
	/usr/local/go/src/regexp/syntax/compile.go:156 +0xb2b
regexp/syntax.(*compiler).compile(0xc0004f7c28, 0xc0005d1c00, 0x2100000010)
	/usr/local/go/src/regexp/syntax/compile.go:138 +0x14f7
regexp/syntax.(*compiler).compile(0xc0004f7c28, 0xc000610b60, 0x2200000011)
	/usr/local/go/src/regexp/syntax/compile.go:142 +0xe42
regexp/syntax.(*compiler).compile(0xc0004f7c28, 0xc0006113b0, 0x2200000004)
	/usr/local/go/src/regexp/syntax/compile.go:154 +0xb03
regexp/syntax.(*compiler).compile(0xc0004f7c28, 0xc000611420, 0xc0004f7588)
	/usr/local/go/src/regexp/syntax/compile.go:163 +0xd22
regexp/syntax.(*compiler).compile(0xc0004f7c28, 0xc0005d1500, 0xc0004f77f0)
	/usr/local/go/src/regexp/syntax/compile.go:138 +0x14f7
regexp/syntax.(*compiler).compile(0xc0004f7c28, 0xc0005d1490, 0x200000001)
	/usr/local/go/src/regexp/syntax/compile.go:138 +0x14f7
regexp/syntax.(*compiler).compile(0xc0004f7c28, 0xc0006127e0, 0xc0004f7c60)
	/usr/local/go/src/regexp/syntax/compile.go:156 +0xb2b
regexp/syntax.Compile(0xc0006127e0, 0xc0006127e0, 0x25, 0x25)
	/usr/local/go/src/regexp/syntax/compile.go:83 +0x10c
regexp.compile(0x5616578d6d58, 0x434, 0xd4, 0xc0005da6e0, 0x0, 0x0)
	/usr/local/go/src/regexp/regexp.go:178 +0xd9
regexp.Compile(...)
	/usr/local/go/src/regexp/regexp.go:133
regexp.MustCompile(0x5616578d6d58, 0x434, 0xc0005da6e0)
	/usr/local/go/src/regexp/regexp.go:309 +0x4d
github.com/docker/cli/vendor/github.com/asaskevich/govalidator.init()
	/go/src/github.com/docker/cli/vendor/github.com/asaskevich/govalidator/patterns.go:66 +0xd2

goroutine 8 [syscall]:
os/signal.signal_recv(0x561656306ab6)
	/usr/local/go/src/runtime/sigqueue.go:147 +0x9e
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x24
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:29 +0x43

goroutine 10 [chan receive]:
github.com/docker/cli/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x56165942a5c0)
	/go/src/github.com/docker/cli/vendor/k8s.io/klog/klog.go:1010 +0x8d
created by github.com/docker/cli/vendor/k8s.io/klog.init.0
	/go/src/github.com/docker/cli/vendor/k8s.io/klog/klog.go:411 +0xd8

rax    0x0
rbx    0x7f347ba58840
rcx    0x7f347b6aaf47
rdx    0x0
rdi    0x2
rsi    0x7f34713cd830
rbp    0x561657aeab54
rsp    0x7f34713cd830
r8     0x0
r9     0x7f34713cd830
r10    0x8
r11    0x246
r12    0x56165a4da3f0
r13    0x0
r14    0x561657a923ac
r15    0x0
rip    0x7f347b6aaf47
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

Output of docker version:

Docker version 19.03.12, build 48a66213fe

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 8
  Running: 5
  Paused: 0
  Stopped: 3
 Images: 7
 Server Version: 19.03.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: qm1qo0fvq7s8edf6cuhjtv0kf
  Is Manager: true
  ClusterID: lwvrpjt0eiwsl0qa0sc8r4uts
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.69.50
  Manager Addresses:
   192.168.69.50:2377
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-42-generic
 Operating System: Ubuntu 18.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.58GiB
 Name: xxx
 ID: QREH:BIN5:4NI5:5OKC:UT7E:F7HX:4HUB:GG3W:AWWA:FLP5:AUTB:BE2V
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
Welcome to Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-42-generic x86_64)

System information as of Tue Aug 18 11:24:46 UTC 2020

System load: 1.12
Usage of /: 18.0% of 114.35GB
Memory usage: 45%
Swap usage: 0%
Processes: 258
Users logged in: 1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ondrejspilkaABB