Unable to exec hugo: pthread_create failed: Resource temporarily unavailable

[Jamesits]

After 1 or 2 builds, following trials will result in full site 404 and logs are here:

caddy[23692]: runtime/cgo: pthread_create failed: Resource temporarily unavailable
caddy[23692]: SIGABRT: abort
caddy[23692]: PC=0x7f22f5ef7428 m=0
caddy[23692]: goroutine 0 [idle]:
caddy[23692]: goroutine 1 [running]:
caddy[23692]: runtime.systemstack_switch()
caddy[23692]:         /usr/lib/go/src/runtime/asm_amd64.s:245 fp=0xc82001a770 sp=0xc82001a768
caddy[23692]: runtime.main()
caddy[23692]:         /usr/lib/go/src/runtime/proc.go:126 +0x62 fp=0xc82001a7c0 sp=0xc82001a770
caddy[23692]: runtime.goexit()
caddy[23692]:         /usr/lib/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc82001a7c8 sp=0xc82001a7c0
caddy[23692]: goroutine 17 [syscall, locked to thread]:
caddy[23692]: runtime.goexit()
caddy[23692]:         /usr/lib/go/src/runtime/asm_amd64.s:1998 +0x1
caddy[23692]: rax    0x0
caddy[23692]: rbx    0x7f22f6286700
caddy[23692]: rcx    0x7f22f5ef7428
caddy[23692]: rdx    0x6
caddy[23692]: rdi    0x5d6c
caddy[23692]: rsi    0x5d6c
caddy[23692]: rbp    0xde500a
caddy[23692]: rsp    0x7ffeb1aca318
caddy[23692]: r8     0x7f22f6287770
caddy[23692]: r9     0x7f22f66c2700
caddy[23692]: r10    0x8
caddy[23692]: r11    0x206
caddy[23692]: r12    0x129b050
caddy[23692]: r13    0xdde684
caddy[23692]: r14    0x0
caddy[23692]: r15    0x8
caddy[23692]: rip    0x7f22f5ef7428
caddy[23692]: rflags 0x206
caddy[23692]: cs     0x33
caddy[23692]: fs     0x0
caddy[23692]: gs     0x0
caddy[23692]: 2016/12/04 11:11:36 exit status 2

Environment: Caddy 0.9.3, Ubuntu 16.04.1 LTS, Linux 4.4.0-49-generic, inside a xen-based VM

● caddy.service - Caddy HTTP/2 web server
   Loaded: loaded (/etc/systemd/system/caddy.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2016-12-04 11:02:01 CST; 14min ago
     Docs: https://caddyserver.com/docs
 Main PID: 23692 (caddy)
    Tasks: 7
   Memory: 13.4M
      CPU: 2.249s
   CGroup: /system.slice/caddy.service
           └─23692 /usr/local/bin/caddy -log stdout -agree=true -conf=/etc/caddy/Caddyfile -root=/var/tmp

Results of ulimit -a:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7850
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7850
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[hacdias]

Hey @Jamesits! I just searched about it and it does not seem a problem with Hugo, nor Caddy, nor Go. Please check this issue on docker to see if it helps please.

[Jamesits]

@hacdias I am not running Caddy inside Docker, though there is Docker service on my machine. As far as I'm concerned this is because caddy has reached some system resource limit, but which limit exactly is not sure. I've checked anything I know but in vain.

[hacdias]

I knew you weren't using Docker. I just sent you that issue because I thought the answers could help.

By the way, when does that happen? When it supposedly runs the hugo binary?

[Jamesits]

This happens randomly. Haven't tried hugo, since it is a production server, usually I just reboot and the problem is gone (temporary). Will try next time. It haven't happened for several days.

[hacdias]

Okay @Jamesits, but let me know if it happens again.

[Jamesits]

For anyone who have a similar problem: try raise max process number from systemd, since systemd ignores other limits (ulimit, sysctl, etc.)

[Service]
# ...
LimitNPROC=256
# ...