Caddy panics on request, "failed to create new OS thread" and "may need to increase max user processes"

[Southclaws]

1. What version of Caddy are you using (caddy -version)?

0.10.3 - was using 0.10.2 and installed the latest yesterday hoping it would resolve this issue

2. What are you trying to do?

Run Caddy as a reverse proxy to some services and basic static site host for a couple of extremely small, single-page portfolios, it was working well for months then randomly decided to start dying on requests.

The machine is one of these: https://documentation.online.net/en/dedicated-server/offers/2016/server-dedibox-classic-2016 and is pretty under-used in terms of resources, the largest apps I have running on there are some Java services (Confluence, Bitbucket and Jira - which I use Caddy to proxy to, successfully for a few months)

I stripped away all the reverse-proxies and the issue is still happening on a single basic static root site, sometimes it loads once but most of the time it just panics as soon as the request is sent.

I found one other post about this issue that mentioned ulimit -u (mine is 128150, no idea what's considered "normal" but this was what my Ubuntu 16.04 installation came with by default)

3. What is your entire Caddyfile?

The stripped down version is as simple as it gets, the page loads sometimes (and it loaded every single time, consistently, up until recently) so I know the directories are all set up correctly.

https://southcla.ws {
        root /var/www/southcla.ws
}

4. How did you run Caddy (give the full command and describe the execution environment)?

The stock systemd unit file

5. Please paste any relevant HTTP request(s) here.

Just a simple curl https://southcla.ws or browser load causes this.

6. What did you expect to see?

A working site.

7. What did you see instead (give full error messages and/or log)?

Jun 18 16:28:02 southclaws caddy[27042]: Activating privacy features... done.
Jun 18 16:28:02 southclaws caddy[27042]: https://southcla.ws
Jun 18 16:28:02 southclaws caddy[27042]: 2017/06/18 16:28:02 https://southcla.ws
Jun 18 16:28:02 southclaws caddy[27042]: http://southcla.ws
Jun 18 16:28:02 southclaws caddy[27042]: 2017/06/18 16:28:02 http://southcla.ws
(I curl or browse to https://southcla.ws to trigger the panic)
Jun 18 16:28:07 southclaws caddy[27042]: runtime: failed to create new OS thread (have 13 already; errno=11)
Jun 18 16:28:07 southclaws caddy[27042]: runtime: may need to increase max user processes (ulimit -u)
Jun 18 16:28:07 southclaws caddy[27042]: fatal error: newosproc
Jun 18 16:28:07 southclaws caddy[27042]: runtime stack:
Jun 18 16:28:07 southclaws caddy[27042]: runtime.throw(0xb2c06f, 0x9)
Jun 18 16:28:07 southclaws caddy[27042]:         /usr/local/go/src/runtime/panic.go:596 +0x95
Jun 18 16:28:07 southclaws caddy[27042]: runtime.newosproc(0xc42005c400, 0xc42030a000)
Jun 18 16:28:07 southclaws caddy[27042]:         /usr/local/go/src/runtime/os_linux.go:163 +0x18c
Jun 18 16:28:07 southclaws caddy[27042]: runtime.newm(0xb5a4a8, 0xc42001e600)
Jun 18 16:28:07 southclaws caddy[27042]:         /usr/local/go/src/runtime/proc.go:1628 +0x137
Jun 18 16:28:07 southclaws caddy[27042]: runtime.startm(0x0, 0xc42006df01)
Jun 18 16:28:07 southclaws caddy[27042]:         /usr/local/go/src/runtime/proc.go:1698 +0x17e
Jun 18 16:28:07 southclaws caddy[27042]: runtime.wakep()
Jun 18 16:28:07 southclaws caddy[27042]:         /usr/local/go/src/runtime/proc.go:1779 +0x48
Jun 18 16:28:07 southclaws caddy[27042]: runtime.resetspinning()
Jun 18 16:28:07 southclaws caddy[27042]:         /usr/local/go/src/runtime/proc.go:2141 +0x71
Jun 18 16:28:07 southclaws caddy[27042]: runtime.schedule()
Jun 18 16:28:07 southclaws caddy[27042]:         /usr/local/go/src/runtime/proc.go:2229 +0x136
Jun 18 16:28:07 southclaws caddy[27042]: runtime.park_m(0xc420106b60)
Jun 18 16:28:07 southclaws caddy[27042]:         /usr/local/go/src/runtime/proc.go:2285 +0xab
Jun 18 16:28:07 southclaws caddy[27042]: runtime.mcall(0x0)
Jun 18 16:28:07 southclaws systemd[1]: caddy.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jun 18 16:28:07 southclaws systemd[1]: caddy.service: Unit entered failed state.
Jun 18 16:28:07 southclaws systemd[1]: caddy.service: Failed with result 'exit-code'.

8. How can someone who is starting from scratch reproduce the bug as minimally as possible?

Not sure since this is very basic usage and I doubt every single user runs into this on a 6 core machine, but I have no idea what configuration is causing this. I haven't made any system-level changes to internals such as max processes - it's all just a default Ubuntu installation, no bells or whistles.

[mholt]

Does this happen if you don't run it with systemd?

Also see #1114

[Southclaws]

Running as root without systemd seems to work fine. I attempted to raise the nofile/nproc values but the systemd deployment still fails on a page load.

[mholt]

Hmm. Sounds like systemd is interfering then, as with the other issue. Sorry, I'm not sure how to help with this or what to do about it (I don't use systemd with Caddy).