Can't start apache container

[vsaw]

I'm trying to get Nextcloud AIO v4.1.0 up and running behind a reverse proxy (running on the same machine as nextcloud)

The command to start nextcloud is

docker run \
    --sig-proxy=false \
    --name nextcloud-aio-mastercontainer \
    --restart always \
    --publish 8080:8080 \
    -e APACHE_PORT=11000 \
    -e APACHE_IP_BINDING=127.0.0.1 \
    -e APACHE_DISABLE_REWRITE_IP=1 \
    --volume nextcloud_aio_mastercontainer:/mnt/docker-aio-config \
    --volume /var/run/docker.sock:/var/run/docker.sock:ro \
    nextcloud/all-in-one:latest

Which ultimately results in the apache container not being started. The AIO logs show the following message

Error: loading initial config: loading new config: http app module: start: listening on :80: listen tcp :80: bind: permission denied
[Fri Jan 13 12:32:33.295439 2023] [php:notice] [pid 122] [client 127.0.0.1:48738] Slim Application Error
Type: GuzzleHttp\\Exception\\ServerException
Code: 500
Message: Server error: `POST http://localhost/v1.41/containers/nextcloud-aio-nextcloud/start` resulted in a `500 Internal Server Error` response:
{"message":"failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process (truncated...)

File: /var/www/docker-aio/php/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php
Line: 113
Trace: #0 /var/www/docker-aio/php/vendor/guzzlehttp/guzzle/src/Middleware.php(69): GuzzleHttp\\Exception\\RequestException::create(Object(GuzzleHttp\\Psr7\\Request), Object(GuzzleHttp\\Psr7\\Response), NULL, Array, NULL)
#1 /var/www/docker-aio/php/vendor/guzzlehttp/promises/src/Promise.php(204): GuzzleHttp\\Middleware::GuzzleHttp\\{closure}(Object(GuzzleHttp\\Psr7\\Response))
#2 /var/www/docker-aio/php/vendor/guzzlehttp/promises/src/Promise.php(153): GuzzleHttp\\Promise\\Promise::callHandler(1, Object(GuzzleHttp\\Psr7\\Response), NULL)
#3 /var/www/docker-aio/php/vendor/guzzlehttp/promises/src/TaskQueue.php(48): GuzzleHttp\\Promise\\Promise::GuzzleHttp\\Promise\\{closure}()
#4 /var/www/docker-aio/php/vendor/guzzlehttp/promises/src/Promise.php(248): GuzzleHttp\\Promise\\TaskQueue->run(true)
#5 /var/www/docker-aio/php/vendor/guzzlehttp/promises/src/Promise.php(224): GuzzleHttp\\Promise\\Promise->invokeWaitFn()
#6 /var/www/docker-aio/php/vendor/guzzlehttp/promises/src/Promise.php(269): GuzzleHttp\\Promise\\Promise->waitIfPending()
#7 /var/www/docker-aio/php/vendor/guzzlehttp/promises/src/Promise.php(226): GuzzleHttp\\Promise\\Promise->invokeWaitList()
#8 /var/www/docker-aio/php/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\\Promise\\Promise->waitIfPending()
#9 /var/www/docker-aio/php/vendor/guzzlehttp/guzzle/src/Client.php(187): GuzzleHttp\\Promise\\Promise->wait()
#10 /var/www/docker-aio/php/vendor/guzzlehttp/guzzle/src/ClientTrait.php(95): GuzzleHttp\\Client->request('POST', 'http://localhos...', Array)
#11 /var/www/docker-aio/php/src/Docker/DockerActionManager.php(182): GuzzleHttp\\Client->post('http://localhos...')
#12 /var/www/docker-aio/php/src/Controller/DockerController.php(48): AIO\\Docker\\DockerActionManager->StartContainer(Object(AIO\\Container\\Container))
#13 /var/www/docker-aio/php/src/Controller/DockerController.php(33): AIO\\Controller\\DockerController->PerformRecursiveContainerStart('nextcloud-aio-n...', true)
#14 /var/www/docker-aio/php/src/Controller/DockerController.php(181): AIO\\Controller\\DockerController->PerformRecursiveContainerStart('nextcloud-aio-a...', true)
#15 /var/www/docker-aio/php/src/Controller/DockerController.php(165): AIO\\Controller\\DockerController->startTopContainer(true)
#16 /var/www/docker-aio/php/vendor/slim/slim/Slim/Handlers/Strategies/RequestResponse.php(38): AIO\\Controller\\DockerController->StartContainer(Object(GuzzleHttp\\Psr7\\ServerRequest), Object(GuzzleHttp\\Psr7\\Response), Array)
#17 /var/www/docker-aio/php/vendor/slim/slim/Slim/Routing/Route.php(358): Slim\\Handlers\\Strategies\\RequestResponse->__invoke(Array, Object(GuzzleHttp\\Psr7\\ServerRequest), Object(GuzzleHttp\\Psr7\\Response), Array)
#18 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(65): Slim\\Routing\\Route->handle(Object(GuzzleHttp\\Psr7\\ServerRequest))
#19 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(65): Slim\\MiddlewareDispatcher->handle(Object(GuzzleHttp\\Psr7\\ServerRequest))
#20 /var/www/docker-aio/php/vendor/slim/slim/Slim/Routing/Route.php(315): Slim\\MiddlewareDispatcher->handle(Object(GuzzleHttp\\Psr7\\ServerRequest))
#21 /var/www/docker-aio/php/vendor/slim/slim/Slim/Routing/RouteRunner.php(68): Slim\\Routing\\Route->run(Object(GuzzleHttp\\Psr7\\ServerRequest))
#22 /var/www/docker-aio/php/vendor/slim/csrf/src/Guard.php(476): Slim\\Routing\\RouteRunner->handle(Object(GuzzleHttp\\Psr7\\ServerRequest))
#23 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(168): Slim\\Csrf\\Guard->process(Object(GuzzleHttp\\Psr7\\ServerRequest), Object(Slim\\Routing\\RouteRunner))
#24 /var/www/docker-aio/php/vendor/slim/twig-view/src/TwigMiddleware.php(115): Psr\\Http\\Server\\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\\Psr7\\ServerRequest))
#25 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\\Views\\TwigMiddleware->process(Object(GuzzleHttp\\Psr7\\ServerRequest), Object(Psr\\Http\\Server\\RequestHandlerInterface@anonymous))
#26 /var/www/docker-aio/php/src/Middleware/AuthMiddleware.php(38): Psr\\Http\\Server\\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\\Psr7\\ServerRequest))
#27 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(269): AIO\\Middleware\\AuthMiddleware->__invoke(Object(GuzzleHttp\\Psr7\\ServerRequest), Object(Psr\\Http\\Server\\RequestHandlerInterface@anonymous))
#28 /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/ErrorMiddleware.php(76): Psr\\Http\\Server\\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\\Psr7\\ServerRequest))
#29 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\\Middleware\\ErrorMiddleware->process(Object(GuzzleHttp\\Psr7\\ServerRequest), Object(Psr\\Http\\Server\\RequestHandlerInterface@anonymous))
#30 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(65): Psr\\Http\\Server\\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\\Psr7\\ServerRequest))
#31 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(199): Slim\\MiddlewareDispatcher->handle(Object(GuzzleHttp\\Psr7\\ServerRequest))
#32 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(183): Slim\\App->handle(Object(GuzzleHttp\\Psr7\\ServerRequest))
#33 /var/www/docker-aio/php/public/index.php(170): Slim\\App->run()
#34 {main}, referer: https://nextcloud-aio.mydomain.com/containers

I don't quite understand what tries to bind to port 80 (which is occupied by my reverse proxy), especially since I use -e APACHE_PORT=11000 when starting the AIO mastercontainer. Any help figuring this out is greatly appreciated.

FYI I think can rule out issues with my reverse proxy, as domain validation passes without issue.

P.S. I know about #1207 which I believe does not apply as I'm running Linux Kernel v 5.2.0.

[szaimen]

Hi, how did you install docker?

[vsaw]

I installed docker from the official Docker Ubuntu repository.

Here's the Docker version output.

Client: Docker Engine - Community
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:28:04 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:25:49 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[szaimen]

So did you install docker via snap?

[vsaw]

No. From their APT repositories. See https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository

Some more info on my installation

containerd -v
containerd containerd.io 1.6.14 9ba4b250366a5ddde94bb7c9d1def331423aa323

runc -v
runc version 1.1.4
commit: v1.1.4-0-g5fd4c4d
spec: 1.0.2-dev
go: go1.18.9
libseccomp: 2.5.3

[szaimen]

Ah I see. Then I don't know why you run into this failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process issue. It works fine over here.

[szaimen]

Is maybe selinux enabled on your host?

[vsaw]

Don't know. I can try and check tomorrow (once I figure out how to check for it 😅)

[vsaw]

Quick check suggests SELinux is not installed on the machine.

[szaimen]

You could also check google for the problem and see if some of the recommended solutions work for you: https://www.google.com/search?q=failed%20to%20create%20shim%20task%3A%20OCI%20runtime%20create%20failed%3A%20runc%20create%20failed%3A%20unable%20to%20start%20container%20process

[vsaw]

Been doing that for a few days, was hoping you have seen this before because most errors seem to be rather specific to the application that was tried to be started.

[szaimen]

Is running docker inside a LXC or other kind of VM?

[vsaw]

Yes. It's running on a VServer by Strato.de. Don't know what kind of VM exactly. Should I try to figure this out?

[szaimen]

Can you try a different vserver by a different provider?

[vsaw]

I can try with installing a different OS (currently running Ubuntu 22.04) but preferably would like to keep the provider.

Could you share your thought what you believe is happening that keeps me from running Nextcloud?

[szaimen]

It is possible that stratos vps' are not really compatible with docker.

[vsaw]

Doubt it. See links below (German Only)

1. https://www.strato.de/faq/server/welche-server-und-betriebssysteme-sind-docker-ready/
2. https://www.strato.de/faq/server/wie-kann-ich-nextcloud-auf-meinem-server-installieren/

But at least that something I can continue to investigate. Thanks for the pointer!

[szaimen]

Hm... Did you see this?

It is from your referenced page https://www.strato.de/faq/server/welche-server-und-betriebssysteme-sind-docker-ready/#server

[vsaw]

Yes, but as far as I understand it, this relates to

Bitte beachten Sie, dass der Einsatz von Docker-Containern, die spezielle Kernel-Module benötigen nicht auf einem Linux V-Server eingesetzt werden können.

Which as far as I know, Nextcloud does not need. In addition I do have Planka and some other services running on this machine via docker without issues.

Additionally I've been playing a bit. Without Nextcloud Talk, I can start the containers. However, accessing nextcloud fails with the following Apache log entries:

{"level":"error","ts":1673701618.9343426,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:8000: connect: connection refused","request":{"remote_ip":"172.17.0.1","remote_port":"54372","proto":"HTTP/1.1","method":"GET","host":"nextcloud.vsaw.net","uri":"/apps/richdocuments/settings/fonts.json","headers":{"Date":["Sat, 14 Jan 2023 13:06:58"],"User-Agent":["COOLWSD HTTP Agent 22.05.9.2"],"X-Forwarded-Scheme":["https"],"X-Forwarded-Proto":["https"],"X-Forwarded-For":["172.17.0.3"],"X-Real-Ip":["172.17.0.3"]}},"duration":0.000648198,"status":502,"err_id":"gxtx64dqb","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
{"level":"error","ts":1673701649.7408082,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:8000: connect: connection refused","request":{"remote_ip":"172.17.0.1","remote_port":"59816","proto":"HTTP/1.1","method":"GET","host":"nextcloud.vsaw.net","uri":"/","headers":{"X-Forwarded-Scheme":["https"],"X-Forwarded-For":["2a02:810d:e80:1724:188b:d916:506b:c7fd"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Site":["same-site"],"Sec-Gpc":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-User":["?1"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:108.0) Gecko/20100101 Firefox/108.0"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["2a02:810d:e80:1724:188b:d916:506b:c7fd"],"Accept-Language":["en,de-DE;q=0.5"],"Referer":["https://nextcloud-aio.vsaw.net/"],"Dnt":["1"]}},"duration":0.001005601,"status":502,"err_id":"mvyc0c8hf","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}

I can reproduce this error message with every refresh. So I assume the reverse proxy is working and for whatever reason, Apache decides to deny my request.

So I'm not 100% convinced it's Dockers fault yet.

[vsaw]

I think I found out what's the problem: My VPS limits the system to 400 PIDs. Without Nextcloud running I get the following baseline

root@h2995053:~# cat /sys/fs/cgroup/pids/pids.max
400
root@h2995053:~# cat /sys/fs/cgroup/pids/pids.current
244

Starting Nextcloud + Collabora brings this number to 389. So I can easily imagine this hitting the limit depending on background tasks etc.

Initial attempts to increase this number seems to be blocked by my VPS provider. I'm currently in touch with their support to see if they will increase it for me. If not I will need to find a different provider!

[szaimen]

Thanks a lot! :)

[Zoey2936]

@Zoey2936 Yes, googling on the issue seems a lot of people run into this. So far none has gotten a higher quota.

This seems to be regardless of Docker, so I expect also "native install" to fail. :-/

I had a VPS from them for a week (a few years ago) and then refunded my money, because my server froze when there was basically nothing running on it, because of the PID limit (They didn't increase it for me). Then I switched to netcup, and I'm still very happy with them.

[ThaDaVos]

Instead of /sys/fs/cgroup/pids/pids.max i checked /proc/sys/kernel/pid_max as the former isn't available on my vps - hope this is also a correct one (Ubuntu 22.04)

[septatrix]

Strato's new V-Server generation (which is what one has been getting since a few months by now) uses a different hypervisor and they no longer have such a limit set.

[septatrix]

Instead of /sys/fs/cgroup/pids/pids.max i checked /proc/sys/kernel/pid_max as the former isn't available on my vps - hope this is also a correct one (Ubuntu 22.04)

No they report different values. In the old strato servers the latter still reported a high value whereas the former contained the smaller limit of 400. The actual limits on the old VMs can be read in the numproc row of /proc/user_beancounters which seems to be specific to the Virtuozzo hypervisor used by Strato in their older VPS.