Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PHP-FPM issue in Docker Desktop 4.27.2: WARNING: [pool www] child 85 exited on signal 11 (SIGSEGV) #7182

Closed
NiklasBr opened this issue Feb 9, 2024 · 22 comments

Comments

@NiklasBr
Copy link

NiklasBr commented Feb 9, 2024

Description

After updating to Docker Desktop 4.27.2 from 4.26.1 PHP-FPM containers cannot run without crashing with the error:

2024-02-09 10:17:13 pimcore-fpm-1    | 172.19.0.6 -  09/Feb/2024:10:17:13 +0100 "GET /index.php" 200
2024-02-09 10:17:14 pimcore-fpm-1    | [09-Feb-2024 10:17:14] WARNING: [pool www] child 85 exited on signal 11 (SIGSEGV) after 239.493217 seconds from start

Reproduce

  1. Build a PHP-based image with a Composer package and put it behind an Nginx proxy.
  2. Run it using docker compose up -d

Expected behavior

It should run as well as it did in 4.26.1

docker version

Client:
 Cloud integration: v1.0.35+desktop.10
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:13:26 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.27.2 (137060)
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435e5f6216828dec57958c490c4f8bae4f98
  Built:            Wed Feb  7 00:39:16 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0


### docker info

```bash
Client:
 Version:    25.0.3
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     /Users/nikbr/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.5-desktop.1
    Path:     /Users/nikbr/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.24
    Path:     /Users/nikbr/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/nikbr/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     /Users/nikbr/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /Users/nikbr/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.0
    Path:     /Users/nikbr/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/nikbr/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.4.1
    Path:     /Users/nikbr/.docker/cli-plugins/docker-scout
WARNING: Plugin "/Users/nikbr/.docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /Users/nikbr/.docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 18
  Running: 11
  Paused: 0
  Stopped: 7
 Images: 36
 Server Version: 25.0.3
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.12-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 9
 Total Memory: 23.44GiB
 Name: docker-desktop
 ID: 8f590826-49c3-4f7b-8c5d-0d6228d1cf1a
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile


### Diagnostics ID

B99BCF91-9CAF-4D12-AE8B-A4858EC43329/20240209095720

### Additional Info

I thought this was supposed to be fixed in 4.27.2 according to the release notes?
@dgageot
Copy link
Member

dgageot commented Feb 9, 2024

@NiklasBr this would really help if you could share the complete Dockerfile and compose files that you use. With it, we can probably fix your issue under an hour. Without, it will take much much longer. If you can provide those files, that would be awesome!

@NiklasBr
Copy link
Author

NiklasBr commented Feb 9, 2024

Nginx Dockerfile:

FROM nginx:1.24

RUN apt-get update && apt-get install -y --no-install-recommends nano && rm -rf /var/lib/apt/lists/*

COPY etc/default.conf.tmpl /etc/nginx/conf.d/default.conf.tmpl
COPY etc/includes/maintenance.conf.tmpl /etc/nginx/conf.d/includes/maintenance.conf.tmpl
COPY etc/nginx.conf /etc/nginx/nginx.conf
COPY etc/extra/ /etc/nginx/extra/
COPY bin/* /usr/local/bin/

RUN mkdir -p /etc/nginx/ssl/cert/

EXPOSE 80

ENV UPLOAD_MAX_FILESIZE 100M
ENV FPM_HOST pimcore-fpm
ENV FPM_PORT 9000
ENV PIMCORE_ROOT /var/www/pimcore
ENV DEBUG false
ENV BASIC_AUTH off

RUN chmod +x /usr/local/bin/docker-entrypoint.sh

ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]
CMD ["nginx", "-g", "daemon off;"]

@NiklasBr
Copy link
Author

NiklasBr commented Feb 9, 2024

PHP application Dockerfile:

FROM php:8.1-fpm

WORKDIR /var/www

RUN apt-get update && apt-get install -y lsb-release \
    && echo "deb http://deb.debian.org/debian $(lsb_release -sc)-backports main" > /etc/apt/sources.list.d/backports.list \
    && apt-get update && apt-get install -y --no-install-recommends \
      autoconf automake libtool nasm make pkg-config libz-dev build-essential openssl g++ \
      libzip-dev libicu-dev libbz2-dev libavif-dev libheif-dev libpng-dev libc-client-dev libonig-dev \
      libkrb5-dev libxml2-dev libxslt1.1 libxslt1-dev locales locales-all \
      libfreetype6-dev ffmpeg ghostscript pngcrush jpegoptim exiftool liblcms2-dev libwebp-dev poppler-utils git wget \
      libx11-dev libjpeg-dev libtiff-dev opencv-data webp graphviz libreoffice cmake unzip rsync \
      msmtp procps gettext-base nano sudo iproute2 libltdl-dev \
      liblqr-1-0-dev libjpeg-turbo-progs libjpeg62-turbo-dev libopenjp2-7-dev \
      libdjvulibre-dev libpango1.0-dev libxpm-dev \
      xfonts-75dpi xfonts-base \
      advancecomp zopfli optipng pngquant

RUN wget https://imagemagick.org/archive/ImageMagick.tar.gz \
      && tar -xvf ImageMagick.tar.gz \
      && cd ImageMagick-7.* \
      && ./configure --enable-shared --with-tiff=yes --disable-docs --with-tcmalloc --without-perl \
      && make --jobs=$(nproc) \
      && make install \
      && ldconfig /usr/local/lib \
      && cd .. \
      && rm -rf ImageMagick* \
   && docker-php-ext-install -j$(nproc) intl mysqli bcmath bz2 gd soap xsl pdo pdo_mysql fileinfo exif zip opcache sockets \
    && docker-php-ext-enable intl mysqli bcmath bz2 gd soap xsl pdo pdo_mysql fileinfo exif zip opcache sockets \
    && pecl install imagick apcu redis \
    && pecl install -f xmlrpc \
    && docker-php-ext-enable redis imagick apcu xmlrpc \
    && docker-php-ext-configure imap --with-kerberos --with-imap-ssl \
    && docker-php-ext-install -j$(nproc) imap \
    && docker-php-ext-enable imap \
    \
    && pecl install -o -f xdebug \
    && cd ~

COPY --from=composer:latest /usr/bin/composer /usr/bin/composer

ENV COMPOSER_ALLOW_SUPERUSER 1
ENV COMPOSER_MEMORY_LIMIT -1
ENV COMPOSER_HOME /var/www/.composer
VOLUME /var/www/.composer

ENV PIMCORE_ROOT /var/www/pimcore
ENV PHP_MEMORY_LIMIT 2G
ENV PHP_ENABLE_XDEBUG false

ENV DEBUG false
ENV UPDATE_UID_GID false
ENV APP_ENV dev
ENV CLUSTER_SETUP false

# PHP opcache default values
ENV OPCACHE_MAX_FILES 16229
ENV OPCACHE_MEMORY 128
ENV OPCACHE_STRINGS_BUFFER 24
ENV OPCACHE_VALIDATE_TIMESTAMPS 1
ENV OPCACHE_REVALIDATE_FREQ 2

# PHP fpm
ENV PHP_PM_MAX_CHILDREN 20
ENV PHP_PM_START_SERVERS 8
ENV PHP_PM_MIN_SPARSE_SERVERS 5
ENV PHP_PM_MAX_SPARSE_SERVERS 12
ENV UPLOAD_MAX_FILESIZE 64M
ENV POST_MAX_SIZE 128M

# New Relic:
ENV NEW_RELIC_LICENSE_KEY must_set
ENV NEW_RELIC_APM_HIGH_SECURITY_MODE false
ENV NEW_RELIC_APM_ENABLED false
ENV NEW_RELIC_PROJECT_NAME must_set
ENV NEW_RELIC_COMMON_APP_NAME Pimcore
ENV NEW_RELIC_TIER_LABEL FPM
ENV NEW_RELIC_ENVIRONMENT_LABEL must_set

COPY etc/bashrc.sh /tmp/
RUN cat /tmp/bashrc.sh >> /etc/bash.bashrc && rm -f /tmp/bashrc.sh

COPY etc/php-fpm.conf.tmpl   /usr/local/etc/
COPY etc/php-fpm.ini /usr/local/etc/php/conf.d/zz-pimcore.ini
COPY etc/php-mail.ini /usr/local/etc/php/conf.d/zz-mail.ini
COPY etc/php-opcache.ini /usr/local/etc/php/conf.d/zz-opcache.ini
COPY etc/php-xdebug.ini /usr/local/etc/php/conf.d/zz-xdebug-settings.ini
COPY etc/php-timezone.ini /usr/local/etc/php/conf.d/zz-timezone.ini
COPY etc/php-newrelic.ini /usr/local/etc/php/conf.d/zz-newrelic.ini
COPY etc/msmtp.conf.tmpl /etc/

COPY bin/* /usr/local/bin/
RUN chmod +x /usr/local/bin/docker-entrypoint.sh
RUN touch /var/log/msmtp.log && chown www-data /var/log/msmtp.log

# Install NR with multiarch support
RUN  VERSION_REGEX='[1-9][0-9]\?\(\.[0-9]\+\)\{3\}' \
   && NEWRELIC_FILE=$(curl 'https://download.newrelic.com/php_agent/release/' | grep -o 'newrelic-php5.*linux\.tar\.gz' | sed -e 's/^.*>//;s/<[^>]*>//g') \
   && NEWRELIC_FOLDER=$(echo ${NEWRELIC_FILE} | sed 's/.tar.gz//g') \
   && curl https://download.newrelic.com/php_agent/release/${NEWRELIC_FILE} | tar -zx \
   && cd ${NEWRELIC_FOLDER} \
   && export NR_INSTALL_USE_CP_NOT_LN=1 \
   && export NR_INSTALL_SILENT=1 \
   && export NR_INSTALL_KEY=TEMPLICENSEAAAAAAAAAAAAAAAAAAAAAAAAAAAAA \
   && ./newrelic-install install \
   && rm -rf ${NEWRELIC_FOLDER}

RUN ln -fs /usr/share/zoneinfo/Europe/Oslo /etc/localtime && echo "Europe/Oslo" > /etc/timezone

ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]

CMD ["php-fpm", "-F"]

@dgageot
Copy link
Member

dgageot commented Feb 9, 2024

@NiklasBr do you run this with Rosetta or Qemu?

@NiklasBr
Copy link
Author

NiklasBr commented Feb 9, 2024

Screenshot 2024-02-09 at 13 42 20

@NiklasBr
Copy link
Author

NiklasBr commented Feb 9, 2024

@dgageot I also tried EXPERIMENTAL_DOCKER_DESKTOP_FORCE_QEMU=1 docker build … but it did not change anything.

Screenshot 2024-02-09 at 13 49 35

@dgageot
Copy link
Member

dgageot commented Feb 9, 2024

@dgageot I also tried EXPERIMENTAL_DOCKER_DESKTOP_FORCE_QEMU=1 docker build … but it did not change anything.

Your issue is a run issue, not a build issue. Could you try with and without Rosetta enabled?
That should at least give you different errors in both cases.

@NiklasBr
Copy link
Author

Same error without Rosetta enabled:

2024-02-12 08:17:15 pimcore-fpm-1    | 172.19.0.11 -  12/Feb/2024:08:17:13 +0100 "GET /index.php" 200
2024-02-12 08:17:15 pimcore-fpm-1    | [12-Feb-2024 08:17:15] WARNING: [pool www] child 113 said into stderr: "qemu: uncaught target signal 11 (Segmentation fault) - core dumped"
2024-02-12 08:17:15 pimcore-fpm-1    | [12-Feb-2024 08:17:15] WARNING: [pool www] child 113 exited on signal 11 (SIGSEGV) after 39.193922 seconds from start
2024-02-12 08:17:15 pimcore-web-1    | 2024/02/12 07:17:15 [error] 32#32: *47 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.65.1, server: localhost, request: "GET /_wdt/46c3d5 HTTP/1.1", upstream: "fastcgi://172.19.0.9:9000",
2024-02-12 08:17:15 pimcore-fpm-1    | [12-Feb-2024 08:17:15] NOTICE: [pool www] child 129 started

@dgageot
Copy link
Member

dgageot commented Feb 13, 2024

qemu: uncaught target signal 11 (Segmentation faul

This can't be a rosetta error

@NiklasBr
Copy link
Author

Not disagreeing, but in real life it is still a blocking problem. Cannot use the Docker license I am paying for at the moment.

@dgageot
Copy link
Member

dgageot commented Feb 13, 2024

@NiklasBr That would be easier to help you with a basic project to repro. You shared dockerfiles but they rely on plenty of files on your system.
You shared Rosetta logs but they are qemu in fact.

I really really want to help you but it requires way too much guessing on my part.

@NiklasBr
Copy link
Author

I have always been willing to share screen and details in a non-public forum due to the fact that they contain customer data.

@dgageot
Copy link
Member

dgageot commented Feb 13, 2024

Can you contact me at david.gageot@docker.com to organise a call?

@delmarr
Copy link

delmarr commented Feb 15, 2024

I've been following this thread and #7037

opcache seems to be failing for me in a simple setup.

Here is my setup
OS: Sonoma 14.3.1
Docker desktop version: Docker Desktop 4.27.2
image

Docker file

FROM registry.access.redhat.com/ubi7/ubi

RUN rpm -ivh \
  "https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm" \
  "https://rpms.remirepo.net/enterprise/remi-release-7.rpm"

RUN yum-config-manager --enable remi-php81 \
 && yum -y install \
    httpd \
    php \
    php-opcache

ENTRYPOINT [ "bash"]

Commands
docker build -t opcache-php Dockerfile

docker run --rm -it opcache-php

Then inside the container

[root@d17a8cde9a6f /]# httpd -k start   
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.245.3. Set the 'ServerName' directive globally to suppress this message
[root@d17a8cde9a6f /]# ps auxf
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0 439744  5140 pts/0    Ss   16:52   0:00 /run/rosetta/rosetta /usr/bin/b
root      5512  1.2  0.1 865428 28012 ?        Ss   17:18   0:00 /run/rosetta/rosetta /usr/sbin/
root      5521  0.0  0.0      0     0 ?        Z    17:18   0:00  \_ [httpd] <defunct>
root      5522  0.0  0.0      0     0 ?        Z    17:18   0:00  \_ [httpd] <defunct>
root      5523  0.0  0.0      0     0 ?        Z    17:18   0:00  \_ [httpd] <defunct>
root      5524  0.0  0.0      0     0 ?        Z    17:18   0:00  \_ [httpd] <defunct>
root      5525  0.0  0.0 479276  5376 pts/0    R+   17:18   0:00 /usr/bin/ps ps auxf

As you can see httpd crashes on start.

@dgageot
Copy link
Member

dgageot commented Feb 15, 2024

Thanks @delmarr, I was able to reproduce. I'll take a close look later today or tomorrow.

Here are the command I used (slightly modified):

cat <<EOF | docker buildx build --platform=linux/amd64 -t opcache-php -f- /var/empty
FROM registry.access.redhat.com/ubi7/ubi

RUN rpm -ivh \
  "https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm" \
  "https://rpms.remirepo.net/enterprise/remi-release-7.rpm"

RUN yum-config-manager --enable remi-php81 \
 && yum -y install \
    httpd \
    php \
    php-opcache

ENTRYPOINT [ "bash"]
EOF

docker run --rm -it --platform=linux/amd64 opcache-php
httpd -X

@dgageot
Copy link
Member

dgageot commented Feb 16, 2024

@delmarr @NiklasBr for both of you, the same code fails both on qemu and rosetta. And it used to work in both cases in Docker Desktop 4.26.1. I think you are seeing a variant of the same issue.

I've got a fix for Rosetta being evaluated. Hopefully it'll make it into 4.28.0. This the continuation of the work to prevent php and friends to segfault under Rosetta when Huge Pages are enabled on the kernel.

I'm also investigating a fix for Qemu. This one is again an issue with Qemu 8.1.5 that we didn't have in 6.something. Not sure if this one will make it into 4.28.0

@NiklasBr
Copy link
Author

@dgageot I see the release notes are taking shape which mentions this issue. The pre-release build you sent me did not resolve it as you remember, is there a new one to test?

@dgageot
Copy link
Member

dgageot commented Feb 19, 2024

Hey @NiklasBr, 4.28.0 should be out very soon. I expect it to fix your issue on Rosetta. Not on Qemu, though. Qemu 8.1.5 still has a regression that is closely related to your issue but that we couldn't totally fix.
I'll ping you as soon as 4.28.0 is out.

@dgageot
Copy link
Member

dgageot commented Feb 26, 2024

@delmarr @NiklasBr Could you give a try to Docker Desktop 4.28? It should be fixed on Rosetta. On QEMU, it depends of the workflow. Some are still broken.

@NiklasBr
Copy link
Author

NiklasBr commented Feb 27, 2024

No more Signal 11 errors for the first few hours! 🎊

Though it introduced a new one: Warning: include(vendor/symfony/console/Event/ConsoleErrorEvent.php): Failed to open stream: Too many open files which does not happen in v4.26.1 and this error pops up all the time. I click "re-apply" and twenty minutes later it's back:

Screenshot 2024-02-27 at 12 58 39

@dgageot
Copy link
Member

dgageot commented Feb 27, 2024

@NiklasBr the popup is a known issue. It's fixed on our side and will ship either with a 4.28.1 or with 4.29.
The too many files error, I don't have enough information to guess where it comes from.

@NiklasBr
Copy link
Author

I'll open a new issue for the Too many open files issue, for what it worth I think my original reported issue is solved, thanks!

@dgageot dgageot removed their assignment Feb 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants