Segmentation Fault - Null Pointer

[tommmoe]

What happened? How did it happen?

I originally noticed this issue in production as a rare, intermittent segmentation fault that would occur roughly once every few weeks. It almost always coincided with bursts of traffic from AI-related bots aggressively scanning the website (large numbers of concurrent requests and JavaScript/Livewire-heavy responses).

In production, the FrankenPHP process would usually restart and recover within the same minute. Because the issue was infrequent and partially mitigated by blocking traffic at the WAF level, I did not initially investigate it further.

More recently, while ramping up activity for the business, I was able to reliably reproduce the crash by running an SEO spider with JavaScript rendering enabled against the development environment. This produces sustained concurrent requests and large HTML responses, closely resembling the traffic pattern that previously triggered the sporadic production crashes.

Under these conditions, FrankenPHP consistently segfaults after a consistent period of sustained load.

The application is running Laravel with Octane on FrankenPHP.

I have removed Sentry and OTLP

---

Crash details

The crash occurs in a PHP worker thread (php-2) with the following signature:

• Segmentation fault in memcpy(dest=0x0, ...)
• Called from zend_stack_push()
• Triggered during PHP output handling / output buffering

This stack trace was captured using the latest FrankenPHP debug build (v1.11.1).

---

Observed state at crash time

At the point of failure, the Zend output handler stack appears to be partially initialized but invalid:

• stack->size = 8
• stack->max = 16
• stack->top = 0
• stack->elements = NULL

This results in zend_stack_push() attempting to write to a NULL buffer, leading directly to the segmentation fault.

---

Additional observations

• The crash frequency does not change when adjusting worker count or min/max thread settings.
• The only configuration change that consistently delays the crash is increasing --max-requests.
• Reducing worker/thread counts does not prevent the issue; it only affects how quickly it reproduces under sustained load.
• This segfault looks suspiciously similar to Segmentation fault during runtime #1841

---

Binary / build information

/var/www/html/frankenphp: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV),
static-pie linked, BuildID[sha1]=d4345b5320add1832b4bb2841a12a128c0d238d1,
with debug_info, not stripped

Build Type

Docker (Debian Trixie)

Worker Mode

Yes

Operating System

GNU/Linux

CPU Architecture

x86_64

PHP configuration

System => Linux 0caf603fcd24 5.15.153.1-microsoft-standard-WSL2 #1 SMP Fri Mar 29 23:14:13 UTC 2024 x86_64
Build Date => Jan 16 2026 23:26:21
Build System => Linux - Docker
Build Provider => https://github.com/docker-library/php
Configure Command =>  './configure'  '--build=x86_64-linux-gnu' '--sysconfdir=/usr/local/etc' '--with-config-file-path=/usr/local/etc/php' '--with-config-file-scan-dir=/usr/local/etc/php/conf.d' '--enable-option-checking=fatal' '--with-mhash' '--with-pic' '--enable-mbstring' '--enable-mysqlnd' '--with-password-argon2' '--with-sodium=shared' '--with-pdo-sqlite=/usr' '--with-sqlite3=/usr' '--with-curl' '--with-iconv' '--with-openssl' '--with-readline' '--with-zlib' '--enable-phpdbg' '--enable-phpdbg-readline' '--with-pear' '--with-libdir=lib/x86_64-linux-gnu' '--enable-embed' '--enable-zts' '--disable-zend-signals' 'build_alias=x86_64-linux-gnu'
Server API => Command Line Interface
Virtual Directory Support => enabled
Configuration File (php.ini) Path => /usr/local/etc/php
Loaded Configuration File => /usr/local/etc/php/php.ini
Scan this dir for additional .ini files => /usr/local/etc/php/conf.d
Additional .ini files parsed => /usr/local/etc/php/conf.d/99-octane.ini,
/usr/local/etc/php/conf.d/docker-php-ext-bcmath.ini,
/usr/local/etc/php/conf.d/docker-php-ext-bz2.ini,
/usr/local/etc/php/conf.d/docker-php-ext-exif.ini,
/usr/local/etc/php/conf.d/docker-php-ext-gd.ini,
/usr/local/etc/php/conf.d/docker-php-ext-gmp.ini,
/usr/local/etc/php/conf.d/docker-php-ext-igbinary.ini,
/usr/local/etc/php/conf.d/docker-php-ext-imagick.ini,
/usr/local/etc/php/conf.d/docker-php-ext-intl.ini,
/usr/local/etc/php/conf.d/docker-php-ext-ldap.ini,
/usr/local/etc/php/conf.d/docker-php-ext-opcache.ini,
/usr/local/etc/php/conf.d/docker-php-ext-pcntl.ini,
/usr/local/etc/php/conf.d/docker-php-ext-pdo_mysql.ini,
/usr/local/etc/php/conf.d/docker-php-ext-pdo_pgsql.ini,
/usr/local/etc/php/conf.d/docker-php-ext-pgsql.ini,
/usr/local/etc/php/conf.d/docker-php-ext-protobuf.ini,
/usr/local/etc/php/conf.d/docker-php-ext-rdkafka.ini,
/usr/local/etc/php/conf.d/docker-php-ext-redis.ini,
/usr/local/etc/php/conf.d/docker-php-ext-sockets.ini,
/usr/local/etc/php/conf.d/docker-php-ext-sodium.ini,
/usr/local/etc/php/conf.d/docker-php-ext-uv.ini,
/usr/local/etc/php/conf.d/docker-php-ext-vips.ini,
/usr/local/etc/php/conf.d/docker-php-ext-zip.ini,
/usr/local/etc/php/conf.d/xx-php-ext-memcached.ini

PHP API => 20230831
PHP Extension => 20230831
Zend Extension => 420230831
Zend Extension Build => API420230831,TS
PHP Extension Build => API20230831,TS
Debug Build => no
Thread Safety => enabled
Thread API => POSIX Threads
Zend Signal Handling => disabled
Zend Memory Manager => enabled
Zend Multibyte Support => provided by mbstring
Zend Max Execution Timers => enabled
IPv6 Support => enabled
DTrace Support => disabled

Relevant log output

#0  0x00007f81e4d32173 in memcpy (dest=0x0, src=<optimized out>, n=8) at src/string/memcpy.c:33
#1  0x00007f81e228e986 in zend_stack_push (stack=0x7f819727db10, element=0x7f81976e8f48)
    at /go/src/app/dist/static-php-cli/source/php-src/Zend/zend_stack.c:40
#2  0x00007f81e1f79fb6 in php_output_handler_start (handler=0x7f8190df8fa0) at /go/src/app/dist/static-php-cli/source/php-src/main/output.c:557
#3  0x00007f81e1f796df in php_output_start_user (output_handler=0x0, chunk_size=0, flags=112)
    at /go/src/app/dist/static-php-cli/source/php-src/main/output.c:441
#4  0x00007f81e1f7c9df in zif_ob_start (execute_data=0x7f8196a149e0, return_value=0x7f81976ecbc0)
    at /go/src/app/dist/static-php-cli/source/php-src/main/output.c:1318
#5  0x00007f81e2193d29 in ZEND_DO_FCALL_BY_NAME_SPEC_RETVAL_UNUSED_HANDLER ()
    at /go/src/app/dist/static-php-cli/source/php-src/Zend/zend_vm_execute.h:1584
#6  execute_ex (ex=0x7f8196a14020) at /go/src/app/dist/static-php-cli/source/php-src/Zend/zend_vm_execute.h:58917
#7  0x00007f81e2052ea7 in zend_call_function (fci=0x7f81976ed140, fci_cache=0x7f81976ed070)
    at /go/src/app/dist/static-php-cli/source/php-src/Zend/zend_execute_API.c:998
#8  0x00007f81e2051203 in _call_user_function_impl (object=0x0, function_name=0x7f81976ed220, retval_ptr=0x7f81976ed230, param_count=1, 
    params=0x7f81976ed240, named_params=0x0) at /go/src/app/dist/static-php-cli/source/php-src/Zend/zend_execute_API.c:790
#9  0x00007f81e22aaa21 in zend_user_exception_handler () at /go/src/app/dist/static-php-cli/source/php-src/Zend/zend.c:1905
#10 0x00007f81e22aae01 in zend_execute_script (type=8, retval=0x0, file_handle=0x7f81976ef620)
    at /go/src/app/dist/static-php-cli/source/php-src/Zend/zend.c:1938
#11 0x00007f81e1f776cc in php_execute_script_ex (primary_file=0x7f81976ef620, retval=0x0)
    at /go/src/app/dist/static-php-cli/source/php-src/main/main.c:2593
#12 0x00007f81e1f77927 in php_execute_script (primary_file=0x7f81976ef620) at /go/src/app/dist/static-php-cli/source/php-src/main/main.c:2633
#13 0x00007f81e1f1f2b1 in frankenphp_execute_script (file_name=0xc00005a720 "//js/filament/notifications/notifications.js") at frankenphp.c:1079
#14 0x00007f81e1f1f358 in php_thread (arg=<optimized out>) at frankenphp.c:948
#15 0x00007f81e4d34958 in start (p=<optimized out>) at src/thread/pthread_create.c:207
#16 0x00007f81e4d361c2 in __clone () at src/thread/x86_64/clone.s:22

[withinboredom]

This sounds suspiciously like a php bug; possibly related to ZTS.

[tommmoe]

Update to Bug Report

Original Crash: FIXED by disabling nunomaduro/collision

• Root cause: Collision using ob_start() after php_output_deactivate()
• Fix: Disable Collision in worker mode
• I went through and disabled everything that could be writing exceptions (even though there weren't any exceptions)

New Crash: Go write layer

• Backtrace shows crash in go_ub_write() during zstd compression
• Writing 429KB response when crash occurs
• Likely client disconnect or broken pipe
• Different issue from original output buffer crash

Back Trace

(gdb) bt
#0  internal/runtime/syscall.Syscall6 ()
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/internal/runtime/syscall/asm_linux_amd64.s:36
#1  0x00007fd0dfe4420d in syscall.RawSyscall6 (trap=18446744073709551584, a1=11, a2=140535085289934, a3=11, a4=824644471361, a5=0, a6=0, 
    r1=<optimized out>, r2=<optimized out>, err=<optimized out>)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/syscall/syscall_linux.go:65
#2  0x00007fd0dfe44266 in syscall.Syscall (trap=1, a1=11, a2=824644471361, a3=25728, r1=<optimized out>, r2=<optimized out>, err=<optimized out>)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/syscall/syscall_linux.go:87
#3  0x00007fd0dfe4257b in syscall.write (fd=<optimized out>, p=..., n=<optimized out>, err=...)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/syscall/zsyscall_linux_amd64.go:964
#4  0x00007fd0dfe68bf4 in syscall.Write (fd=11, p=..., n=<optimized out>, err=...)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/syscall/syscall_unix.go:211
#5  internal/poll.ignoringEINTRIO (fd=11, p=..., fn=<optimized out>)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/internal/poll/fd_unix.go:738
#6  internal/poll.(*FD).Write (fd=0xc000688680, p=..., ~r0=<optimized out>, ~r0=<optimized out>, ~r1=..., ~r1=...)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/internal/poll/fd_unix.go:374
#7  0x00007fd0dfeddb45 in net.(*netFD).Write (fd=0xffffffffffffffe0, p=..., nn=<optimized out>, err=...)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/net/fd_posix.go:109
#8  0x00007fd0dfeef945 in net.(*conn).Write (c=0xc00011d9a0, b=..., ~r0=<optimized out>, ~r1=...)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/net/net.go:208
#9  0x00007fd0e0cacdaa in github.com/caddyserver/caddy/v2/modules/caddyhttp.(*http2Conn).Write (~p0=<optimized out>, b=..., n=<optimized out>, err=...)
    at ./<autogenerated>:1
#10 0x00007fd0e01347e6 in net/http.checkConnErrorWriter.Write (w=..., p=..., n=<optimized out>, err=...)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/net/http/server.go:4066
#11 0x00007fd0dff0a427 in bufio.(*Writer).Write (b=0xc0003dd6c0, p=..., nn=<optimized out>, err=...)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/bufio/bufio.go:685
#12 0x00007fd0e0125205 in net/http.(*chunkWriter).Write (cw=0xc0008b49a0, p=..., n=<optimized out>, err=...)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/net/http/server.go:391
#13 0x00007fd0dff0a427 in bufio.(*Writer).Write (b=0xc0003a2900, p=..., nn=<optimized out>, err=...)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/bufio/bufio.go:685
#14 0x00007fd0e012b12d in net/http.(*response).write (w=0xc0008b4960, lenData=28353, dataB=..., dataS=..., n=<optimized out>, err=...)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/net/http/server.go:1694
#15 0x00007fd0e012ae0a in net/http.(*response).Write (w=0xffffffffffffffe0, data=..., n=<optimized out>, err=...)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/net/http/server.go:1657
#16 0x00007fd0e0c938b6 in github.com/caddyserver/caddy/v2/modules/caddyhttp.(*responseRecorder).Write (rr=0xc000264b00, data=..., ~r0=<optimized out>, 
    ~r1=...)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go/pkg/mod/github.com/caddyserver/caddy/v2@v2.10.2/modules/caddyhttp/responsewriter.go:180
#17 0x00007fd0e0c938b6 in github.com/caddyserver/caddy/v2/modules/caddyhttp.(*responseRecorder).Write (rr=0xc000264c40, data=..., ~r0=<optimized out>, 
    ~r1=...)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go/pkg/mod/github.com/caddyserver/caddy/v2@v2.10.2/modules/caddyhttp/responsewriter.go:180
#18 0x00007fd0e0ca9009 in github.com/caddyserver/caddy/v2/modules/caddyhttp.(*ResponseWriterWrapper).Write (~p0=<optimized out>, ~p1=..., 
    ~r0=<optimized out>, ~r1=...) at ./<autogenerated>:1
#19 0x00007fd0e0d1f118 in github.com/klauspost/compress/zstd.(*Encoder).nextBlock (e=0xc00096c140, final=false, ~r0=...)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go/pkg/mod/github.com/klauspost/compress@v1.18.3/zstd/encoder.go:285
#20 0x00007fd0e0d1e81a in github.com/klauspost/compress/zstd.(*Encoder).Write (e=0xc00096c140, p=..., n=<optimized out>, err=...)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go/pkg/mod/github.com/klauspost/compress@v1.18.3/zstd/encoder.go:177
#21 0x00007fd0e0cec782 in github.com/caddyserver/caddy/v2/modules/caddyhttp/encode.(*responseWriter).Write (rw=0xc00068f0e0, p=..., ~r0=<optimized out>, 
    ~r1=...)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go/pkg/mod/github.com/caddyserver/caddy/v2@v2.10.2/modules/caddyhttp/encode/encode.go:344
#22 0x00007fd0e112e65f in github.com/dunglas/frankenphp.go_ub_write (threadIndex=<optimized out>, 
    cBuf=0x7fd065c6a018 "<!DOCTYPE html>\n<html lang=\"en\">\n    <head>\n        <!-- Google Tag Manager -->\n<script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':\nnew Date().getTime(),event:'gtm.js'});var f=d.getElem"..., length=429277, ~r0=<optimized out>, ~r1=<optimized out>)
    at /go/src/app/frankenphp.go:440
#23 0x00007fd0e1139c25 in _cgoexp_634fea83287e_go_ub_write (a=0x7fd0967f5210) at ./_cgo_gotypes.go:1924
#24 0x00007fd0dfd71ca9 in runtime.cgocallbackg1 (fn=0x7fd0e1139c00 <_cgoexp_634fea83287e_go_ub_write>, frame=0x7fd0967f5210, ctxt=0)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/runtime/cgocall.go:446
#25 0x00007fd0dfd71932 in runtime.cgocallbackg (fn=0x7fd0e1139c00 <_cgoexp_634fea83287e_go_ub_write>, frame=0x7fd0967f5210, ctxt=0)
    at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/runtime/cgocall.go:350
#26 0x00007fd0dfdec5eb in runtime.cgocallbackg (fn=0x7fd0e1139c00 <_cgoexp_634fea83287e_go_ub_write>, frame=0x7fd0967f5210, ctxt=0) at ./<autogenerated>:1
#27 0x00007fd0dfde9bad in runtime.cgocallback () at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/runtime/asm_amd64.s:1082
#28 0x00007fd0dfde9de1 in runtime.goexit () at /go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go-xcaddy/src/runtime/asm_amd64.s:1693
#29 0x0000000000000000 in ?? ()

[tommmoe]

Another update, disabled compression in my Caddyfile # encode zstd br gzip

SEO scan completed using full Javascript on all of our 2,000+ product pages

I assume that I've run into two bugs here, if anything else is needed for troubleshooting, I'll keep the debug container.

[henderkes]

The php crash seems to be quite understandable and should be reported to php-src. The zstd crash (it's visible in your last stacktrace) leads deep into syscalls, though. I'm wondering if it's possibly down to a difference in linux headers between the build machine and the run machine. Linux headers are extremely stable, but perhaps something slipped by?

Is there any way you can reproduce the latter crash when using the Docker image, rather than the static binary?

[tommmoe]

Is there any way you can reproduce the latter crash when using the Docker image, rather than the static binary?

I was using FROM dunglas/frankenphp-dev:php8.3 AS base and then copying over the binary, maybe I was losing my mind but I didn't get symbols from the binary that was in frankenphp-dev:php8.3

[henderkes]

The stack trace shows that you are using the standalone binary found in the GitHub releases and not the frankenphp from the frankenphp-dev:php8.3:

/go/src/app/dist/static-php-cli/pkgroot/x86_64-linux/go/pkg/mod/github.com/caddyserver/caddy/v2@v2.10.2

static-php-cli is only used in the Docker images.

I didn't get symbols from the binary that was in frankenphp-dev:php8.3

Ah! That's weird, I'm not familiar with the dev images, perhaps we accidentally compile frankenphp with -W -s there?

[henderkes]

https://github.com/php/frankenphp/blob/main/dev.Dockerfile

Everything seems to be in order. But I suspect you may not have gotten php debug symbols for the first crash.

[AlliBalliBaba]

@henderkes that segfault looks very similar to #2121 and #1851 where an output in an exception handler causes the crash (zend_user_exception_handler). Still possible that #2142 fixes it, could also be something in php-src, still no real way to reproduce it.

Actually pretty sure we're passing expired data here to the write syscall. My guess is still that shutdown handlers are collecting everything for error reporting and including some request-specific data that is already expired, so we should definitely test if this still happens after #2142 is released.