Empty response on FreeBSD 14 using `SWOOLE_PROCESS`

[nanaya]

Please answer these questions before submitting your issue.

1. What did you do? If possible, provide a simple script for reproducing the error.

<?php

use Swoole\Http\Server;
use Swoole\Http\Request;
use Swoole\Http\Response;

$server = new Swoole\HTTP\Server("127.0.0.1", 9501, SWOOLE_PROCESS);

$server->on("Start", function(Server $server)
{
    echo "Swoole http server is started at http://127.0.0.1:9501\n";
});

$server->on("Request", function(Request $request, Response $response)
{
    $response->end(str_repeat('a', 1854));
});

$server->start();

And then check the response with curl http://127.0.0.1:9501

The problem doesn't happen when using SWOOLE_BASE or when the response is shorter (1853 in my system).

2. What did you expect to see?

Long string of a

3. What did you see instead?

curl: (52) Empty reply from server

4. What version of Swoole are you using (show your php --ri swoole)?

swoole

Swoole => enabled
Author => Swoole Team <team@swoole.com>
Version => 5.1.0
Built => Oct 21 2023 01:14:20
coroutine => enabled with boost asm context
eventfd => enabled
kqueue => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
http2 => enabled
json => enabled
zlib => 1.3
mutex_timedlock => enabled
pthread_barrier => enabled
mysqlnd => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_fiber_mock => Off => Off
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 262144 => 262144

5. What is your machine environment used (show your uname -a & php -v & gcc -v) ?

# uname -a
FreeBSD nanaka.nanaya.net 14.0-RC3 FreeBSD 14.0-RC3 #0 releng/14.0-n265368-c6cfdc130554: Fri Oct 27 05:57:28 UTC 2023     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
# php -v
PHP 8.2.11 (cli) (built: Oct 17 2023 02:36:30) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.2.11, Copyright (c) Zend Technologies
    with Zend OPcache v8.2.11, Copyright (c), by Zend Technologies
# clang -v
FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152)
Target: x86_64-unknown-freebsd14.0
Thread model: posix
InstalledDir: /usr/bin

[NathanFreeman]

I can not reproduce it.
Please use gdb to track the issue.

[nanaya]

This is the closest I can get (after which curl exits with the empty reply response):

Breakpoint 5, swoole::PHPCoroutine::main_func (_args=0x8209ccae0) at /root/git/swoole-src/ext-src/swoole_coroutine.cc:692
692             if (swoole_isset_hook(SW_GLOBAL_HOOK_ON_CORO_START)) {
(gdb) s
swoole_isset_hook (type=SW_GLOBAL_HOOK_ON_CORO_START) at /root/git/swoole-src/src/core/base.cc:250
250         assert(type <= SW_GLOBAL_HOOK_END);
(gdb)
251         return SwooleG.hooks[type] != nullptr;
(gdb)
swoole::PHPCoroutine::main_func (_args=0x8209ccae0) at /root/git/swoole-src/ext-src/swoole_coroutine.cc:697
697             if (EXPECTED(SWOOLE_G(enable_fiber_mock) && ctx->fci_cache.function_handler->type == ZEND_USER_FUNCTION)) {
(gdb)
707             zend_call_function(&ctx->fci, &ctx->fci_cache);
(gdb)
710             exception_caught = catch_exception();

the master process seems to be a bit more useful?

Thread 1 hit Breakpoint 1, swoole::network::Socket::send (this=0x3a24d3a34e00, __buf=0x8209cd588, __n=40, __flags=0) at /root/git/swoole-src/src/network/socket.cc:747
747         } while (retval < 0 && errno == EINTR);
(gdb) s
749         if (retval > 0) {
(gdb)
750             total_send_bytes += retval;
(gdb)
751             if (send_timer) {
(gdb)
754         }
(gdb)
758         return retval;
(gdb)
swoole::Reactor::_write(swoole::Reactor*, swoole::network::Socket*, void const*, unsigned long)::$_0::operator()() const (this=0x3a24d3aae718) at /root/git/swoole-src/src/reactor/base.cc:291
291             return send_bytes;
(gdb)
swoole::write_func(swoole::Reactor*, swoole::network::Socket*, unsigned long, std::__1::function<long ()> const&, std::__1::function<void (swoole::Buffer*)> const&) (reactor=0x3a24d3bc5900, socket=0x3a24d3a34e00, __len=40, send_fn=..., append_fn=...) at /root/git/swoole-src/src/reactor/base.cc:243
243             if (retval > 0) {
(gdb)
244                 if ((ssize_t) __len == retval) {
(gdb)
245                     return retval;
(gdb)
285     }
(gdb)
std::__1::function<void (swoole::Buffer*)>::~function() (this=0x8209cd430) at /usr/include/c++/v1/__functional/function.h:1143
1143    function<_Rp(_ArgTypes...)>::~function() {}
(gdb)
std::__1::__function::__value_func<void (swoole::Buffer*)>::~__value_func[abi:v160006]() (this=0x8209cd430) at /usr/include/c++/v1/__functional/function.h:468
468             if ((void*)__f_ == &__buf_)
(gdb)
469                 __f_->destroy();
(gdb)
std::__1::__function::__func<swoole::Reactor::_write(swoole::Reactor*, swoole::network::Socket*, void const*, unsigned long)::$_1, std::__1::allocator<swoole::Reactor::_write(swoole::Reactor*, swoole::network::Socket*, void const*, unsigned long)::$_1>, void (swoole::Buffer*)>::destroy() (this=0x8209cd430) at /usr/include/c++/v1/__functional/function.h:338
338         __f_.destroy();
(gdb)
std::__1::__function::__alloc_func<swoole::Reactor::_write(swoole::Reactor*, swoole::network::Socket*, void const*, unsigned long)::$_1, std::__1::allocator<swoole::Reactor::_write(swoole::Reactor*, swoole::network::Socket*, void const*, unsigned long)::$_1>, void (swoole::Buffer*)>::destroy[abi:v160006]() (this=0x8209cd438) at /usr/include/c++/v1/__functional/function.h:202
202         void destroy() _NOEXCEPT { __f_.~__compressed_pair<_Target, _Alloc>(); }
(gdb)
std::__1::__function::__func<swoole::Reactor::_write(swoole::Reactor*, swoole::network::Socket*, void const*, unsigned long)::$_1, std::__1::allocator<swoole::Reactor::_write(swoole::Reactor*, swoole::network::Socket*, void const*, unsigned long)::$_1>, void (swoole::Buffer*)>::destroy() (this=0x8209cd430) at /usr/include/c++/v1/__functional/function.h:339
339     }
(gdb)
std::__1::__function::__value_func<void (swoole::Buffer*)>::~__value_func[abi:v160006]() (this=0x8209cd430) at /usr/include/c++/v1/__functional/function.h:472
472         }
(gdb)
std::__1::function<long ()>::~function() (this=0x8209cd460) at /usr/include/c++/v1/__functional/function.h:1143
1143    function<_Rp(_ArgTypes...)>::~function() {}

os is freebsd 14.0-RC4, php is php82-8.2.11 from freebsd package repository, swoole is compiled from master with patches from ports and configured with --enable-swoole-dev.

[NathanFreeman]

<?php

use Swoole\Http\Server;
use Swoole\Http\Request;
use Swoole\Http\Response;

$server = new Swoole\HTTP\Server("127.0.0.1", 9501, SWOOLE_PROCESS);
$server->set([
    'enable_coroutine' => false
]);
$server->on("Start", function(Server $server)
{
    echo "Swoole http server is started at http://127.0.0.1:9501\n";
});

$server->on("Request", function(Request $request, Response $response)
{
    $response->end(str_repeat('a', 1854));
});

$server->start();

Let's try closing the coroutine and see if this issue occurs.

[nanaya]

Same empty response, yes. FWIW, I first found this problem through laravel octane which from what I can tell also disables coroutine.

~# curl -v http://127.0.0.1:9501
*   Trying 127.0.0.1:9501...
* Connected to 127.0.0.1 (127.0.0.1) port 9501
> GET / HTTP/1.1
> Host: 127.0.0.1:9501
> User-Agent: curl/8.4.0
> Accept: */*
>
* Empty reply from server
* Closing connection
curl: (52) Empty reply from server

[NathanFreeman]

It is possible that this issue is related to the operating system. In my testing, I was unable to reproduce it on FreeBSD 13.

[nanaya]

yes, it's fine on freebsd 13. I found my apps broken after upgrading to 14.

It's already RC4 so I don't expect much change until release. Yep, still empty response on 14.0-release.

[RicardoSette]

@nanaya try setting sysctl net.local.dgram.maxdgram to 8192 in FreeBSD 14.x

sysctl -w net.local.dgram.maxdgram=8192

[nanaya]

it's already of that value.

but anyway it seems to work fine now with swoole 5.1.3 on freebsd 14.1.