Onboard FreeBSD GitHub Action

[nibanks]

Describe the feature you'd like supported

We have no coverage for FreeBSD, but are trying to support it. We should onboard some automation to try to run tests on it.

Proposed solution

I found this that might allow us to run our tests: https://github.com/marketplace/actions/freebsd-vm.

Additional context

No response

[nibanks]

Walkthrough

Leverage the link in the description to onboard FreeBSD build & test automation.

[0xFA11]

I managed to run FreeBSD14 inside a Hyper-V VM on Windows and bumped into several compile errors.

I don't think FreeBSD support is too hard to achieve, it should be similar to MacOS for the most part.

PR #2918 from ~2 years ago fixed/ported some of the platform specific things to FreeBSD but without FreeBSD in the CI pipeline and not being an "official" platform, things get added (mainly to platform/datapath CXPLAT_ stuff) and ultimately FreeBSD is now broken.

I'm not sure about vmactions/freebsd-vm action but it'd be interesting to have a premade Windows image that comes with Hyper-V enabled, FreeBSD14 VM configured and ready-to-go, that premade Windows image then being used to test basic platform support as a part of CI pipeline. I personally didn't get to make this workflow work but worth noting for future fellow devs to consider.

I'm posting compiler output here for context and future reference:

root@freebsd14vm:~/msquic/build # cmake -G 'Unix Makefiles' ..
-- CMAKE Version: 3.28.3
-- Source Dir: /root/msquic
-- Host System name: FreeBSD
-- Setting policy 0091
-- System name: FreeBSD
-- System version: 14.0-RELEASE
-- Platform version:
-- Build type: Release
-- QUIC Platform: linux
-- Version Build ID: 0
-- Version Suffix: -private
-- UDP_SEGMENT is missing. Send performance will be reduced
-- SO_ATTACH_REUSEPORT_CBPF is missing. Server receive performance will be reduced
-- Enabling OpenSsl configuration tests
-- Enabling CA file tests
-- QUIC_ENABLE_LOGGING is false. Disabling logging
-- libatomic not found. If build fails, install libatomic
-- libnuma not found. If build fails, install libnuma
-- Enabling shared ephemeral port work around
-- Configuring for OpenSSL 1.1
-- Setting openssldir to /etc/ssl
-- Configuring OpenSSL: /root/msquic/submodules/openssl/config;CC=/usr/bin/cc;CXX=/usr/bin/c++ enable-tls1_3;no-makedepend;no-dgram;no-ssl3;no-psk;no-srp;no-zlib;no-egd;no-idea;no-rc5;no-rc4;no-afalgeng;no-comp;no-cms;no-ct;no-srp;no-srtp;no-ts;no-gost;no-dso;no-ec2m;no-tls1;no-tls1_1;no-tls1_2;no-dtls;no-dtls1;no-dtls1_2;no-ssl;no-ssl3-method;no-tls1-method;no-tls1_1-method;no-tls1_2-method;no-dtls1-method;no-dtls1_2-method;no-siphash;no-whirlpool;no-aria;no-bf;no-blake2;no-sm2;no-sm3;no-sm4;no-camellia;no-cast;no-md4;no-mdc2;no-ocb;no-rc2;no-rmd160;no-scrypt;no-seed;no-weak-ssl-ciphers;no-shared;no-tests;--openssldir="/etc/ssl";--prefix=/root/msquic/build/_deps/opensslquic-build/openssl
-- Configuring for OpenSSL
-- Configuring done (0.1s)
-- Generating done (0.0s)
-- Build files have been written to: /root/msquic/build
root@freebsd14vm:~/msquic/build # cmake --build .
[  1%] Building C object src/core/CMakeFiles/core.dir/ack_tracker.c.o
[  3%] Building C object src/core/CMakeFiles/core.dir/api.c.o
[  5%] Building C object src/core/CMakeFiles/core.dir/binding.c.o
[  6%] Building C object src/core/CMakeFiles/core.dir/configuration.c.o
[  8%] Building C object src/core/CMakeFiles/core.dir/congestion_control.c.o
[ 10%] Building C object src/core/CMakeFiles/core.dir/connection.c.o
[ 12%] Building C object src/core/CMakeFiles/core.dir/crypto.c.o
[ 13%] Building C object src/core/CMakeFiles/core.dir/crypto_tls.c.o
[ 15%] Building C object src/core/CMakeFiles/core.dir/cubic.c.o
[ 17%] Building C object src/core/CMakeFiles/core.dir/bbr.c.o
[ 18%] Building C object src/core/CMakeFiles/core.dir/datagram.c.o
[ 20%] Building C object src/core/CMakeFiles/core.dir/frame.c.o
[ 22%] Building C object src/core/CMakeFiles/core.dir/library.c.o
[ 24%] Building C object src/core/CMakeFiles/core.dir/listener.c.o
[ 25%] Building C object src/core/CMakeFiles/core.dir/lookup.c.o
[ 27%] Building C object src/core/CMakeFiles/core.dir/loss_detection.c.o
[ 29%] Building C object src/core/CMakeFiles/core.dir/mtu_discovery.c.o
[ 31%] Building C object src/core/CMakeFiles/core.dir/operation.c.o
[ 32%] Building C object src/core/CMakeFiles/core.dir/packet.c.o
[ 34%] Building C object src/core/CMakeFiles/core.dir/packet_builder.c.o
[ 36%] Building C object src/core/CMakeFiles/core.dir/packet_space.c.o
[ 37%] Building C object src/core/CMakeFiles/core.dir/path.c.o
[ 39%] Building C object src/core/CMakeFiles/core.dir/range.c.o
[ 41%] Building C object src/core/CMakeFiles/core.dir/recv_buffer.c.o
[ 43%] Building C object src/core/CMakeFiles/core.dir/registration.c.o
[ 44%] Building C object src/core/CMakeFiles/core.dir/send.c.o
[ 46%] Building C object src/core/CMakeFiles/core.dir/send_buffer.c.o
[ 48%] Building C object src/core/CMakeFiles/core.dir/sent_packet_metadata.c.o
[ 50%] Building C object src/core/CMakeFiles/core.dir/settings.c.o
[ 51%] Building C object src/core/CMakeFiles/core.dir/stream.c.o
[ 53%] Building C object src/core/CMakeFiles/core.dir/stream_recv.c.o
[ 55%] Building C object src/core/CMakeFiles/core.dir/stream_send.c.o
[ 56%] Building C object src/core/CMakeFiles/core.dir/stream_set.c.o
[ 58%] Building C object src/core/CMakeFiles/core.dir/timer_wheel.c.o
[ 60%] Building C object src/core/CMakeFiles/core.dir/worker.c.o
[ 62%] Building C object src/core/CMakeFiles/core.dir/version_neg.c.o
[ 63%] Building C object src/core/CMakeFiles/core.dir/sliding_window_extremum.c.o
[ 65%] Building C object src/core/CMakeFiles/core.dir/inline.c.o
[ 67%] Linking C static library ../../obj/Release/libcore.a
[ 67%] Built target core
[ 70%] Built target OpenSSL_Target
[ 72%] Building C object src/platform/CMakeFiles/platform.dir/crypt.c.o
[ 74%] Building C object src/platform/CMakeFiles/platform.dir/hashtable.c.o
[ 75%] Building C object src/platform/CMakeFiles/platform.dir/pcp.c.o
[ 77%] Building C object src/platform/CMakeFiles/platform.dir/platform_worker.c.o
[ 79%] Building C object src/platform/CMakeFiles/platform.dir/toeplitz.c.o
[ 81%] Building C object src/platform/CMakeFiles/platform.dir/inline.c.o
[ 82%] Building C object src/platform/CMakeFiles/platform.dir/platform_posix.c.o
[ 84%] Building C object src/platform/CMakeFiles/platform.dir/storage_posix.c.o
[ 86%] Building C object src/platform/CMakeFiles/platform.dir/cgroup.c.o
[ 87%] Building C object src/platform/CMakeFiles/platform.dir/datapath_kqueue.c.o
/root/msquic/src/platform/datapath_kqueue.c:158:32: error: redefinition of 'CXPLAT_SOCKET_CONTEXT'
typedef struct QUIC_CACHEALIGN CXPLAT_SOCKET_CONTEXT {
                               ^
/root/msquic/src/platform/platform_internal.h:692:32: note: previous definition is here
typedef struct QUIC_CACHEALIGN CXPLAT_SOCKET_CONTEXT {
                               ^
/root/msquic/src/platform/datapath_kqueue.c:232:3: error: typedef redefinition with different types ('struct (unnamed struct at /root/msquic/src/platform/datapath_kqueue.c:158:32)' vs 'struct CXPLAT_SOCKET_CONTEXT')
} CXPLAT_SOCKET_CONTEXT;
  ^
/root/msquic/src/platform/platform_internal.h:756:3: note: previous definition is here
} CXPLAT_SOCKET_CONTEXT;
  ^
/root/msquic/src/platform/datapath_kqueue.c:237:16: error: redefinition of 'CXPLAT_SOCKET'
typedef struct CXPLAT_SOCKET {
               ^
/root/msquic/src/platform/platform_internal.h:761:16: note: previous definition is here
typedef struct CXPLAT_SOCKET {
               ^
/root/msquic/src/platform/datapath_kqueue.c:294:3: error: typedef redefinition with different types ('struct (unnamed struct at /root/msquic/src/platform/datapath_kqueue.c:237:16)' vs 'struct CXPLAT_SOCKET')
} CXPLAT_SOCKET;
  ^
/root/msquic/src/platform/platform_internal.h:832:3: note: previous definition is here
} CXPLAT_SOCKET;
  ^
/root/msquic/src/platform/datapath_kqueue.c:299:32: error: redefinition of 'CXPLAT_DATAPATH_PARTITION'
typedef struct QUIC_CACHEALIGN CXPLAT_DATAPATH_PARTITION {
                               ^
/root/msquic/src/platform/platform_internal.h:837:32: note: previous definition is here
typedef struct QUIC_CACHEALIGN CXPLAT_DATAPATH_PARTITION {
                               ^
/root/msquic/src/platform/datapath_kqueue.c:348:3: error: typedef redefinition with different types ('struct (unnamed struct at /root/msquic/src/platform/datapath_kqueue.c:299:32)' vs 'struct CXPLAT_DATAPATH_PARTITION')
} CXPLAT_DATAPATH_PARTITION;
  ^
/root/msquic/src/platform/datapath_kqueue.c:153:42: note: previous definition is here
typedef struct CXPLAT_DATAPATH_PARTITION CXPLAT_DATAPATH_PARTITION;
                                         ^
/root/msquic/src/platform/datapath_kqueue.c:354:16: error: redefinition of 'CXPLAT_DATAPATH'
typedef struct CXPLAT_DATAPATH {
               ^
/root/msquic/src/platform/platform_internal.h:881:16: note: previous definition is here
typedef struct CXPLAT_DATAPATH {
               ^
/root/msquic/src/platform/datapath_kqueue.c:386:3: error: typedef redefinition with different types ('struct (unnamed struct at /root/msquic/src/platform/datapath_kqueue.c:354:16)' vs 'struct CXPLAT_DATAPATH')
} CXPLAT_DATAPATH;
  ^
/root/msquic/src/platform/platform_internal.h:940:3: note: previous definition is here
} CXPLAT_DATAPATH;
  ^
/root/msquic/src/platform/datapath_kqueue.c:423:29: error: no member named 'SendBufferPool' in 'struct CXPLAT_DATAPATH_PARTITION'
        &DatapathPartition->SendBufferPool);
         ~~~~~~~~~~~~~~~~~  ^
/root/msquic/src/platform/datapath_kqueue.c:428:29: error: no member named 'LargeSendBufferPool' in 'struct CXPLAT_DATAPATH_PARTITION'
        &DatapathPartition->LargeSendBufferPool);
         ~~~~~~~~~~~~~~~~~  ^
/root/msquic/src/platform/datapath_kqueue.c:433:29: error: no member named 'SendDataPool' in 'struct CXPLAT_DATAPATH_PARTITION'
        &DatapathPartition->SendDataPool);
         ~~~~~~~~~~~~~~~~~  ^
/root/msquic/src/platform/datapath_kqueue.c:528:52: error: no member named 'SendDataPool' in 'struct CXPLAT_DATAPATH_PARTITION'
        CxPlatPoolUninitialize(&DatapathPartition->SendDataPool);
                                ~~~~~~~~~~~~~~~~~  ^
/root/msquic/src/platform/datapath_kqueue.c:529:52: error: no member named 'SendBufferPool' in 'struct CXPLAT_DATAPATH_PARTITION'
        CxPlatPoolUninitialize(&DatapathPartition->SendBufferPool);
                                ~~~~~~~~~~~~~~~~~  ^
/root/msquic/src/platform/datapath_kqueue.c:530:52: error: no member named 'LargeSendBufferPool' in 'struct CXPLAT_DATAPATH_PARTITION'
        CxPlatPoolUninitialize(&DatapathPartition->LargeSendBufferPool);
                                ~~~~~~~~~~~~~~~~~  ^
/root/msquic/src/platform/datapath_kqueue.c:1104:24: error: no member named 'CurrentRecvBlock' in 'struct CXPLAT_SOCKET_CONTEXT'
    if (SocketContext->CurrentRecvBlock != NULL) {
        ~~~~~~~~~~~~~  ^
/root/msquic/src/platform/datapath_kqueue.c:1105:46: error: no member named 'CurrentRecvBlock' in 'struct CXPLAT_SOCKET_CONTEXT'
        CxPlatRecvDataReturn(&SocketContext->CurrentRecvBlock->RecvPacket);
                              ~~~~~~~~~~~~~  ^
/root/msquic/src/platform/datapath_kqueue.c:1108:47: error: no member named 'PendingSendDataHead' in 'struct CXPLAT_SOCKET_CONTEXT'
    while (!CxPlatListIsEmpty(&SocketContext->PendingSendDataHead)) {
                               ~~~~~~~~~~~~~  ^
/root/msquic/src/platform/datapath_kqueue.c:1111:54: error: no member named 'PendingSendDataHead' in 'struct CXPLAT_SOCKET_CONTEXT'
                CxPlatListRemoveHead(&SocketContext->PendingSendDataHead),
                                      ~~~~~~~~~~~~~  ^
/root/msquic/src/inc/quic_platform.h:46:26: note: expanded from macro 'CXPLAT_CONTAINING_RECORD'
    ((type *)((uint8_t*)(address) - offsetof(type, field)))
                         ^~~~~~~
/root/msquic/src/platform/datapath_kqueue.c:1118:101: error: no member named 'IoCqeType' in 'struct CXPLAT_SOCKET_CONTEXT'
        EV_SET(&DeleteEvent, SocketContext->SocketFd, EVFILT_READ, EV_DELETE, 0, 0, &SocketContext->IoCqeType);
                                                                                     ~~~~~~~~~~~~~  ^
/usr/include/sys/event.h:58:16: note: expanded from macro 'EV_SET'
            .udata = (f),                       \
                      ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
gmake[2]: *** [src/platform/CMakeFiles/platform.dir/build.make:202: src/platform/CMakeFiles/platform.dir/datapath_kqueue.c.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:240: src/platform/CMakeFiles/platform.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2

[am11]

cc @Thefrank @scopedog does the build error look familiar?

[scopedog]

Well, support for FreeBSD is not that easy as kqueue handling of msquic is actually not correct. Even if you succeed with compiling, it won't work as you wish. Interestingly, the implementation and behavior of kqueue on MacOS seems to be different from that on FreeBSD. Therefore, you have to rewrite most part of kqueue code in msquic.

[Thefrank]

The compiler is correct here, they are being redefined. Changing the CX_PLATFORM_LINUX out to something that is not defined (or using include guards) will continue the build but it will still fail as ipi_spec_dst is not defined (nor the entire IP_PKTINFO struct) under FreeBSD. You can define (or remove) ipi_spec_dst and get the build to finish but I don't think it will work correctly as @scopedog mentioned kqueue is different between OSX and FreeBSD.

Getting it to build should not be too difficult as the bulk of the work has already been done.
Getting it working correctly under FreeBSD is different story :(

[scopedog]

Moreover, I actually modified msquic so it worked with FreeBSD's kqueue (https://github.com/scopedog/msquic-freebsd) but it was really slow as kqueue needed to be restarted (or reinitialized?) every time data was received. Also, it didn't pass their automated tests (I don't know the details of the erros). I think it is possible to make it more efficient but I don't have time.

[am11]

ChatGPT thinks other options can be tried:

If FreeBSD users are finding that kqueue is slow in the MsQuic port, a few alternative approaches might improve performance:

1. Direct use of epoll via a Linux Compatibility Layer

   FreeBSD offers a Linux Compatibility Layer that can enable the use of epoll, which is known to be efficient on Linux for handling large numbers of file descriptors and high-performance network operations. While not native to FreeBSD, this could provide a faster alternative to kqueue.

2. Polling via select or poll

   While select and poll are not as scalable or efficient as kqueue or epoll, they could provide a more predictable performance for certain types of workloads, especially if MsQuic’s usage of kqueue is causing bottlenecks. However, these alternatives typically don’t scale well for high-throughput scenarios.

3. Libevent or Libuv Libraries

   Using libraries like Libevent or Libuv, which abstract the underlying event mechanism (kqueue, epoll, select) and optimize for performance across platforms, could be beneficial. These libraries can handle the backend differences and may provide a more optimized handling of events in FreeBSD.
   Libevent supports kqueue, poll, and epoll, and its optimizations might alleviate the slowness seen in MsQuic's current FreeBSD port.

4. Tuning Kqueue Parameters

   Performance might also be improved by tuning kernel parameters related to kqueue in FreeBSD. Modifying buffer sizes, adjusting CPU affinity, or optimizing I/O queues might mitigate the performance issues being observed.

5. Custom MsQuic Event Loop Implementation

   If none of the above alternatives significantly boost performance, a custom event loop for FreeBSD could be developed within MsQuic, tailored to FreeBSD’s strengths and quirks. This would involve bypassing the kqueue layer for performance-critical sections of the code and utilizing a more efficient polling or batching strategy for network events.

These are potential alternatives, and some may be more feasible than others depending on the specific nature of the bottleneck and the MsQuic workload on FreeBSD.

Option 3 is not bad either as it's pretty battle-tested; e.g. the node.js runtime is using it for about forever and a day.