Fault injection - #346
Merged
Merged
Conversation
|
An automated preview of the documentation is available at https://346.corosio.prtest3.cppalliance.org/index.html If more commits are pushed to the pull request, the docs will rebuild at the same URL. 2026-09-02 21:20:41 UTC |
|
GCOVR code coverage report https://346.corosio.prtest3.cppalliance.org/gcovr/index.html Build time: 2026-09-02 21:34:49 UTC |
Add boost_corosio_fault_tests, a separate test target that shadows the OS entry points corosio calls and fails them on demand, so the error branches after each system call can be covered without touching the library's production symbols. Mechanism per platform: - ELF (Linux, FreeBSD; static and shared): strong extern "C" definitions in the executable forward through dlsym(RTLD_NEXT); the shared library's PLT binds to them. A startup readback proves the binding. Fortify aliases are shadowed so optimized builds reach the hooks, and a statically linked liburing falls back to the shared object for the real body. - Mach-O (macOS): the same definitions serve static builds; for shared builds the corosio dylib's import slots are rebound at startup by value match, with an independent readback. - PE (Windows): the import tables of the executable and boost_corosio.dll are patched, following ws2_32's ordinal imports; AcceptEx, ConnectEx and the Nt* entry points are reached by substituting the pointers WSAIoctl and GetProcAddress hand back. - io_uring: liburing's exported calls are shadowed, SQ exhaustion is reached by clamping the ring, and completions are rewritten in the mapped CQ ring. IOCP completions are rewritten in the GetQueuedCompletionStatus hook. Faults are armed per thread through fault_scope (nth call, error code or short count), with an opt-in process-wide arm for pool-thread and callback-thread calls, four independent arms per thread, and cqe and completion scopes for completion-side errors. Tests cover the posix-common, epoll, select, kqueue, io_uring and IOCP error branches. The suite builds and runs on every CI leg including FreeBSD; sanitizer configurations whose runtimes patch the same symbols (TSan, Windows ASan) are skipped explicitly. Line coverage with the published flags: Linux 92.0% to 93.2%, macOS 92.8% to 94.2%, Windows 92.6% to 94.5%.
The Boost superproject defaults BUILD_TESTING to OFF, so the FreeBSD CMake step built nothing under --target tests and ctest reported no tests. Every other CMake leg gets the option from the cpp-actions workflow.
An EINTR or EBADF from select() returned from run_task with the scheduler lock released, so the next dispatch pushed to the ready queue unsynchronized and then unlocked a lock it did not own, raising std::system_error out of io_context::run(). The other reactors fall through to the same epilogue that re-acquires the lock; select now does too, and run_task documents that it returns with the lock owned.
The SQ-full path wrote EAGAIN to the op's error slot but left res at zero, and every handler re-derives the result from res: a read reported eof, a write reported success with no bytes moved, and a connect or wait reported success for an operation that was never submitted. The path now sets res to -EAGAIN so every op kind decodes the same retryable error.
prep_multishot_poll gave up silently when no SQE was available after one flush, and register_signal_reader then returned success from a submit of nothing; the signal service latched the reader as registered and every async_wait hung. The poll now reports whether it was armed, register_signal_reader returns EAGAIN when it was not, and the registration is retried by the next add(). A full submission queue reports EAGAIN on every path, and the rulebook lists it among the corosio-generated codes.
…reated open_signal_pipe returned a bool, so a pipe() or fcntl() failure reached the caller as io_error. It now returns the errno through make_err, so descriptor exhaustion reports too_many_files_open like the scheduler's own self-pipe does.
…ract The kqueue and select traits claimed macOS lacks MSG_NOSIGNAL; Darwin defines it, and the reactor datagram paths already use it there. The flag is simply not universal across kqueue platforms, and the writev() and write() paths take no flags at all, which is why SO_NOSIGPIPE is set on every descriptor. reactor_scheduler::run_task now documents that it returns with the scheduler lock owned.
The scheduler constructor creates the completion port and then registers the timer, resolver and wait-reactor services, any of which can throw; the port handle was leaked on that path. The constructor now closes the port before rethrowing. Because the context has no service unregistration, a service registered before the throw outlives the scheduler; that is safe because a scheduler that never finished constructing handed out no timer, resolver or file, so every shutdown walks an empty list. The reason is recorded at the catch.
…nnect On Windows connect_pair builds the pair with a listener and a worker thread that connects to it. When the worker's socket creation or connect failed, the caller blocked forever in accept(). The listener is now non-blocking and the caller polls it while the worker publishes its verdict on every exit path, returning the worker's error instead of waiting for a connection that will never come.
…is constructed The wait reactor was built on the first wait(), and a failure to create its loopback wakeup pair was silent: the poll thread started on an invalid socket, every write and error wait parked forever, and run() never returned. The pair is now created when the scheduler is constructed, so the failure throws from the io_context constructor like every other backend's infrastructure, while the poll thread still starts on the first wait. The reactor owns its own Winsock reference, a zero last-error can no longer disguise a failure, and a cancel for a wait that was never parked is ignored rather than answered against the next wait; a wait registered after stop() completes as cancelled. The thread is still the first wait's cost, but a system that refuses one now completes that wait with resource_unavailable_try_again and leaves the reactor to try again on the next, rather than throwing out of an async initiator.
The ring was created lazily on the first run() or operation, so an initialization failure escaped from whichever call happened first. It is now created at the end of every io_context constructor, after the options that select its flags have been applied, and a failure throws from the constructor like every other backend. A failed initialization rolls back the wakeup eventfd it opened. The fault harness rewrites completions from every submit shadow, since a buffered write can complete inside the submit itself.
The IOCP wait reactor treated a failed wakeup send as fatal from inside a noexcept function, and epoll and kqueue latched their armed flag before the wake call and discarded its result, so one failure disabled every later interrupt. A failed wake now leaves the flag clear so the next interrupt retries; the only cost is the interrupts already in flight. Fault tests arm the wake call to fail once on epoll, kqueue and IOCP and assert a later stop() still wakes the loop.
When WSAPoll failed the reactor thread left its loop and drained the parked waits as cancelled, but a wait registered afterwards parked with nothing to complete it. The reactor now records that it is dead under the same lock that drains the queue, and later registrations and cancellations complete as cancelled at the caller.
Three sites built the error code by hand from errno or GetLastError instead of through make_err, bypassing the normalization every other path applies.
…t arm The multishot accept arm is deliberately not counted as outstanding work, but the SQ-full path queued it for dispatch like any other op, so the dispatch's work_finished() underflowed the count and run() never returned. An uncounted op that cannot be submitted is no longer queued; the acceptor records the failure, completes every parked accept with EAGAIN, and a later listen() re-arms it. Which of the two answers an op needs is a property of its call site, not state the op has to carry, so the submission helper says it in its name rather than reading a bit: io_uring_submit_op for the ops a work_started() paid for, which answer a full ring through their own EAGAIN completion and so report nothing, and io_uring_try_submit_op for the ones nothing counted, which hand the refusal back. That leaves the invariant to the type system instead of to a flag every io_uring op would carry and any of them could set by mistake, and the ops keep the layout they had.
sgerbino
force-pushed
the
pr/fault-injection
branch
from
September 2, 2026 20:40
a81f089 to
ead4cb8
Compare
…s, per-process suites Seams the coverage rounds needed and the harness did not have. fault_scope::count() reports the calls an arm has seen, so a test that must reach past the calls the library makes on the way in can read the ordinal off a counting scope instead of hard-coding one. cqe_fault_scope gained an overload that clears CQE flag bits and an fd of -1 that matches on the opcode alone, since IORING_CQE_F_MORE is cleared only by the kernel on descriptors the library never hands out. hook_is_live answers on POSIX as well as Windows, and fd_wall raises every free low descriptor so the kernel's next one lands above FD_SETSIZE. Suites that fault state created once per process get an executable of their own: the two signal suites on POSIX and, on Windows, iocp_dissociate_faults for the NtSetInformationFile lookup that a function-local static caches on the first release().
in_child ends the child with _Exit, which runs no atexit handler, so an instrumented child discarded everything it counted: a branch only the child reached measured as unreached however often the test passed. The child now calls __gcov_dump when the binary has one, found by dlsym at runtime because a weak declaration is refused by ld64 and a weak reference does not extract the archive member anyway. The CMake build forces the entry point in with -u under coverage flags, spelled ___gcov_dump on Mach-O. Boost's coverage script passes --coverage as a raw linkflag rather than <coverage>on, so the b2 build probes for libgcov by linking against it and only then adds -u together with --export-dynamic-symbol, which the runtime lookup needs on ELF.
select() addresses descriptors by bit position in an fd_set, so a number at or above FD_SETSIZE has nowhere to go. The three entry points that can be handed one reject it, and none of those rejections had ever run: a descriptor number is not something a caller picks. The tests adopt a descriptor duplicated above the range and raise a wall over every free low number so open() and accept() land above it. Where the platform defines SO_NOSIGPIPE the backend sets it on every descriptor it creates or accepts and treats a refusal as fatal, since its write() fallback carries no per-call suppression. Both arms check the descriptor the failure path owns was released and that the accept arm reports the option's errno rather than the close()'s.
…datagram, the shutdown walk The signal service faults state created once per process, the self-pipe and the sigaction table, so those tests run as suites of their own. The rest fill in the plain-syscall legs the posix layer owns: connect_pair's rollback when the second socket cannot be made, the local datagram available() probe and the throwing convenience constructors. The signal service's shutdown walk deletes registrations still parked on a wait. It is reached from a forked child, since an abandoned signal_set leaves its SIGINT registration process-wide, and the test requires the child to have parked something before the walk runs.
An acceptor reaches the reactor two ways and only one of them was tested. listen() registers a descriptor the library made; assign() registers one the caller made, and only that path has to hand the descriptor back untouched when the reactor refuses it. The accept tests so far all resolved on the initiator's own speculative accept. A first accept with no peer waiting parks the operation, so the reactor's retry runs a different set of arms: the accept's own error leg and the registration of the accepted descriptor from the posted completion, with its rollback.
Three registration and configuration legs had no kqueue caller. An acceptor's open goes through a configure function of its own, an acceptor adopted through assign registers from a third place, and an accept the reactor dispatches registers the descriptor it produced from the completion rather than from the initiator. kevent is this backend's only registration symbol and the run loop waits on it too, so an ordinal over the whole call sequence was stale the moment scheduling shifted. The arm now lands on the registration itself, a slice of the kevent shadow that fires only on a changelist carrying EV_ADD; when both arms match one call the plain kevent arm publishes last and its errno is the one the caller sees.
Every io_uring object type creates and configures its descriptor with plain syscalls, and each of those legs returns an error rather than throwing. None had run: the ring's own faults reach the operations, not the surface underneath them. The tests walk socket, listen, bind and shutdown across the TCP, UDP, AF_UNIX stream and AF_UNIX datagram types and their acceptors, and refuse the SO_ACCEPTCONN query an adopted descriptor is probed with.
The dispatch completes a parked operation with the error instead of running its I/O, and it has a separate arm for each kind: read, write, wait-for-read, wait-for-write, wait-for-error. Only the read arm had ever run. Each test parks its own kind and faults the SO_ERROR probe, so what the operation reports is the armed code rather than whatever the kernel recorded. A writable descriptor never parks, so the write-direction tests park with a refused small write rather than with backpressure, check that premise themselves and bound the runs a failed premise could leave going. They stay on Linux: kqueue reports a socket error through the read filter, and select's except set answers only for urgent data, so those backends carry no write-direction error arm to reach. The probe for writability and the except set is asked in one select and the premise the reactor's round did not carry is reported rather than asserted.
…ters never loaded, POLLPRI The deferred queue had no test at all: every post in the suite succeeded, so neither the queue nor the drain that empties it was ever reached. Three arms on the same symbol walk a post from the continuation overload through the allocating handle overload and into the drain's own re-post. load_extension_functions asks Winsock for AcceptEx and ConnectEx through a socket of its own; failing it leaves both pointers null and every accept and connect after it refused. The wait reactor asks WSAPoll for POLLPRI on an error wait. Rather than assume what the provider makes of that, a test polls with POLLPRI directly and then measures what an error wait costs the context: on the Microsoft provider the poll fails, the reactor's loop ends and a write wait registered afterwards is refused as canceled. The SO_ERROR probe is reached through a write wait instead, with the peer reset before the wait is registered.
Three of these are not failures at all: a data-only flush that declines still syncs, an accept that reports WSAEWOULDBLOCK after a readiness report still forms the pair, and a reverse lookup whose conversion gives up still succeeds with an empty name. Each is a branch nothing reached because nothing had a reason to take it. The rest are ordinary rollbacks: adoption refused on a file handle, SetEndOfFile refused on a truncating open, and the second of connect_pair's two adoptions refused, which is the one that has a live socket on each side to close.
… at shutdown The parked-operation teardown tests reach the services that cancel an outstanding operation, but never the handler arm that runs when a completion is already sitting in the scheduler queue at destruction: the run loop had always delivered everything it reaped. Making two operations ready and dispatching one leaves the other for shutdown, which is the proactor's !owner path and the reactor's queued-op destroy, for reads, waits, writes, connects, datagrams, accepts and acceptor waits. A service's shutdown walk over implementations still alive had no caller either, since an io object cannot outlive its context. Keeping the object inside an abandoned coroutine frame reaches it.
events_for_wait mapped wait_type::error to POLLPRI, which the Microsoft Winsock provider does not implement and refuses the whole WSAPoll call for. One error wait therefore ended the reactor's polling loop for the entire io_context: it marked itself dead, drained every op it held as aborted, and refused every later register_wait, so wait(wait_type::write) on that context came back canceled from then on. Measured on CI, not inferred -- the poll returns SOCKET_ERROR for a set carrying the bit. Ask for POLLRDBAND instead. It carries the same out-of-band meaning, the provider implements it, and the error conditions an error wait is really after arrive in revents whether or not they were asked for, so a peer reset answers the wait as it does on the reactor backends. A poll refused on account of one descriptor no longer takes the other registrations with it either: the reactor asks about each entry alone, answers the ones the provider will not take with the refusal, and keeps polling the rest. A socket closed under a parked wait reaches that path in ordinary use, since the reactor is told to drop the entry one pass after the handle is already gone. A failure no entry accounts for still ends the loop, and the ops it was holding are now told what happened rather than that someone cancelled them. So is every wait registered afterwards: the reactor latches the code it died of and answers with it, so a caller learns why its waits can no longer be served instead of hearing that something cancelled them. canceled is left to the ops close() and cancel() flag, and to the ops a stop() drain finds parked. Tests: the fault suite asks the provider directly what it makes of the bit the reactor registers with, then asserts the contract -- a reset answers the error wait, and a wait registered afterwards is answered too -- and a new test closes a descriptor under a parked wait to show the refusal costs that wait alone. The templated tcp_socket suite gains the cross-backend half: an error wait, then a wait that has to work.
A pool completion posted after the scheduler had drained its queue was neither run nor destroyed. The operation's keepalive on the file or resolver implementation, and the coroutine frame waiting on it, leaked. Services shut down newest first, and everything a scheduler's constructor nests inside itself is older than that scheduler. The file, random-access-file and resolver services created the pool from their own constructors, so the pool was older still and joined its workers only after the drain had run: the item the last worker executed posted into a queue nothing would ever read again. The io_context now creates the pool once the backend is constructed, and those services bind to it on first use rather than in their constructors. That puts the pool after the scheduler in the shutdown walk, so the workers are joined and whatever they posted on the way out is drained. Reordering the services instead would move the guarantee away from the thing that has to be quiet before the drain — the pool, not its callers — and would leave the configured pool size nowhere to be created, since a service constructed before the scheduler cannot be the one the options size. What the walk needs in that position is the service, not its threads. Starting them with it would park a worker in every process that never opens a file and never resolves a name, so the constructor now records the size and nothing else, and the workers start on the first post. Starting a thread can fail, and post() runs on an initiator's thread, where nothing may throw. It answers with a code instead, and the initiator completes the operation with it there and then: the refusal is known before any part of the operation has gone cross-thread, so it takes the exit the closed-descriptor and zero-length contracts already take a few lines above it rather than marking the operation cancelled and posting a completion back through the scheduler to undo the lie. A system that will not give the pool a worker is reported as the error it is, never as the cancellation a stop token means. A pool that is shutting down answers with that cancellation, which is what its callers have reported for the refusal all along. A start that yields fewer threads than asked for still yields a pool that runs everything posted to it, so only a start that yields none refuses the post; a pool short of workers never tries for the rest again, since topping it up would put a thread creation on the initiator's path for every operation after a refusal. The pool service is symbol-visible for the same reason. A service is keyed by its type, and binding on first use moves that lookup out of the compiled library and into whichever translation unit instantiated the initiator; hidden behind a shared library boundary the type would be two types, and the second module would create a pool of its own. The pool's destructor joins as a backstop. The context's walk stays the normal path; a pool created after that walk never gets a `shutdown()` at all, and joinable threads left in one would terminate the process. The resolver's worker dropped its implementation keepalive as soon as it posted, which was harmless only while nothing read the queue again. It now hands the keepalive to the completion, as the file operations already do, and releases it on the drain path. The two resolver operations hold it in the slot coro_op already documents for it rather than in a member of their own: a keepalive nobody remembers to declare is how this was missed, and an inherited slot is one a reviewer can look for. Moving them onto that base also puts their work accounting on the executor the initiator counted, at both ends and on the drain path, which is what the file operations were already doing. The regression tests park a real operation behind a blocked pool worker that a test service releases at the top of teardown, and own the task they start so that the frame the library abandons is not mistaken for the leak under test. The blocker is the first item posted, so it is the one the worker it starts picks up, and the operation behind it stays queued.
The signal service's shutdown walk deleted each registration node with a bare delete: it never decremented the process-global registration count, never restored the disposition when that count reached zero, and never reset the flags the first registration established. A signal_set held by an abandoned coroutine frame -- the only way one can still be registered when its io_context is destroyed -- therefore left the signal installed for the life of the process, and the next add() of it was refused as an incompatible-flags conflict. The walk now gives each registration's count and disposition back the way clear() does. It drops the per-service table wholesale instead of unlinking node by node: every live registration hung off an implementation the walk just deleted, so the whole table goes stale at once, and deliver_signal() walks this service until the destructor unlinks it. The Windows service had the same hole and gets the same treatment.
The POSIX pool-path stream_file::file_op and random_access_file::raf_op each hand-rolled the coroutine handle, executor, output pointers, cancelled flag, stop_callback and impl keepalive that coro_op already provides. Rebase both onto coro_op so that state, the canceller wiring and start() are inherited: the per-op keepalive becomes the shared impl_ptr slot, and each op's nested canceller (which only recorded the request) is subsumed by coro_op's default on_cancel(). raf_op keeps its typed file_ raw back-pointer for the work path; only the keepalive loses its type, and it was never dereferenced as typed. Each op retains its genuinely-extra state: iovecs, iovec_count, errn, bytes_transferred, and raf_op's offset. Keepalive lifetime, virtual dispatch, cancellation and work accounting are unchanged.
A wait(error) satisfied by IORING_OP_POLL_ADD reports the readiness in the CQE's res as revents, so res stays >= 0 and the handler produced an empty error_code -- indistinguishable from a benign readiness signal, where epoll/select/kqueue/IOCP all deliver a named code. When the revents carry POLLERR/POLLHUP/POLLNVAL, probe SO_ERROR and complete with that (EIO when the kernel exposes none), mirroring the reactor and the IOCP wait reactor. Out-of-band data (POLLPRI) stays a readiness signal, not an error.
An accepted descriptor at or above FD_SETSIZE cannot be monitored by select(), the same logical failure the adoption (validate_assigned_fd) and creation (set_fd_options) range checks already report as EMFILE. The accept path reported EINVAL instead; make it EMFILE so a caller distinguishing "too many files" from "bad argument" sees one answer for one condition.
select() raises its exceptional set for TCP urgent/out-of-band data as well as for genuine faults, and the dispatch folded that set into the error condition and synthesized EIO when SO_ERROR came back zero. A pending read or write on an otherwise-healthy socket carrying an urgent byte then completed with EIO on select, where epoll (EPOLLPRI is not the error bit) and kqueue (OOB is not EV_ERROR) return the data cleanly. Out-of-band data is a readiness condition, not an error: an I/O operation now completes on a real (non-zero) SO_ERROR only, so a genuine fault still surfaces through the probe every reactor backend uses. Only wait(error) keeps the EIO fallback, so a wait that fires on the exceptional condition still names a code. Fixing the read path also corrects the write-direction fault-injection tests, which were asserting the spurious EIO on a socket that was in no way broken.
The forward GetAddrInfoExW path was the only resolver path (of the IOCP reverse path and both POSIX paths) that held no impl keepalive across its completion. The op is embedded in the win_resolver, its OS callback calls work_finished() before posting the completion, and the completion then waits in the scheduler queue -- so abandoning or tearing down a context with a forward lookup in flight could free the win_resolver before the queued op drained, a use-after-free. Mirror the reverse path: take shared_from_this() into the op's impl_ptr at initiation and move it out on both do_complete arms -- a suicide local on the drain arm, held across the dispatch on the resume arm -- so the implementation outlives the queued completion.
sgerbino
force-pushed
the
pr/fault-injection
branch
from
September 2, 2026 21:16
ead4cb8 to
72f550f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces a fault injection harness and numerous fixes related to that work. Overall improvement to error handling, consistency between backends, and code coverage.