Repository navigation
Example glibc condlock
A small C workload built to look for a known glibc condition-variable lost wakeup (glibc bug 25847), together with the self-tests of its stuck-state detector. It is the example to read if you want to see how much care a property needs before you can trust it, and what an honest "we did not get a result" looks like.
Files: examples/glibc-condlock. Full write-up: docs/showcase-glibc-lost-wakeup.md.
Build and run the workload and its detector self-tests on your own machine, see the planted faults fire the detector, and understand what the glibc experiment did and did not measure.
Read this first. The full experiment (a pinned glibc built from source, run under Hermit) needs a dedicated Linux x86-64 machine and scripts that must be edited before they run. It is a research artefact. It stopped, by its own stopping rule, before the counted Hermit runs, so it makes no statement about whether Hermit reaches the bug on the unmodified library. The steps below are the part a beginner can run: the workload, its unit tests and its detector self-tests, on a Mac or on Linux.
glibc's condition variable splits waiters into two groups and moves group slots forward as signals are delivered. In the affected source state a waiter that wakes takes a signal with a compare-and-swap and only then checks whether the signal belonged to a newer group. If it is descheduled between those two steps while the group slots rotate twice, it can wrongly conclude it "stole" a signal and return one to a group that has no waiter left. Later signals then go to an empty group while real waiters sleep for ever. The upstream fix removes signal stealing altogether and was backported to the 2.38 to 2.40 branches. The window is a few instructions with no system call in it, so it needs a preemption at exactly the wrong point.
condlock has N worker threads (default 8) contending for a lock built from a mutex and a condition variable (an untimed pthread_cond_wait, an unconditional pthread_cond_signal on release). They meet at a barrier every round. A watchdog thread decides whether a signal was lost.
The property, glibc.cond.signal_wakes_one_of_blocked, is evaluated by the watchdog. A releaser holding the lock's own mutex records which workers were blocked in pthread_cond_wait when it signalled. After a quorum of ticks with no progress, the watchdog takes the mutex (by trylock and short sleeps, never an untimed lock) and checks one rule: the lock is free, the recorded set is not empty, and every worker in that set is still blocked in the same wait. If so, the property is false and the run ends with a state dump.
If the program is quiet but the rule does not hold, it is a stall, not a property failure. The subtypes:
| Subtype | Meaning |
|---|---|
| Q | lock free, at least one worker still blocked on the lock's condition variable (rule did not hold) |
| L | lock held (leaked or held): nobody can make progress |
| B | lock free, no lock waiter, barrier incomplete |
| M | the lock's mutex could not be taken for a whole tick |
condlock [-n N] [-r R] [-tick-ms T] [-q Q] defaults 8, 500, 100, 10
exit status: 0 completed, 3 property false, 4 stall, 5 harness invariant, 64 usage (never 1)
It never exits 1, so that a status of 1 cannot be confused with Hermit's own report. scripts/classify.sh turns one run's exit status and outputs into a class.
The design goal is a property that does not fire falsely. Under Hermit's time model a runnable thread that stays unscheduled for the whole quorum would be needed for a false fire; natively the rule is a heuristic, because an operating-system scheduler can starve a runnable thread, so false fires of the native heuristic were counted on the fixed library.
- The bug is known in advance.
- The workload is targeted at that bug.
- Its structure is taken from a public reproducer (no code was copied).
- It is instrumented with the C SDK (Guest-SDKs-by-Language).
| For | You need |
|---|---|
| The steps below |
make, a C11 compiler, POSIX threads. Works on macOS and Linux |
| The glibc experiment | Linux x86-64, a build of glibc release/2.40 from source with -U_FORTIFY_SOURCE, the workload linked statically against it, and optionally Hermit (Installation) |
Native runs on macOS use the macOS pthread library, not glibc. They are informational only: they test the detector and the harness, not glibc.
Copy the example and the SDK out of your clone so that the build writes elsewhere. The Makefile finds the SDK at ../../sdks/c:
mkdir ~/condlock-try && cp -R examples sdks ~/condlock-try/ && cd ~/condlock-try/examples/glibc-condlockmake unitThis builds and runs five groups of checks (the rule, the config, the report, and the glibc state dump against two stand-in headers). We saw:
24 checks, 0 failed
24 checks, 0 failed
31 checks, 0 failed
12 checks, 0 failed
11 checks, 0 failed
That is 102 checks, matching the README's count. make unit works on Linux too, and it is the only target CI runs.
make condlock
./build/condlock -n 4 -r 50Standard output is one line (the report goes to standard error):
condlock libc=none hdr=none rounds= 200 signals= 200 blocked1= 117 maxb= 3 maxstag= 0 ticks= 0 spanned= 58
Exit status 0 (completed). libc=none hdr=none says the build is not against glibc, so the glibc-specific state dump is not compiled in. rounds is workers times rounds; blocked1 is how many releases saw a blocked waiter; spanned is how many waits spanned two releases (the situation the bug needs). The numbers vary from run to run.
The planted builds are separate binaries used only to test the detector; the counted binary contains no planted code. Build them and run one of each:
make selftest-bins
./build/planted/condlock-drop-k2 -n 4 -r 50 -tick-ms 20 -q 5 # from the 2nd release on, never signal
./build/planted/condlock-leak-k2 -n 4 -r 50 -tick-ms 20 -q 5 # a leaked lock
./build/planted/condlock-barrier-k2 -n 4 -r 50 -tick-ms 20 -q 5 # a skipped barrier broadcastand classify each with sh scripts/classify.sh <exit status> <stdout file> <stderr file>. We got:
| Planted build | Exit status | Class from classify.sh
|
|---|---|---|
condlock-drop-k2 |
3 | HIT subtype=Q+attributed prop_false=1 |
condlock-leak-k2 |
4 | STALL subtype=L prop_false=0 |
condlock-barrier-k2 |
4 | STALL subtype=B prop_false=0 |
That is the point of the self-tests: the injected lost signal makes the property false, while a leaked lock and a skipped barrier end as stalls with the property true. A protocol-level deadlock is never reported as a lost wakeup. These tests show the detector fires on an injected end state. They do not show it would fire on the real bug.
make testThis ran unit tests, the workload's behaviour tests, the classifier tests and the results-directory guard (41 and 15 checks in the last two groups) and finished in about 22 seconds, all passing. make test, scripts/run-selftests.sh and scripts/pack-results.sh are macOS-only: they use sw_vers, sysctl, shasum and BSD tar flags. On Linux, use make unit.
All of this is from the write-up and the example README; we did not repeat it.
-
Natively, on one machine (Ubuntu 24.04.5, kernel 6.17.0-42-generic, Intel Xeon E5-1410 v2, gcc 13.3.0), the property site was false at least once in 1,025 of 2,029 runs on the unmodified
release/2.40/masterstate and in 0 of 2,029 runs on the fixed state (the fixed state is compared at arm level only). - Under the pinned Hermit, two positive controls (the unmodified library plus one hunk that widens the race window; both assisted) did not make the property false in any of the runs made (90 runs including 30 informational, and 60). By the experiment's own stopping rule, the counted runs under Hermit were never started.
- So: nothing is claimed about whether Hermit reaches the bug's state, at any frequency.
The write-up lists what you can take from it instead: how to build and pin a glibc source state no distribution ships and show a program is linked against that glibc and nothing else; a workload and a stuck-state property designed not to fire falsely; planted faults that check the detector; per-seed determinism checks under Hermit; and the native observation above.
-
src/cl_rule.c(the rule, pure and unit-tested),src/cl_watchdog.c(the watchdog),src/cl_lock.c(the lock under test),src/cl_condraw.c(the glibc-only state dump). -
PINS(the C SDK commit this example is built against),BUILD-RECORDandBUILD-RECORD-fix1. -
experiment/README.md: every script, what it did, and what you must edit (EDIT:comments, the user nameubuntu, the Hermit path and its hash) before running anything. The scripts refuse to run on inputs they were not pinned to; that is intended.
Known limitations of the example, from its README: the watchdog ends the process with hs_exit while workers may still be running; the state dump reads glibc internals with plain loads, so it can be inconsistent if taken while the variable changes (it is a description, not part of the property); the worker count is capped at 16; and the Makefile does not rebuild the C SDK when its sources change, so run make clean first.
rm -rf ~/condlock-try. Nothing was written outside it. In your clone nothing was written.
docs/showcase-glibc-lost-wakeup.md and the example README. For how to read claims like these, see Limits-and-Honesty. Next: Examples-Overview.
Understand
- What-is-Hairspring
- Concepts-and-Vocabulary
- How-Deterministic-Simulation-Works
- Sweeps-and-Confirmation
- Limits-and-Honesty
Use
- Installation
- Your-First-Run
- Running-Your-Own-Software
- Guest-SDKs-by-Language
- Examples-Overview
- Hairspring-in-CI
- Operate-a-Server
- Upgrading-and-Versions
- Troubleshooting
Reference