-
-
Notifications
You must be signed in to change notification settings - Fork 0
Development Notes
Engineering lessons from building SynapseOS, kept because almost all of them were silent failures — something kept reporting success while doing nothing. Each entry ends with the generalisable rule.
This page is for people working on the system. If something is broken on a machine you are using, start at Troubleshooting.
The meta-rule: never trust a status command. Go and look at the running process.
| Question | Don't ask | Ask |
|---|---|---|
| Is my module the one I just built? | dkms status |
modinfo <mod> | grep vermagic vs uname -r
|
| Is this process using the library I built? | the config file | grep <lib> /proc/<pid>/maps |
| Is systemd running the unit I edited? | the file you edited | systemctl show -p FragmentPath <unit> |
| Is the daemon on the GPU? | the log line | grep -c nvidia /proc/$(pidof synapd)/maps |
| Did my child inherit a broken signal state? | assume it didn't |
/proc/<pid>/status → SigIgn, SigBlk
|
| Did my fix ship? | the commit |
pacman -Q <pkg>, then run the binary |
The most expensive class of wasted time in this project: fixing something,
committing it, bumping pkgrel, and testing against the old binary.
None of those three steps puts code on a machine. Only makepkg + pacman -U
does — and the source tarball has to be regenerated in between, because most
PKGBUILDs here consume a tarball rather than your working tree. Forget that and
makepkg packages the old code and exits 0.
pacman -Q synui # is the pkgrel you built the one installed?
pacman -Qo /usr/bin/synui-sound # does this file belong to that package?synui's PKGBUILD now refuses to build if the tarball is older than src/ —
run ./mktarball.sh first. And never rm -rf src/ inside a component directory;
see the src collision hazard.
Rule: the question is never "did I fix it", it is "is the fixed thing the thing that is running".
Symptom: after a kernel upgrade the module fails to load with ENOEXEC,
while dkms status insists it is installed.
Cause: three failures stacked. dkms.conf built in source/ — a symlink to
the shared /usr/src tree — so stale .o files were reused across kernels. The
rebuild hook trusted dkms status, which only checks that a .ko exists,
not that it matches the running kernel. And the status helper never compiled at
all, because its build step ended in || true.
Rule: dkms status is not a health check — compare vermagic to uname -r.
And never let a build step end in || true.
Symptom: you rebuild a library, install it, and the running program keeps the old behaviour. Nothing errors.
Cause: a .so that was once cp'd into /usr/lib by hand is owned by no
package, so pacman will never remove or replace it — and it silently wins in
the ld.so cache over the properly packaged one.
The tell: ldconfig complaining that /usr/lib/libfoo.so is not a symbolic
link.
Rule: check /proc/<pid>/maps for what the process actually mapped.
pacman -Qo on a library should always name an owner; "No package owns" is the
bug.
There is no pacman hook running
ldconfigfor arbitrary paths — packaged libraries belong in/usr/lib, not somewhere under/optthat nothing searches.
Symptom: you ship hardened unit files in a package, install it, and none of the hardening takes effect.
Cause: a stale copy in /etc/systemd/system/ overrides the packaged unit.
systemd is doing exactly what it documents; the surprise is that you forgot the
/etc copy existed.
Rule: systemctl show -p FragmentPath <unit> tells you which file is in
force. Never trust the file you just edited.
-
There are no inline comments. A trailing
# commenton a directive parses into the value. The directive is not commented out — it is silently wrong, with no warning. -
Never order a service
Before=a.socketit depends on. That is an ordering cycle, and systemd resolves a cycle by dropping a job — so something you asked for simply never runs. -
Hardening can silently break the thing being hardened.
ProtectControlGroupsdefeats cgroup-based containment while leaving the service looking healthy. -
Dependency lists are additive-only. An empty
Requires=in a drop-in does not reset the list, whatever the empty-string reset does forExecStart=. A drop-in written to remove a dependency reads as correct and changes nothing — confirmed across bothdaemon-reloadanddaemon-reexec. Removing one means editing the unit fragment itself.
Stopping a daemon does not stop what can start it. If anything still listening
Requires= that daemon, the next connection restarts it — and if the listener is
reachable from the network, the trigger arrives from another machine, where
nothing local shows why.
Symptom: you stop something, it is down, and seconds later it is running again
with the journal showing only Started ….
- Check the whole reverse dependency set, not just the unit:
systemctl list-dependencies --reverse <unit>, andsystemctl show <unit> -p RequiredBy -p WantedBy -p TriggeredBy. - A proxy in front of a socket-activated Unix socket does not need
Requires=on the backend. The socket activates the backend by itself, so the dependency adds nothing except the ability to resurrect it — while with the backend's socket stopped, the connection simply fails, which is what "stopped" is supposed to mean. - "Off" switches must close every listener, or they undo themselves.
SIG_IGN and the blocked signal mask survive exec(). A child inherits both
from whoever spawned it and has no idea.
-
Xwayland aborted itself. The compositor had
SIGCHLDset toSIG_IGN; that leaked into Xwayland, whosewait4()then never reaped. -
An idle-inhibit died quietly for 21 hours. systemd's
IgnoreSIGPIPE=yes(the default) meant aSIGPIPEthat should have killed a dead pipe write was ignored instead. -
Every app the compositor spawned was immune to
SIGTERM, because thesignalfdblocked mask was inherited by every child.
Rule: reset all signal dispositions and unblock the full mask in the child,
between fork() and exec().
grep -E 'SigIgn|SigBlk' /proc/<pid>/statusA swallowed non-zero status is how a hard failure becomes a silent one. Two
instances: a text-to-speech path ignored aplay's status, so a broken audio
route was a silent mute rather than an error; and a build step suffixed
|| true never compiled at all.
When the sound-theme picker learned that a directory needs index.theme,
stereo/ or mono/ to be a theme, filtering alone would have left anyone
already sitting on an invalid selection silent and none the wiser.
So the invalid value is also named, in the CLI and in the panel:
theme alsa <-- NOT a sound theme, nothing will play
Rule: a filter fixes new choices and abandons existing ones. Report the state you are now rejecting.
A state file had volume as both the master level and an event name. Reading the
level picked up the event's line (off) and silenced everything.
The fix renamed the event, and the per-event sample override is keyed
<event>_sound and matched by building the exact key, never by prefix — an
event added later whose name extends another's would otherwise steal its line.
handle_keybinding takes the first match, so a second bind on a combo is not
something anyone notices — it is the older feature silently going dead. The
compositor now logs DUPLICATE default bind at startup, because one such
collision only turned up when the table happened to get grepped.
Several things exist in more than one place and drift silently. The full list is in Building and Packaging; the worst offender is worth repeating:
create_source_tarball existed in two collectors, and every fix to it landed
in one and not the other — four times out of four. Both now delegate to the
component's own <pkg>/mktarball.sh. When a package needs particular files,
write it a mktarball.sh rather than teaching two collectors about it.
build-all.sh keeps two lists: KNOWN=, which decides whether an argument
is legal, and the sequence of build_component / build_script_pkg calls,
which decides what is actually built. A component was added to the first and
never to the second:
./build-all.sh <component> -> passes the KNOWN= argument check
-> matches no build rule
-> builds nothing, exits 0
syn-update reads exit status, so it reported success, published nothing, and
offered the identical update again on the next run — for two releases, which
also meant the fix that component carried could never install. What reached the
user was "this update keeps coming back after I apply it and log back in", a
symptom that points at the updater, the session, the package cache — everything
except the build script.
The rule: when one list validates input and another list acts on it, something
must assert that everything accepted was also handled. want() now records
every name a build rule asks about, and the run exits non-zero with no build rule for: <name> if anything named on the command line was never dispatched.
Screen capture is damage-driven — a frame only when the screen changes — so the
file is variable-rate and declares r_frame_rate as the 90 kHz timebase
(90000/1), not any rate it runs at. Three unrelated-looking symptoms came out
of that single field:
- the capture stamped itself H.264 level 6.2, a level meant for 8K, because the encoder read 90000 as the frame rate;
- every video editor conformed the clip on import, since no timeline runs at the ~67 fps such a capture averages; and
- conforming dropped frames — a 313-frame capture reached DNxHR as 226, with nothing said about the 87 that went missing.
On a variable-rate file avg_frame_rate is the honest field and r_frame_rate
is not; a converter reading the wrong one asks for a 90000 fps output.
The rule: fix a variable rate at the source, not downstream. Capturing at a constant rate costs nothing — on a high-refresh screen it is the smaller file, because damage capture emits frames faster than 60 whenever anything moves — and it makes every later conversion a clean rewrap instead of a lossy conform. One flag at the source; every downstream tool inherits it.
Anything the compositor prints — and anything it spawns that fails — lands on
tty1, not in journalctl. A GUI component failing with "no message" almost
always means the message went to a stream you aren't reading.
Two that cost time this way: a QML shell missing pragma UseQApplication (which
kills menus and the system tray entirely), and Nerd Font glyphs silently reducing
to the empty string, which made an icon-only bar module collapse to zero width
and vanish. The glyph fix was to store every icon as a \u escape and verify the
codepoints mechanically — fc-query --format %{charset} against the font —
rather than by eye.
A helper daemon can hold every resource it needs, report healthy, and paint
nothing. The wallpaper renderer did exactly that after a suspend: still running,
0% CPU, blocked in poll() forever, because it had lost its Wayland surfaces
and had no code path that rebuilds one. Its output-removal handler was a literal
// todo, and the only caller of its surface setup ran once at init.
Two lessons, both general:
-
When a client cannot recover its own state, restart it — from the event you
actually have. synui re-runs the control script on logind's
PrepareForSleep(false)and on an output arriving after one was lost, coalesced into one delayed timer. Both triggers are armed because it was never proven which one fires; a delay is also required, since the restore path drops any saved entry naming an output the compositor can't currently see, and firing mid-reprobe would persist a partial layout. -
ps %cpuis a lifetime average. It read 20% for a process that had done nothing for hours. Sample/proc/PID/statfields 14+15 over a few seconds, ortop -b -n2. Whenstraceis blocked bykernel.yama.ptrace_scope,/proc/PID/task/*/wchanstill works unprivileged and names the exact sleep.
And when the evidence lives in a log the fix truncates on restart (>"$LOG"),
copy it before running the fix. That one was lost.
The monocle layout hides every window on a monitor except the focused one. It
did that correctly — and there was still no way to change which window you were
looking at, because the code that hid windows only ever ran from the layout
pass, and nothing on the focus path called the layout pass. Focus moved; the
scene nodes did not. Three windows, after one Alt+Tab:
pid A focused=false enabled=true <- still the one on screen
pid B focused=true enabled=false <- has the keyboard, invisible
pid C focused=false enabled=false
You were typing into a window you could not see.
Why it survived so long: almost anything you try next fixes it. Opening a
window, closing one, switching desktop, Super+F, retile and a config reload
all re-run the layout, so the desktop rights itself the moment you fidget. A
bug you cannot reproduce twice in a row reads as "I must have imagined that".
The rules:
-
When one thing is derived from another, derive it at the point the source
changes. Visibility was derived from focus, but only recomputed at the
layout pass. Putting the recompute at the focus choke point — the single
function every focus path already goes through — fixes every route in at
once: both cycles, the dock, a click. Fixing it inside
Alt+Tabwould have leftSuper+Jbroken. - Then guard the write. That choke point is now hot, and the layout it runs hands every window a size. Compare before configuring, or a reflow-per-focus becomes a configure sent to every client on every keypress. The window boxes under monocle never change, so the compare rejects all of them.
- A comment claiming a behaviour is not a test of it. The file's own header had said "cycle with Alt+Tab" since the layout was written. Nothing had ever checked it. The test that does now asserts the invariant — exactly one window enabled per output, and it is the focused one — re-checked after each way of moving focus, rather than asserting one window ends up on screen.
Diagnostic that found it: synctl clients reports enabled alongside
focused precisely to answer "why is this window invisible when every other
field says it is fine". Two fields that should agree, printed side by side, is
what turned a vague "monocle feels broken" into a one-line fix.
A control script serialised itself with flock -w 30 on fd 9 and then launched a
daemon, which inherited that fd. The lock therefore lived as long as the
daemon did: every later invocation sat out the full 30-second timeout, printed
its "continuing anyway" warning, and then ran unserialised — the exact race the
lock existed to prevent, now with a 30-second stall attached.
Close the lock fd explicitly in anything long-lived you spawn (9>&-).
Short-lived children that finish while you still hold the lock are harmless.
A list view and an icon view of the same data are usually two separate delegates, and a feature added to one of them is simply absent from the other. Nothing warns: the menu entry still exists, the shortcut still fires, the state it sets is still set — and nothing is drawn to act on it, so the control reads as a dead button.
This has now happened twice in the same file, to two different features:
- drag-and-drop, wired into the list delegate only, so dragging did nothing at all in icon view — which is the view anybody with thumbnails turned on is looking at;
- inline rename, the same omission, reached from both the right-click menu and
F2, both of which appeared to do nothing.
The rule: when two delegates render the same row, treat them as one feature with two renderings. Anything added to one is unfinished until it is in the other, and the test that proves it has to be run in each view — a check that runs in whichever view is the default is a check that will pass while half the application is broken.
The same shape appears one level down, in input handling: a full-size
MouseArea declared after the controls it covers swallows their clicks,
because Qt Quick delivers a press to the last matching child first. An editor,
a small button or an eject glyph that "does nothing" is usually this. Give the
control a z above the row, or declare the row-wide handler first.
QT_QPA_PLATFORM=offscreen quickshell -p <file> proves a QML file loads
("Configuration Loaded" versus "Failed to load configuration"), which catches
most breakage for free. It also does more than that: append a Timer to a
copy of the file in a scratch directory, set the state you want to test, and
print what became visible. That is a real functional test on a machine with no
spare seat.
Two gotchas cost time:
- A window's visual tree hangs off
contentItem. Walkingchildrenfrom the window object itself finds nothing, which looks exactly like "the item isn't there". - Run the probe against the shipped file as well as the fixed one. "The editor is visible now" is only evidence when paired with "it was not visible before" — otherwise a probe that never worked reports success.
Reading a binary format means trusting offsets somebody wrote down. One of them was wrong by eight bytes — a video track header's dimensions were read from the wrong field and from four bytes of the next structure entirely.
What made that a five-minute bug instead of a shipped one is that the parser validates before it believes: the garbage failed a plausibility check, so the feature reported nothing rather than a number that was merely odd. An offset that had landed a few bytes earlier, still inside the structure, would have produced a confident wrong answer that nobody would have questioned.
Three rules for anything that parses a file it did not write:
- Validate the answer, not just the read. A dimension of four billion is not a dimension. Refusing it is what makes a mistake visible.
- A short read is normal. Requiring a fixed-size header block up front broke a ten-byte file that legitimately carries everything it needs in ten bytes. Read what is there and check the length before each comparison.
- Decide the format by magic bytes, never by the extension. It is not politeness towards oddly-named files: it is what stops a text file with an image's extension being parsed at all.
wlroots/scenefx-specific, and specific enough that they cost real days each. They lived on the synui page until that page was cut back to what a user of the compositor needs; this is where the engineering half went.
Signal inheritance is the other one that bit the compositor hardest, and it is its own section above — it is not compositor-specific.
Xwayland is itself a client of that event loop, and it starts lazily. A blocking xcb round-trip made from the event loop deadlocked the whole compositor at login — black screen, dead input, no core dump. Use a ready-flag plus a worker thread.
Or they swallow clicks. (Found via cat mode, of all things.) The QML equivalent
is mask: Region {} on a desktop widget — without it the widget is an invisible
rectangle that eats every click over it.
Two click sites passed the raw (sub)surface to focus_view, sending
wl_keyboard.enter to a render subsurface. Firefox therefore never made its
persistent-storage doorhanger the active modal: the panel was unresponsive and
Tab skipped it. A doorhanger is a subsurface, not a popup — none of the
popup/grab machinery applies to it.
-
Don't call
focus_viewon popup clicks. It disrupts thexdg_popupgrab, which broke menu activation in Firefox. -
Unconstrain nested popups too.
parent_viewis only set for toplevel-parented popups, so nested ones were never unconstrained and every submenu got its full requested size — an application list ran off-screen instead of scrolling. -
Pointer motion must be delivered during an implicit grab. Without
wl_pointer.motionwhile a button is held, region-select (slurp) and every client-side drag are broken.
Anything on the unmap or decoration path that reaches for xdg_surface
unconditionally will SEGV the moment an X11 window is involved.
And when you need to crop something, crop the buffer, not the tree —
wlr_scene_subsurface_tree_set_clip() recurses into every subsurface tree below
the node it is given, and xdg popups are parented beside the client tree, so
clipping the parent crops the menus.
The bar reserved its strip while auto-hidden and lost the reservation when it should have had it, because the zone was set too early. The same ordering is why desktop-icon layout pass 1 saw the whole output: the bar reserves its strip after startup, so anything laid out at startup must re-run once the zone lands.
Committing whole-output damage after the state is built shows the previous frame's buffer for one frame — the "flash on focus change" bug.
A blur node that keeps its old size after a resize is the "shadow ghost" people report as a rendering artifact. It is a scenefx blur node that was never resized.
A decorated window is two stacked buffers whose corner radii encode the
titlebar/content seam. Anything built by growing those per-buffer boxes
inherits the seam and comes out square where it should be round. Frame-level
effects belong on view->frame, not on the buffers. See
Window Effects.
The live /usr/local/bin/synui-session and two blocks in syn-install.sh.
Change one, change all three — the same shape as
two copies of everything, one copy worse.
XCURSOR_SIZE is the one that shows: unset, libXcursor sizes the cursor from
the virtual screen width, which on a multi-monitor desk is thousands of pixels
and gives Xwayland clients an enormous pointer.
The nested-headless rig is how most rendering work gets verified. Four rules that are not optional:
-
scenefx's
fx_rendererignoresWLR_RENDERER— it is GLES2-only.WLR_RENDERER_FORCE_SOFTWARE=1plusWLR_RENDERER_ALLOW_SOFTWARE=1is the only headless path, and the only onegrimcan capture from. -
Stub
synui-apply-themeonPATHfirst. It hardcodes$HOME, so a nested session with a scratchXDG_CONFIG_HOMEwill still re-theme the live desktop.gsettingsis worse: it reaches dconf over the session bus, so even a faked$HOMEdoes not contain it. Stubgsettingsandkwriteconfig5/6too. -
Confirm which compositor you are talking to. A nested instance takes the
next free Wayland name, and
SYNUI_SOCKETbeatsWAYLAND_DISPLAY— check thatsynctl outputsreports the headless output before trusting a result, or a dispatched action opens a panel on the live desktop. -
Drive it with
wtype, never uinput. uinput events go to the real seat.
wtype gives a false negative against Qt/quickshell clients — they need a
real keypress, so zero key events in a headless wtype test proves nothing about
a QML panel.
-
pkill -f <pattern>kills the shell that ran it when the pattern matches its own argv (exit 144), and the rest of a compound command silently never runs. That is how acprestoring a config got skipped, making a working config knob read as broken. Kill by PID. -
SIGHUPis not a clean A/B. Reloading config in place leaves other state moving between screenshots, so a whole-image pixel diff is worthless. Restart per arm and compare only the region you care about. -
A corrupt tail passes a prefix check. Verifying a USB write by comparing
the first N MB proves nothing about the rest —
cmp -nthe full length, or compare the written medium's own package list. -
Unmounting a loop device is not detaching it. Set
Loop.Autoclearor detach explicitly.
EOF is body text. << EOF runs on to the next line that is exactly the
terminator at column 0 — and only <<- strips indentation, and only tabs, never
spaces. In an installer this cost 70 lines: an indented terminator swallowed a
bootloader install, an entire elif arm and a mkdir, so choosing one
bootloader silently installed a different one.
bash -n passes. The file still parses; it just isn't the program anyone
wrote, and the block looks correctly indented in an editor. Two checks worth
having in any repo with heredocs:
# 1. no terminator may be indented
grep -oE '<<[ \t]*"?[A-Za-z_][A-Za-z0-9_]*"?' script.sh | sed 's/<<[ \t]*//; s/"//g' \
| sort -u | while read -r t; do grep -nE "^[[:space:]]+$t\$" script.sh; done
# 2. strip every heredoc BODY, then assert the commands that must run are still codeThe second is the one that catches the consequence: after stripping bodies, the commands you rely on should still be there. Ours had gone to zero occurrences while the file still looked right.
A long non-interactive step before a prompt — a package install, a build, a
download — is a chance for the user to press a key. That keypress sits in the
terminal's input buffer until the next read consumes it, so the menu is drawn
and answered in the same frame and the program reports a choice nobody made.
Everything typed after it answers the following prompts, in order.
Fix it in the one helper every prompt prints through, not at the prompt where you noticed it:
prompt() {
if [ -t 0 ]; then # only when stdin is a terminal —
local _junk # a piped run's answers are real
read -r -t 0.1 -N 4096 _junk 2>/dev/null || true
fi
printf ' %s ' "$1"
}All three paths are testable without a disk: printf '3\n' | script -qec …
proves queued input is dropped, { sleep 0.6; printf '3\n'; } | script -qec …
proves input typed after the prompt still reads, and a plain pipe proves
automation survives.
strip | grep -q pattern exits 141 when grep -q matches early: it stops
reading, the producer takes SIGPIPE, and pipefail propagates that. Any test
written as "this string must be ABSENT" then passes whether the string is there
or not — a whole class of checks that silently assert nothing.
Match from a here-string instead: grep -qF -- "$1" <<<"$text". It also hides
well: an interactive shell aliased to a grep implementation that does not exit
early will show the check working, while the suite runs the one that does.
Rule: prove a negative check by making it fail. Inject the forbidden string, confirm exactly the expected check goes red, then restore and diff.
Config-driven generators tend to fail open and silent: a typo produces zero units, no error, and a machine that behaves exactly like one that was never configured. Shipping the package without the config file does the same thing.
So verify the output, not the input — run the generator and assert the unit
exists. Most have a test mode that makes this cheap (ZRAM_GENERATOR_ROOT for
zram-generator), and it works against a target tree during an install.
See also: Building and Packaging, Cutting an ISO Release, synui, Troubleshooting.
Using it
- Installation
- Updating
- Software
- Files
- Keybindings
- Commands
- Nix
- Gaming
- DaVinci Resolve
- Secure Boot
- Troubleshooting
Customising it
Components
Apps
Hacking on it