test: ui-smoke phase 2, g-code execution and endpoint check

[grandixximo]

Phase 2 of the GUI test work tracked in #3756. Stacked on #3999 (now merged).

Summary

Extends each per-GUI ui-smoke test landing in #3999 with an end-to-end g-code execution stage:

• Estop reset, machine on, home all (via c.home(-1), respects HOME_SEQUENCE).
• MODE_AUTO, program_open on a shared _lib/smoke.ngc, auto(AUTO_RUN, 0).
• Poll linuxcnc.stat until interp_state == INTERP_IDLE and queue == 0 for 5 consecutive 10 ms polls (settle window guards the inter-line moment where interp briefly reports IDLE while queueing the next move).
• Assert stat.position[:3] delta against --expect-delta-mm 1,1,0, converted via stat.linear_units so the same arg works on inch (axis, touchy) and mm (gmoccapy, qtdragon) sims.

No new test directories. The four existing tests/ui-smoke/{axis,touchy,gmoccapy,qtdragon}/test.sh now also pass the --run-program and --expect-delta-mm flags through run-gui.sh. The Phase 1 connect-and-settle path stays as-is when those flags are omitted.

smoke.ngc

G21
G91
G0 X1 Y1
G90
M2

Forces mm input units (G21) and uses relative motion (G91) so the same file is sim-agnostic. The driver records stat.position[:3] after homing and checks (final - start) against the expected delta converted to machine units. This sidesteps each sim's HOME offset (axis homes to 0/0/0, qtdragon to 20/20/-10, etc).

State machine handling

linuxcnc.command.wait_complete() on a state or mode change only proves the NML message was acked, not that task_state / task_mode has transitioned. Polling the stat fields is the only deterministic signal. Two GUIs (gmoccapy reliably, qtdragon intermittently) re-issue their own mode commands during their own startup and revert task_mode AUTO -> MANUAL immediately after the driver sets it; without retry, auto(AUTO_RUN, 0) is then rejected and interp_state stays at IDLE.

ensure_state / ensure_mode helpers implement an issue + wait + stability-check + retry pattern, up to STATE_RETRY_BUDGET = 6 attempts. Intermediate timeouts use a quiet wait variant so spurious UI_SMOKE_FAIL lines do not pollute the log during retries (checkresult.sh greps for ^UI_SMOKE_FAIL on any line, so a retry that ultimately succeeds must not emit the marker).

qtdragon-specific bits

The qtdragon sim needs three CI-only workarounds layered in tests/ui-smoke/qtdragon/test.sh:

1. Writable config mirror. qtvcp's INI-driven LOG_FILE is rooted in the config dir. CI mounts the workspace read-only for the runtime user, so a relative LOG_FILE = qtdragon.log resolves to a path qtvcp cannot create and hal_bridge exits before the driver can attach. The test now mirrors configs/sim/qtdragon/qtdragon_xyz/ into mktemp -d and rewrites LOG_FILE to ~/qtdragon.log.
2. Offscreen Qt platform. qtvcp under xvfb + xcb on Ubuntu 24.04 segfaults during widget construction (no backtrace). Setting QT_QPA_PLATFORM=offscreen and LINUXCNC_OPENGL_PLATFORM=offscreen renders entirely in memory; xvfb-run still wraps the call so scripts/linuxcnc's X-display assumptions hold.
3. Block QtWebEngine import. qtdragon's UI embeds WebWidget (QWebEngineView, Chromium). Chromium's browser-process init segfaults inside the qtvcp PID under offscreen + xvfb on the Ubuntu runner even with --no-sandbox --single-process --disable-gpu (Chromium logs Sandboxing disabled by user. and then crashes). Rather than chase Chromium flags for a widget the smoke test never touches, the test drops a sitecustomize.py on PYTHONPATH that installs a sys.meta_path finder blocking qtpy.QtWebEngineWidgets (and PyQt5.QtWebEngineWidgets). lib/python/qtvcp/widgets/web_widget.py already has a fail-safe path that swaps the QWebEngineView for a plain QWidget when that import fails, so the UI loads cleanly with the Web tab inert.

debian/control

Adds python3-zmq under the <!nocheck> profile. Without it qtdragon's hal_bridge fails on startup and qtvcp tears down before the driver can attach.

Performance

ui-smoke total wall time on this machine:

	Tests	Wall	Marginal over Phase 1 baseline (~2 min)
Phase 1 only (#3999)	4	~2 min	baseline
Phase 1 + Phase 2 (this PR)	4	~2m43s	+~40s

Phase 2 reuses each test's existing linuxcnc startup and xvfb instance; CI cost is the Phase 2 sequence per GUI (estop + home + mode + run + verify, ~5 to 16 s depending on GUI), not the full ~30 s startup/shutdown overhead each test pays once.

Sequential by necessity: every ui-smoke test launches linuxcnc which claims a fixed set of SHM keys (SHM_KEYS in _lib/cleanup-runtime.sh), so parallel ui-smoke runs would race. Per-test SHM-key isolation would be a separate refactor.

Test plan

• 5 consecutive local scripts/runtests tests/ui-smoke runs, 4/4 pass, 0 shmem errors, clean tree
• Per-test diagnostics confirm gmoccapy hits the mode-revert path on every run and the retry recovers cleanly
• scripts/shellcheck.sh clean on changed scripts
• python3 -m py_compile clean on drive.py
• CI: rip-and-test, rip-and-test-clang, rip-rtai, and all 6 package-arch / 3 package-indep jobs green on run 26361844630

Out of scope

• Phase 3 (screenshot or video on failure, per test: UI smoke tests for axis, touchy, gmoccapy, qtdragon #3999 discussion) tracked on Add tests starting GUIs, likely falling back to xvfb for it #3756.
• Per-test SHM-key isolation for parallel ui-smoke runs, see Performance.

[grandixximo]

I will do more testing locally on an ubuntu 24.04 before marking as ready

[grandixximo]

CI will fail on this branch until a fix for the qtdragon ui-smoke dbus
segfault lands on master. Root cause is a half-initialized
dbus.SessionBus in qtvcp/lib/sys_notify.py whose mainloop-attached
QSocketNotifier keeps the C connection alive after init raises
ServiceUnknown; the next Qt event-loop tick dispatches into
_dbus_list_unlink and SEGVs. Captured under gdb, full backtrace in
#4064 which proposes the one-line fix. Idling this PR (no further
pushes) until #4064 or an equivalent fix is merged, then I will rebase.

[grandixximo]

Rebased, CI passes, phase 2 ready for review

[BsAtHome]

Interestingly, gmoccapy, qtdragon and touchy do not exit on SIGTERM. The output contains:

WARN: linuxcnc did not exit on SIGTERM, escalating to KILL

Wonder why that is.

Interesting warning:

[Gmoccapy][WARNING]  No virtual keyboard installed, we checked for <onboard>. Try 'sudo apt-get install onboard'. (gmoccapy:2311)

Should we try to have a virtual keyboard? Or is this just, well, don't care? Probably not usable anyway.

This one may be interesting:

[Gmoccapy][ERROR]  Found an error!
The following information may be useful in troubleshooting:
Traceback (most recent call last):
  File "/.../grandixximo/lib/python/gladevcp/hal_bar.py", line 232, in expose
    def expose(self, widget, event):
    
KeyboardInterrupt
 (gmoccapy:62)

This one is present in multiple:

/.../grandixximo/lib/python/gladevcp/gtk_action.py:312: SyntaxWarning: 'return' in a 'finally' block
  return npath

The touchy test shows this:

* * * * * * * * * * - - - - - WARNING - - - - - * * * * * * * * * *
*    CMS_DISPLAY_ASCII_UPDATER may not function properly due      *
*    to range limitations of some of the update() functions.      *
* * * * * * * * * * - - - - - WARNING - - - - - * * * * * * * * * *

It may indicate that the NML file is not found or a wrong one is used.

Qtdragon and touchy output is also riddled with warnings. And a deprecation warning in touchy:

(touchy:2765547): Gtk-WARNING **: 13:31:04.022: Theme parsing error: <data>:1:26: Junk at end of value for font-family
/.../grandixximo/bin/touchy:651: DeprecationWarning: Gtk.Widget.override_font is deprecated
  w.override_font(self.control_font)

[BsAtHome]

BTW, I also noticed that running touchy uses 100% CPU on one core. There is probably a busy-loop what should have a one millisecond sleep in it.

[grandixximo]

Thanks for the careful look. I built the PR head locally (uspace RIP, a26c5fb) and ran each sim headless under xvfb. None of these are regressions from this PR; they are pre-existing GUI/lib behaviors the end-to-end stage exercises.

1 + 3 are the same gmoccapy bug (verified). Teardown sends gmoccapy SIGTERM (strace confirms; no SIGINT is ever sent). gmoccapy's GLib/PyGObject loop catches SIGTERM and PyGObject's wakeup-fd bridge (gi/_ossighelper.py) surfaces it into Python as a KeyboardInterrupt, so the label is misleading and the reported frame (hal_bar.py:232) is just whatever was running when it landed. gmoccapy's error handler then swallows it, logs the traceback, and keeps running. Because the signal is swallowed, gmoccapy never exits on SIGTERM, which is why the harness escalates to KILL. One pre-existing issue, not introduced here.

On the harness side, the grace before KILL is 60s in launch.sh, so a GUI that swallows SIGTERM costs that full 60s. I can drop it to ~10s to save CI time, or keep 60s. Preference?

7 touchy CPU, measured, and touchy specific. The pegged core is the X server, not touchy's Python: py-spy shows touchy ~85% idle in the GTK main loop (~10% CPU), while Xvfb sits at 90-96%. axis/gmoccapy/qtdragon keep X near 0%. So it is not a busy-loop; touchy floods X with continuous redraws (likely the periodic set_label updates). Pre-existing; I will file it separately.

4 real bug, separate PR. lib/python/gladevcp/gtk_action.py:312 has return npath inside a finally:, which swallows exceptions. One-line fix.

5 CMS_DISPLAY_ASCII_UPDATER. Informational banner, printed because touchy opens an ASCII-format NML channel (touchy.ini sets no NML_FILE). Not a missing or wrong NML file; axis does not trigger it.

6 GTK warnings. Confirmed in touchy (override_font at bin/touchy:651, theme font-family parse error). GTK3 deprecation noise, separate cleanup.

2 no action. onboard virtual keyboard is not needed for a headless smoke test.

Plan: tune the SIGTERM grace here if you agree, and file 4, 6, and the gmoccapy-SIGTERM / touchy-CPU issues separately so this PR stays focused.

[BsAtHome]

Yes, they are all existing troubles in the GUIs :-)
Now that we test them, we should clean it up too.

What happens when sending SIGINT or SIGQUIT (or, fwiw, SIGHUP)? Lowering the grace period is like fixing the symptom and not the actual problem. (Maybe there should be input some X events to click on exit/quit?)
I think we need to consider our options and possibilities here to make the "right" fix, whatever that may be.

The NML problem needs to be looked into a bit more. That Ascii updater should never have been instantiated and I'd like to know why it was. The default NML files in configs/common correctly use xdr. Somehow none of those files were used.

[grandixximo]

I dug into the signal behavior. Measured each GUI in its real main loop (uspace RIP, sim configs under xvfb), sending each signal and checking whether the process exits:

Signal	gmoccapy	touchy	qtdragon
SIGINT	stays up	stays up	stays up
SIGQUIT	stays up	stays up	stays up
SIGHUP	exits	exits	exits
SIGTERM	stays up	stays up	stays up

So all three ignore INT, QUIT and TERM (absorbed by GLib/PyGObject for the GTK GUIs and by Qt for qtdragon), and only SIGHUP terminates them, because none of them install a SIGHUP handler so it hits the default action. That is why the harness has to escalate SIGTERM to SIGKILL.

On the right fix: I think the proper answer is that each GUI should honor SIGTERM and shut down gracefully (estop plus quit), the same way it would on a normal window close. SIGTERM is the standard "please terminate" signal, so the harness should not have to escalate to SIGKILL, lower the grace, or resort to a SIGHUP/X-event workaround; those treat the symptom. gmoccapy's SIGTERM hang traced to a concrete bug (PyGObject surfaces the signal as a KeyboardInterrupt, which reaches gmoccapy's sys.excepthook, which pops a modal dialog and blocks); PR #4076 makes it exit cleanly. touchy and qtdragon hang for their own toolkit reasons, and I will start working on the equivalent clean-shutdown handlers for each so they honor SIGTERM too. Once they do, the existing teardown works as-is with no long grace.

One question while I do that: is it worth adding ui-smoke coverage for the quit paths, so I do not regress them? Concretely, per GUI: (1) a clean quit via the GUI's own exit/quit, and (2) SIGTERM, asserting the process exits within a short window. I could extend it to SIGINT/SIGQUIT/SIGHUP later once we agree what "correct" is for each. Each extra quit case is roughly one more start-and-quit cycle (about 3 to 4 s per GUI in my measurements), so clean-quit plus SIGTERM across the four GUIs adds on the order of 30 s. Happy to wire that in if you think it is worth the coverage.

Separately, on the NML question: root-caused. The emcCommand buffer is correctly xdr. The CMS_DISPLAY_ASCII_UPDATER is a transient string-conversion updater created by NML::msg2str(), called from emctaskmain.cc only under EMC_DEBUG_TASK_ISSUE (0x10). touchy.ini was the only sim shipping DEBUG = 0x10, so task logged every command as an ASCII string and instantiated it. PR #4073 sets touchy DEBUG = 0; PR #4074 also gates the updater's unconditional stderr banner behind the CMS debug flag so it stays quiet for any legitimate transient use.

One more thing I hit while testing: gmoccapy pops a modal "Important change(s)" release-notes dialog at startup on a fresh profile, which blocks headless startup entirely (it spins a nested loop in the constructor). Filed as #4072.

[grandixximo]

Update on the SIGTERM right fix: clean-shutdown handlers done for the two remaining GUIs.

• touchy: SIGTERM handler quits the GTK loop via the idle queue, so atexit cleanup (child panels, saved prefs) still runs.
• qtdragon (qtvcp): quits the Qt loop on SIGTERM and SIGINT, leaving the X-button "are you sure?" confirmation intact for interactive use.

Both exit promptly now instead of being escalated to SIGKILL.

Caveat on touchy: PyGObject still prints one cosmetic KeyboardInterrupt line on exit. It comes from PyGObject's C-level wakeup-fd bridge, not Python, so it is unsuppressible from signal.signal, GLib.unix_signal_add, sys.excepthook or sys.unraisablehook (tried all). Only os._exit hides it, at the cost of skipping atexit cleanup. It is pre-existing, so no regression.

PRs to follow shortly.