prototypes desktop issues

Prototypes Desktop Issues — GTK3/4, Windowing, Overlays

Scope: lenzu-prototypes workspace investigations into GTK3→4 migration, transparent overlay windows, X11 pointer tracking, and related desktop issues.

Counterpart: lenzu-desktop-issues.md (main repo perspective)

1. GTK3→GTK4 Migration — Complete

This section has been consolidated into GTK-Migrations.md. All workaround tables, official guide cross-references, and migration architectural decisions now live there. This section remains as a stub index.

Migration topic	Location
Full workaround table (22 rows)	`GTK-Migrations.md` § Workaround Table
Official guide cross-reference (14 rows)	`GTK-Migrations.md` § Guide Cross-Reference
Key combo architecture (Invariant 5)	`GTK-Migrations.md` § Key Combo Architecture
Summary: GTK3 vs GTK4 per invariant	`GTK-Migrations.md` § Summary Table
Architectural decisions timeline	`GTK-Migrations.md` § Architectural Decisions
Workspace dependency graph	`GTK-Migrations.md` § Workspace Dependency Graph
X11 window management pattern	See the workaround table entries (all use x11rb)

2. Pitfalls & Lessons Learned (Post-Migration Runtime Issues)

Even after the code compiles cleanly, several runtime issues were discovered that required additional fixes:

2a. Window XID Discovery: `X11Surface::xid()` vs PID Scanning

Wrong approach: Scan root window children matching _NET_WM_PID:

// DON'T — fragile, fails in practice
let tree = conn.query_tree(root).ok()?.reply().ok()?;
for &child in &tree.children {
    let pid = get_property(child, "_NET_WM_PID");
    if pid == std::process::id() { /* found it */ }
}

Why it fails:

The GTK4 window may not be mapped yet when the first poll fires.
Window managers (especially xfwm4, mutter) reparent the client window under a frame window that is NOT a direct child of root — the scan never finds it.
_NET_WM_PID is not always set immediately after present().

Correct approach: Use GDK4's own X11 surface, always available after present():

// DO — works every time on X11
let surface = window.surface()?;
let x11_surface = surface.downcast::<gdk4_x11::X11Surface>().ok()?;
let xid = x11_surface.xid() as u32;   // u64 → u32 for x11rb

Must call AFTER window.present() — window.surface() returns None if the window is not yet realized.

2b. x11rb Connection Init Must Not Panic

Wrong approach:

let conn = ONCE.get_or_init(|| RustConnection::connect(None).ok().unwrap());
//                                                       ^^^^^^^^ PANICS

If $DISPLAY is unset or the X server is unreachable, the .unwrap() kills the entire app. Always handle X11 connection failure gracefully:

fn init_x11() -> bool {
    if ONCE.get().is_some() { return true; }
    if let Ok((conn, _)) = RustConnection::connect(None) {
        let _ = ONCE.set(conn);
        true
    } else {
        false
    }
}

All cursor-tracking and window-management functions should be no-ops when X11 is unavailable (the app remains functional as a static overlay).

2c. `set_window_state` (keep-above) Must Follow `present()`

window.present();                    // must be first
set_window_state(&window);           // window.surface() is now Some
input_shape_clickthrough(&window);   // same

Calling set_window_state() before present() silently fails because window.surface() returns None.

2d. `surface.set_input_region()` Takes `Option<&Region>`

// Correct:
let region = cairo::Region::create();
surface.set_input_region(Some(&region));   // takes Option<&Region>

Not &Region. The Option wrapper was silently wrong in earlier versions.

2e. `RefCell` Borrow Must Be Dropped Before Blocking Calls

Wrong approach — holds mutable borrow across blocking sequence:

// DON'T — RefCell panic when draw/animation callbacks fire during iteration
let mut s = state.borrow_mut();
s.is_loading = true;
window.set_visible(false);
while glib::MainContext::default().iteration(false) {}  // ← timer/draw callbacks
std::thread::sleep(Duration::from_millis(400));          //   try to borrow state
s.pixels = Some(...);

Holding a RefCell mutable guard across glib::MainContext::iteration(false) or std::thread::sleep() causes a panic when other callbacks (draw function, animation timer) try to borrow()/borrow_mut() the same RefCell.

Correct approach — scope-guard the mutable borrow:

// DO — drop the mutable guard before the blocking sequence
{
    let mut s = state.borrow_mut();
    s.is_loading = true;
    s.status = "CAPTURING...";
}
window.set_visible(false);
while glib::MainContext::default().iteration(false) {}
std::thread::sleep(Duration::from_millis(400));
// re-borrow after blocking
let mut s = state.borrow_mut();
s.pixels = Some(...);

Check also with try_borrow() / try_borrow_mut() for the initial condition check rather than calling borrow_mut() directly, to avoid panicking if another borrow is active.

2f. `Session`/Model Loading Must Be Optional

The ONNX detection prototype (x11-gtk-lens-test) panicked on startup with .expect("No valid .onnx file found (>1MB)!") when no model file was in cwd. Always make model loading graceful:

// DON'T — panics when no .onnx file exists
let model = Session::builder().unwrap()
    .commit_from_file(&model_path).unwrap();

// DO — model is Option<Session>
let model = fs::read_dir(".")
    .ok()?
    .find(|p| p.extension() == Some("onnx"))
    .and_then(|p| Session::builder().ok()?.commit_from_file(&p).ok());

All inference and detection code should check if let Some(ref mut m) = model before attempting to run the model.

2g. Transparent Lens: Draw Must Clear with RGBA(0,0,0,0) + Operator::Source

Wrong approach — paints opaque black over the entire lens area:

// DON'T — opaque black fills the window, hides transparent CSS background
cr.set_source_rgb(0.0, 0.0, 0.0);
cr.paint().ok();

The CSS window { background: transparent; } alone is insufficient — the DrawingArea's draw_func paints over it with opaque black, blocking any transparency.

Correct approach — clear with fully transparent colour using Source operator:

// DO — transparent clear, then switch back to Over for overlays
cr.set_source_rgba(0.0, 0.0, 0.0, 0.0);
cr.set_operator(cairo::Operator::Source);
cr.paint().ok();
cr.set_operator(cairo::Operator::Over);

The Operator::Source replaces the existing pixels entirely (including the window's transparent background) with the source colour. Switching back to Operator::Over after the clear ensures subsequent drawing operations composite correctly.

This pattern is used in jp_ocr_app and was applied to x11-gtk-lens-test in the same fix.

2h. SCIM Stderr Noise — GTK4 GDK Auto-Launches Broken SCIM

Both jp_ocr_app and x11-gtk-lens-test print to stderr on every launch:

Loading socket Config module ...
Creating backend ...
Loading x11 FrontEnd module ...
Failed to load x11 FrontEnd module.

Source: These messages come from SCIM (Smart Common Input Method), a CJK input method framework. GTK4's GDK X11 backend auto-detects SCIM and forks a scim-launcher -d -c socket -e socket -f x11 --no-stay -d child process.

The launch sequence (observed via strace):

GTK4 GDK X11 calls scim -h to check availability.
scim-im-agent is spawned.
It launches scim-launcher -f x11, which fails.

Why it fails: The X11 frontend module at /usr/lib/x86_64-linux-gnu/scim-1.0/1.4.0/FrontEnd/x11.so loads (all shared library deps resolve) but internal initialization fails. SCIM also looks for /etc/scim/global and ~/.scim/global config files, which don't exist in stock installs.

Why GTK_IM_MODULE= doesn't help: The SCIM launch is built into GDK's X11 backend source code, independent of GTK's IM module system. Neither GTK_IM_MODULE=ibus nor GTK_IM_MODULE= empty suppresses it. An already- running SCIM daemon (launched at session login via scim-launcher -c simple -e all -f socket) coexists but doesn't affect the GTK4 child process.

Impact: Cosmetic only — the messages go to stderr from the forked child process. Our GTK app runs unaffected. No functionality is degraded.

To suppress:

Uninstall SCIM (sudo apt remove scim), or
Redirect stderr (cargo run -p jp_ocr_app 2>/dev/null), or
Create /etc/scim/global or ~/.scim/global config (SCIM repeatedly looks for these; missing config may trigger the init failure).

3. Workspace Dependency Graph (Post-Migration)

lenzu-prototypes workspace (all GTK4)
├── GTK4 0.11 members: jp_ocr_app, x11-gtk-lens-test, gtk_gdk_test, gtk4_dialogbox_test
│   └── gtk4 0.11.3 → glib 0.22.7, gdk-pixbuf 0.22.0, pango 0.22.6, pangocairo 0.22.0
│   └── gdk4-x11 0.11.0 (for XID queries)
│
├── No-GTK members: winit-test, dbnet-test, manga-ocr-test, sarashina-onnx-test, etc.
└── libraronity (workspace member in other repo)

All members now use the same glib/gdk-pixbuf/pango generation (0.22.x), eliminating the unsound dual-glib state.

Remaining workspace issues

gtk3 0.18.2 (the final GTK3 release, marked UNMAINTAINED) is no longer a dependency in any workspace member, but Cargo.lock may retain it if any transitive dep references it.
gdk4-wayland and gdk4-win32 are deps of gtk_gdk_test only (for platform testing), pinned to 0.11.x matching the gtk4 generation.

4. Dependabot PR Situation (as of 2026-06-20)

5 open PRs, all dependabot bumps, all failing review / review check:

#	Crate	Bump	Files	Status
#35	`tokenizers`	0.20.4 → 0.23.1	`Cargo.toml`+`Cargo.lock`	Failing review check
#34	`gdk4-win32`	0.8.2 → 0.11.0	`Cargo.lock` only	Failing review check
#33	`gdk-pixbuf`	0.19.8 → 0.22.0	`Cargo.lock` only	Failing review check
#32	`gtk4`	0.8.2 → 0.11.3	`Cargo.lock` only	Failing review check
#31	`pangocairo`	0.18.0 → 0.22.0	`Cargo.lock` only	Failing review check

All fail on a Gemini AI review step (infrastructure timeout), not code issues. PRs #31–#34 bump GTK4-rs crates and are tightly coupled — they must land together or not at all. PR #35 (tokenizers) is independent.

Note: PRs #31–#34 are now effectively superseded — the workspace already uses gtk4 0.11.x / glib 0.22.x ecosystem. These PRs only affect Cargo.lock.

5. Decision Log

Date	Decision	Rationale
Pre-2026-04	GTK4 evaluated and rejected — moved to `GTK-Migrations.md`	graphene/gobject dep complexity, API churn, build failures
2026-04	Electron for HUD (not GTK)	X11 transparency works, Web UI flexibility
2026-06-20	GTK4 migration issues — moved to `GTK-Migrations.md`	All workaround tables, cross-references, and architectural decisions consolidated there
2026-06-20	Shared docs convention established	`lenzu/docs/prototypes-desktop-issues.md` (from here) + `lenzu/docs/lenzu-desktop-issues.md` (from main repo)
2026-06-21	PID scanning for XID abandoned	`_NET_WM_PID` on root window children is unreliable (WM reparenting, unmapped windows, timing). Use `gdk4_x11::X11Surface::xid()` after `present()` instead.
2026-06-21	x11rb connection must init gracefully	`get_or_init(
2026-06-21	`set_window_state`/`surface()` requires realization	`window.surface()` returns `None` before `window.present()`. All surface/XID access must happen after.
2026-06-21	Unit tests added for known runtime panics	9 tests across `jp_ocr_app` (4) and `x11-gtk-lens-test` (5) catch RefCell borrow-across-blocking panics, Pixbuf creation, model-loading graceful failure, and try_borrow pattern — now verifiable in `cargo test` without X11 display.
2026-06-21	`x11-gtk-lens-test` transparent draw fix	Draw function used `cr.set_source_rgb(0,0,0)` → `paint()`, filling the lens with opaque black and hiding the transparent CSS window background. Changed to `rgba(0,0,0,0)` + `Operator::Source` + switch back to `Over` (matching `jp_ocr_app` pattern).
2026-06-21	SCIM stderr noise identified	Both lens apps print SCIM init errors on every launch. Traced to GTK4 GDK X11 backend auto-forking `scim-launcher -f x11`. Cosmetic only — app unaffected. `GTK_IM_MODULE` has no effect; suppress via stderr redirect or `apt remove scim`. Documented in §2h.
2026-06-21	Trunk GTK3 dual-glib fix + CI	`gdk-pixbuf = \"0.19\"` in GTK3 members conflicted with `gdk 0.18`'s internal `gdk-pixbuf 0.18.5`. Pinned to `0.18`. Added GitHub Actions CI (`cargo test --workspace`). Removed all Gemini AI workflows.
2026-06-21	jp_ocr_app spinner lollipop tail fixed	Cairo `arc()` draws an implicit line from the current path point to the arc start. `show_layout()` leaves the current point at the last glyph, so the arc was connected to it — producing a straight "lollipop tail". Fixed by inserting `cr.new_sub_path()` before `cr.arc()`. See §2i.
2026-06-21	jp_ocr_app white window + frozen spinner during capture fixed	`flash_alpha=1.0` painted the entire lens area with an opaque white rectangle on every capture, causing the "white window". Additionally, `std::thread::sleep(400ms)` blocked the GTK main loop, preventing the animation timer from firing and keeping the spinner frozen (static). Fixed by removing `flash_alpha` entirely and replacing the blocking sleep with `glib::timeout_add_local(400ms)`. See §2j.

2i. Spinner Lollipop Tail — jp_ocr_app (FIXED 2026-06-21)

Symptom: The loading spinner in the UI panel appeared distorted — a straight line extended from the spinner arc to an off-center point, like a lollipop stick.

Root cause: Cairo's cr.arc() does NOT start a fresh path. If a current point exists in the path, arc() implicitly draws a straight line_to() from that point to the arc's start position before drawing the arc itself.

The draw function calls pangocairo::functions::show_layout() to render the status text (e.g. "CAPTURING...") just before drawing the spinner. PangoCairo's show_layout() leaves the cairo current point at the last glyph position in the layout. The subsequent cr.arc() then drew a line from that glyph position back to the arc's start, creating the visible tail artifact.

Fix:

cr.new_sub_path();   // ← break the implicit line-to
cr.arc(0.0, 0.0, 8.0, 0.0, 1.5 * std::f64::consts::PI);

cr.new_sub_path() starts a new sub-path without moving the current point, so Cairo has no start point to draw a line from. The same fix applies to any arc() call that follows text rendering or other drawing that leaves an open path.

Commits: be26043 (jp_ocr_app); same pattern fixed in lenzu at 75f7dbf.

2j. White Window and Frozen Spinner During Capture — jp_ocr_app (FIXED 2026-06-21)

Symptoms:

The lens window turned solid white for ~160ms whenever a screen capture was triggered.
The loading spinner appeared static/frozen during the capturing phase (no rotation), then suddenly jumped to a new angle when the OCR result arrived.

White window

Root cause: flash_alpha = 1.0 was set immediately after capture success, painting a fully opaque white cr.rectangle() over the entire 400×400 lens area (via cairo::Operator::Over). The animation timer decremented flash_alpha -= 0.1 per 16ms frame, fading it back to zero over ~160ms. During that fade the lens appeared white.

Fix: Removed flash_alpha entirely — the field was deleted from AppState, dropped from the draw function, and the animation step removed from the 16ms timer. The lens now shows the captured image directly with no overlay.

Frozen spinner

Root cause: The capture flow called std::thread::sleep(400ms) on the main GTK thread to give the compositor time to hide the window before GetImage. Sleeping on the main thread blocks the glib main loop entirely, preventing every registered glib::timeout_add_local callback — including the 16ms animation timer — from firing. As a result spinner_angle was never incremented during the entire 400ms wait, so the spinner appeared as a static arc.

Fix: Replaced the blocking sleep with glib::timeout_add_local(400ms, ...):

// Before (blocks main loop — animation timer cannot fire):
window_poll.set_visible(false);
while glib::MainContext::default().iteration(false) {}
std::thread::sleep(Duration::from_millis(400));
// ... capture, set_visible(true) ...

// After (main loop stays live during the 400ms compositor wait):
window_poll.set_visible(false);
while glib::MainContext::default().iteration(false) {}
glib::timeout_add_local(Duration::from_millis(400), move || {
    // ... capture, set_visible(true) ...
    glib::ControlFlow::Break
});

With the async timer, the main loop keeps running during the 400ms window-hide delay: the animation timer fires every 16ms, spinner_angle advances, and the spinner is already rotating when the window reappears after capture.

Note: The borrow_mut() scope inside the timeout callback is kept tight — state is released before set_visible(true) and queue_draw() are called, per the §2e RefCell scope-guard rule.

Commits: abc70b4.

2k. Spinner Does Not Rotate During OCR Wait — jp_ocr_app (TODO)

Symptom: The loading spinner shows as a static C/U-shaped arc during the "CAPTURING…" / OCR-in-progress phase. spinner_angle is incremented by the 16ms animation timer and queue_draw() is called, but the window does not visually update.

Suspected cause: The capture flow calls window.set_visible(false) then window.set_visible(true) to hide the window during screen grab. The hide/show cycle unmaps and re-maps the GDK surface. After re-map, GTK4's frame-clock scheduling may not resume delivering frames to the window until some internal threshold is met, causing queue_draw() calls to be silently dropped or deferred indefinitely.

Why it works in Lenzu: The production lenzu app avoids set_visible(false/true) during capture — it moves the window off-screen (via x11rb configure_window) so the GDK surface is never unmapped. queue_draw() always hits a live, mapped surface and the frame clock keeps ticking.

Investigation starting points:

Replace set_visible(false/true) with move_window(-2000, -2000) / move_window(win_x, win_y) and add an is_capturing guard in the poll timer to prevent the continuous move_window loop from overriding the off-screen position.
Alternatively, call window.queue_draw() AND area.queue_draw() (the DrawingArea child) after set_visible(true) — GTK4 may not propagate invalidation to children on re-map.

Tracked in code: TODO(proto) comment in prototypes/jp_ocr_app/src/main.rs next to the window_anim.queue_draw() call.

| 2026-06-22 | query_keymap() polling replaced with X11 passive key grabs | XGrabKey delivers events to our x11rb connection with zero round-trip overhead. Polled in 16ms timer with rising-edge + 200ms debounce. 16 grabs registered (4 combos × 4 lock states). Implemented in main lenzu crate (lenzu/src/main.rs). | | 2026-06-22 | GTK-Migrations wiki page created | Community-facing post documenting the GTK4 EventControllerKey regression on override_redirect windows, with validated XGrabKey workaround. |

See also lenzu-desktop-issues.md (main crate perspective). Prototype source code referenced here is at github.com/HidekiAI/lenzu-prototypes.

prototypes desktop issues

Prototypes Desktop Issues — GTK3/4, Windowing, Overlays

1. GTK3→GTK4 Migration — Complete

2. Pitfalls & Lessons Learned (Post-Migration Runtime Issues)

2a. Window XID Discovery: X11Surface::xid() vs PID Scanning

2b. x11rb Connection Init Must Not Panic

2c. set_window_state (keep-above) Must Follow present()

2d. surface.set_input_region() Takes Option<&Region>

2e. RefCell Borrow Must Be Dropped Before Blocking Calls

2f. Session/Model Loading Must Be Optional

2g. Transparent Lens: Draw Must Clear with RGBA(0,0,0,0) + Operator::Source

2h. SCIM Stderr Noise — GTK4 GDK Auto-Launches Broken SCIM

3. Workspace Dependency Graph (Post-Migration)

Remaining workspace issues

4. Dependabot PR Situation (as of 2026-06-20)

5. Decision Log

2i. Spinner Lollipop Tail — jp_ocr_app (FIXED 2026-06-21)

2j. White Window and Frozen Spinner During Capture — jp_ocr_app (FIXED 2026-06-21)

White window

Frozen spinner

2k. Spinner Does Not Rotate During OCR Wait — jp_ocr_app (TODO)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Home

README

User docs

Technical design

Desktop issues

Evaluations

Planning & history

Development

Prototypes reference

Clone this wiki locally

2a. Window XID Discovery: `X11Surface::xid()` vs PID Scanning

2c. `set_window_state` (keep-above) Must Follow `present()`

2d. `surface.set_input_region()` Takes `Option<&Region>`

2e. `RefCell` Borrow Must Be Dropped Before Blocking Calls

2f. `Session`/Model Loading Must Be Optional