Skip to content

Fix dmenu keyboard grab on Wayland#227

Merged
paperbenni merged 3 commits intomainfrom
fix-wayland-dmenu-keyboard-grab-7409422313332858039
Mar 14, 2026
Merged

Fix dmenu keyboard grab on Wayland#227
paperbenni merged 3 commits intomainfrom
fix-wayland-dmenu-keyboard-grab-7409422313332858039

Conversation

@paperbenni
Copy link
Copy Markdown
Member

@paperbenni paperbenni commented Mar 13, 2026

Fixes an issue where dmenu would occasionally crash when trying to start on Wayland because it failed to acquire the keyboard grab. The problem happened because dmenu requests a keyboard grab right after creating its window, but the window might not be fully tracked in the window_index yet since it's an unmanaged overlay. We now also search self.space.elements() to find the surface for the keyboard grab.


PR created automatically by Jules for task 7409422313332858039 started by @paperbenni

Summary by Sourcery

Bug Fixes:

  • Prevent crashes when dmenu fails to acquire a keyboard grab by falling back to searching Wayland space elements for the corresponding surface.

Summary by CodeRabbit

  • Bug Fixes

    • Improved keyboard focus reliability in Wayland by adding a fallback surface-to-window resolution when primary window lookups fail, reducing lost or misdirected keyboard input.
  • Refactor

    • Simplified and clarified keyboard-focus lookup flow to make focus determination more robust and easier to maintain.

…lications that request an exclusive keyboard grab right after mapping would often fail because their surface hasn't been assigned to a managed window yet. Now `keyboard_focus_for_xsurface` also searches `self.space.elements()` for unmanaged surfaces.

Co-authored-by: paperbenni <15818888+paperbenni@users.noreply.github.com>
@google-labs-jules
Copy link
Copy Markdown
Contributor

👋 Jules, reporting for duty! I'm here to lend a hand with this pull request.

When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down.

I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job!

For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with @jules. You can find this option in the Pull Request section of your global Jules UI settings. You can always switch back!

New to Jules? Learn more at jules.google/docs.


For security, I will only act on instructions from the user who triggered this task.

@sourcery-ai
Copy link
Copy Markdown

sourcery-ai bot commented Mar 13, 2026

Reviewer's guide (collapsed on small PRs)

Reviewer's Guide

Adjusts Wayland XWaylandKeyboardGrabHandler focus resolution to fall back to scanning space elements for the wl_surface when the window_index lookup fails, preventing dmenu crashes when its unmanaged overlay window is not yet indexed.

Sequence diagram for updated XWayland keyboard grab focus resolution

sequenceDiagram
    actor DmenuProcess
    participant XWayland as XWaylandServer
    participant Compositor as WaylandState
    participant WindowIndex as window_index
    participant Space as space

    DmenuProcess->>XWayland: Request keyboard grab
    XWayland->>Compositor: keyboard_focus_for(surface)

    alt Surface is indexed window
        Compositor->>WindowIndex: get(win_id_for_surface(surface))
        WindowIndex-->>Compositor: Window
        Compositor-->>XWayland: KeyboardFocusTarget_Window
        XWayland-->>DmenuProcess: Keyboard grab granted
    else Surface is unmanaged overlay (not in window_index)
        Compositor->>WindowIndex: get(win_id_for_surface(surface))
        WindowIndex-->>Compositor: None
        Compositor->>Space: elements()
        Space-->>Compositor: iterator over windows
        Compositor->>Space: find window with wl_surface == surface
        Space-->>Compositor: Window (cloned)
        Compositor-->>XWayland: KeyboardFocusTarget_Window
        XWayland-->>DmenuProcess: Keyboard grab granted (no crash)
    end
Loading

Class diagram for updated WaylandState XWaylandKeyboardGrabHandler implementation

classDiagram
    class WaylandState {
        window_index
        space
        keyboard_focus_for(surface: WlSurface) Option_KeyboardFocus
    }

    class XWaylandKeyboardGrabHandler {
        <<interface>>
        keyboard_focus_for(surface: WlSurface) Option_KeyboardFocus
    }

    class KeyboardFocusTarget {
        Window
    }

    class Window {
        wl_surface() WlSurface
    }

    class Space {
        elements() Iterator_Window
    }

    WaylandState ..|> XWaylandKeyboardGrabHandler
    WaylandState o--> Space
    WaylandState o--> WindowIndex
    WindowIndex --> Window
    Space --> Window
    KeyboardFocusTarget --> Window

    class WindowIndex {
        get(win_id: WindowId) Window
    }
Loading

File-Level Changes

Change Details Files
Make keyboard focus resolution more robust by falling back from window_index to searching space elements for the target wl_surface.
  • Replaced chained ? lookups on window_id_for_surface and window_index with explicit if-let checks to avoid early return when the window is not indexed
  • Added fallback search over self.space.elements() comparing each element's wl_surface to the requested surface and cloning the matching window
  • Mapped the found space element into a KeyboardFocusTarget::Window, ensuring a valid KeyboardFocusTarget is returned when possible instead of None
src/backend/wayland/compositor/handlers.rs

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

Copy link
Copy Markdown

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • The new fallback search over self.space.elements() introduces a linear scan on every keyboard grab; consider whether this handler is called frequently enough to warrant caching or a helper that avoids repeated full scans.
  • The early-return logic in keyboard_focus_for_surface could be simplified by factoring the two lookup steps into a small helper (e.g., window_for_surface) to keep the control flow tighter and avoid duplicated lookup logic in the future.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The new fallback search over `self.space.elements()` introduces a linear scan on every keyboard grab; consider whether this handler is called frequently enough to warrant caching or a helper that avoids repeated full scans.
- The early-return logic in `keyboard_focus_for_surface` could be simplified by factoring the two lookup steps into a small helper (e.g., `window_for_surface`) to keep the control flow tighter and avoid duplicated lookup logic in the future.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Mar 13, 2026

Warning

Rate limit exceeded

@paperbenni has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 0 minutes and 21 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: fb1d3429-8915-407e-b41f-59fc95d18f9b

📥 Commits

Reviewing files that changed from the base of the PR and between 24620b5 and 8b6f2a4.

📒 Files selected for processing (1)
  • src/backend/wayland/compositor/state.rs
📝 Walkthrough

Walkthrough

Reworks surface→window lookup and keyboard focus resolution: adds WaylandState::window_for_surface to locate a Window by WlSurface (direct surface equality or surface_under with WindowSurfaceType::ALL) and updates keyboard_focus_for_xsurface to use that as a fallback when window_id/window_index lookups fail.

Changes

Cohort / File(s) Summary
Keyboard focus handler
src/backend/wayland/compositor/handlers.rs
Refactors keyboard_focus_for_xsurface to perform an explicit nested option check for window_id_for_surface + window_index, and add a fallback that calls WaylandState::window_for_surface(surface) to map a WlSurface to KeyboardFocusTarget::Window when earlier lookups yield None.
Wayland state lookup
src/backend/wayland/compositor/state.rs
Adds pub(crate) fn window_for_surface(&self, surface: &smithay::reexports::wayland_server::protocol::wl_surface::WlSurface) -> Option<Window> which finds a Window by direct wl_surface() match or by checking surface_under(..., WindowSurfaceType::ALL).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

Poem

🐰 In tunnels of code where surfaces roam,
I sniff for wl_surfaces, guide them home.
When ids go missing and lookups stall,
I hop through elements and answer the call.
Focus found — a little rabbit's triumph, after all. 🥕✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Fix dmenu keyboard grab on Wayland' directly addresses the main issue being resolved—dmenu crashing when trying to acquire a keyboard grab on Wayland due to unmanaged window tracking. It is concise, specific, and clearly summarizes the primary change.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch fix-wayland-dmenu-keyboard-grab-7409422313332858039
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@paperbenni
Copy link
Copy Markdown
Member Author

@jules
Please address the comments from this code review:

Overall Comments

  • The new fallback search over self.space.elements() introduces a linear scan on every keyboard grab; consider whether this handler is called frequently enough to warrant caching or a helper that avoids repeated full scans.
  • The early-return logic in keyboard_focus_for_surface could be simplified by factoring the two lookup steps into a small helper (e.g., window_for_surface) to keep the control flow tighter and avoid duplicated lookup logic in the future.

…lications that request an exclusive keyboard grab right after mapping would often fail because their surface hasn't been assigned to a managed window yet. Now `keyboard_focus_for_xsurface` also searches `self.space.elements()` for unmanaged surfaces.

Co-authored-by: paperbenni <15818888+paperbenni@users.noreply.github.com>
Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
update_script_3.sh (1)

1-2: Consider dropping this one-off rewrite script before merge.

This script is PR-scoped migration tooling tied to exact source formatting; keeping it in-tree adds maintenance noise and accidental re-run risk.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@update_script_3.sh` around lines 1 - 2, The one-off rewrite script
update_script_3.sh should be removed from the PR to avoid keeping PR-scoped,
formatting-dependent tooling in-tree; delete the file (or move it out of the
repository to private/archival location) and ensure the actual code change it
produced (the rename of window_id_for_surface to window_for_surface and the new
implementation matching smithay::desktop::WindowSurfaceType usage) is committed
separately in the src/backend/wayland/compositor/state.rs change set; also
verify there are no CI or project references to update_script_3.sh left behind.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/backend/wayland/compositor/state.rs`:
- Around line 1074-1080: The current check in the closure over
self.space.elements() uses Window::surface_under((0.0,0.0), ...) which is a
point hit-test and misses descendant surfaces not covering that point; replace
that logic with Window::with_surfaces() to iterate the window's surface tree and
compare each surfaced wl_surface to the target surface. Concretely, inside the
find closure for each window value w (referencing w.wl_surface()), call
w.with_surfaces(|surf, _| surf == surface) (or equivalent) and return true if
any match; keep the direct w.wl_surface() == Some(surface) check as before but
remove the surface_under(...) branch. Ensure you reference the existing symbols
w.with_surfaces, w.wl_surface, and surface in the updated condition.

---

Nitpick comments:
In `@update_script_3.sh`:
- Around line 1-2: The one-off rewrite script update_script_3.sh should be
removed from the PR to avoid keeping PR-scoped, formatting-dependent tooling
in-tree; delete the file (or move it out of the repository to private/archival
location) and ensure the actual code change it produced (the rename of
window_id_for_surface to window_for_surface and the new implementation matching
smithay::desktop::WindowSurfaceType usage) is committed separately in the
src/backend/wayland/compositor/state.rs change set; also verify there are no CI
or project references to update_script_3.sh left behind.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: dc518a38-6770-4a84-ab7c-88edc7f9a98d

📥 Commits

Reviewing files that changed from the base of the PR and between 2c97e85 and 24620b5.

📒 Files selected for processing (3)
  • src/backend/wayland/compositor/handlers.rs
  • src/backend/wayland/compositor/state.rs
  • update_script_3.sh
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/backend/wayland/compositor/handlers.rs

Comment on lines +1074 to +1080
self.space.elements().find(|w| {
if w.wl_surface().as_deref() == Some(surface) {
return true;
}
w.surface_under((0.0, 0.0), WindowSurfaceType::ALL)
.map(|(hit_surface, _)| hit_surface == *surface)
.unwrap_or(false)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

In smithay 0.7.0, does Window::surface_under((0.0, 0.0), WindowSurfaceType::ALL)only test that coordinate, or can it reliably identify any descendantWlSurfaceof the window? What API is recommended to verify whether an arbitraryWlSurfacebelongs to aWindow surface tree?

💡 Result:

Window::surface_under(point, WindowSurfaceType::ALL) is strictly a point-based hit-test: it walks the window’s relevant surface trees (toplevel and, if enabled, popups; plus subsurfaces depending on the flags) and returns the topmost surface whose input region contains that point. So calling surface_under((0.0, 0.0), ...) only tells you what (if anything) is hit at (0,0); it is not a reliable way to identify “any descendant WlSurface of the window” unless that descendant happens to be under the tested coordinate. [1]

To verify whether an arbitrary WlSurface belongs to a given Window’s surface tree, the intended API is to iterate the window’s surfaces and compare against the target surface, via:

  • Window::with_surfaces(|wl_surface, _surface_data| { ... }) (includes the window’s surface tree and, if you use PopupManager, also its popups). [1][2]

Sources:
[1] smithay Window implementation (shows surface_under is a point hit-test and with_surfaces traverses the surface trees) https://docs.rs/smithay/latest/src/smithay/desktop/wayland/window.rs.html
[2] smithay Window docs (documents with_surfaces / surface_under) https://docs.rs/smithay/latest/smithay/desktop/struct.Window.html


Switch to Window::with_surfaces() for reliable surface tree membership check.

The code uses surface_under((0.0, 0.0), WindowSurfaceType::ALL) to verify if a surface belongs to a window. However, surface_under() is strictly a point-based hit-test—it only identifies surfaces whose input region contains that coordinate. Any descendant surface not covering (0,0) will be missed, breaking grab fallback for such surfaces.

Use Window::with_surfaces() instead, which iterates the window's surface tree and enables reliable membership checking regardless of surface position.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/backend/wayland/compositor/state.rs` around lines 1074 - 1080, The
current check in the closure over self.space.elements() uses
Window::surface_under((0.0,0.0), ...) which is a point hit-test and misses
descendant surfaces not covering that point; replace that logic with
Window::with_surfaces() to iterate the window's surface tree and compare each
surfaced wl_surface to the target surface. Concretely, inside the find closure
for each window value w (referencing w.wl_surface()), call
w.with_surfaces(|surf, _| surf == surface) (or equivalent) and return true if
any match; keep the direct w.wl_surface() == Some(surface) check as before but
remove the surface_under(...) branch. Ensure you reference the existing symbols
w.with_surfaces, w.wl_surface, and surface in the updated condition.

@paperbenni
Copy link
Copy Markdown
Member Author

@jules
Verify each finding against the current code and only fix it if needed.

Inline comments:
In @src/backend/wayland/compositor/state.rs:

  • Around line 1074-1080: The current check in the closure over
    self.space.elements() uses Window::surface_under((0.0,0.0), ...) which is a
    point hit-test and misses descendant surfaces not covering that point; replace
    that logic with Window::with_surfaces() to iterate the window's surface tree and
    compare each surfaced wl_surface to the target surface. Concretely, inside the
    find closure for each window value w (referencing w.wl_surface()), call
    w.with_surfaces(|surf, _| surf == surface) (or equivalent) and return true if
    any match; keep the direct w.wl_surface() == Some(surface) check as before but
    remove the surface_under(...) branch. Ensure you reference the existing symbols
    w.with_surfaces, w.wl_surface, and surface in the updated condition.

Nitpick comments:
In @update_script_3.sh:

  • Around line 1-2: The one-off rewrite script update_script_3.sh should be
    removed from the PR to avoid keeping PR-scoped, formatting-dependent tooling
    in-tree; delete the file (or move it out of the repository to private/archival
    location) and ensure the actual code change it produced (the rename of
    window_id_for_surface to window_for_surface and the new implementation matching
    smithay::desktop::WindowSurfaceType usage) is committed separately in the
    src/backend/wayland/compositor/state.rs change set; also verify there are no CI
    or project references to update_script_3.sh left behind.

@google-labs-jules
Copy link
Copy Markdown
Contributor

@jules
Verify each finding against the current code and only fix it if needed.

Inline comments:
In @src/backend/wayland/compositor/state.rs:

  • Around line 1074-1080: The current check in the closure over
    self.space.elements() uses Window::surface_under((0.0,0.0), ...) which is a
    point hit-test and misses descendant surfaces not covering that point; replace
    that logic with Window::with_surfaces() to iterate the window's surface tree and
    compare each surfaced wl_surface to the target surface. Concretely, inside the
    find closure for each window value w (referencing w.wl_surface()), call
    w.with_surfaces(|surf, _| surf == surface) (or equivalent) and return true if
    any match; keep the direct w.wl_surface() == Some(surface) check as before but
    remove the surface_under(...) branch. Ensure you reference the existing symbols
    w.with_surfaces, w.wl_surface, and surface in the updated condition.

Nitpick comments:
In @update_script_3.sh:

  • Around line 1-2: The one-off rewrite script update_script_3.sh should be
    removed from the PR to avoid keeping PR-scoped, formatting-dependent tooling
    in-tree; delete the file (or move it out of the repository to private/archival
    location) and ensure the actual code change it produced (the rename of
    window_id_for_surface to window_for_surface and the new implementation matching
    smithay::desktop::WindowSurfaceType usage) is committed separately in the
    src/backend/wayland/compositor/state.rs change set; also verify there are no CI
    or project references to update_script_3.sh left behind.

Thank you for the review! I have updated window_for_surface to use with_surfaces to properly check the whole surface tree as suggested, and I have removed the temporary script.

…lications that request an exclusive keyboard grab right after mapping would often fail because their surface hasn't been assigned to a managed window yet. Now `keyboard_focus_for_xsurface` also searches `self.space.elements()` for unmanaged surfaces, checking their whole surface tree.

Co-authored-by: paperbenni <15818888+paperbenni@users.noreply.github.com>
@paperbenni paperbenni merged commit e95f578 into main Mar 14, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant