Skip to content

Implement virtio-input for keyboard and mouse#122

Merged
jserv merged 6 commits intosysprog21:masterfrom
Mes0903:vinput-pr
Apr 25, 2026
Merged

Implement virtio-input for keyboard and mouse#122
jserv merged 6 commits intosysprog21:masterfrom
Mes0903:vinput-pr

Conversation

@Mes0903
Copy link
Copy Markdown
Collaborator

@Mes0903 Mes0903 commented Mar 19, 2026

This pull request is derived from #34 and #121, implements VirtIO-Input keyboard and mouse devices for semu, enabling host keyboard and mouse input to reach the guest through an SDL-backed window path, in conformance with the VirtIO specification v1.3.

VirtIO-Input

  • Implements separate VirtIO-Input keyboard and mouse devices
  • Uses the VirtIO MMIO transport layer
  • Supports:
    • keyboard key press and release events
    • mouse button events
    • absolute pointer motion
    • scroll wheel events
  • Implements the VirtIO-Input configuration selectors required by the Linux driver:
    • VIRTIO_INPUT_CFG_ID_NAME
    • VIRTIO_INPUT_CFG_ID_SERIAL
    • VIRTIO_INPUT_CFG_ID_DEVIDS
    • VIRTIO_INPUT_CFG_PROP_BITS
    • VIRTIO_INPUT_CFG_EV_BITS
    • VIRTIO_INPUT_CFG_ABS_INFO
  • Handles both EVENTQ and STATUSQ
    • EVENTQ is used for keyboard and mouse event delivery
    • STATUSQ is drained and acknowledged so guest LED updates can make forward progress without stalling the queue

Prerequisites

Linux

sudo apt install libsdl2-dev

macOS

brew install sdl2

Running semu

make check

If SDL is available, semu should open a window for input testing.

Basic guest verification

After the guest has booted, verify that the VirtIO-Input devices are present:

ls /dev/input/event*
cat /proc/bus/input/devices

The guest should expose /dev/input/event* nodes, and /proc/bus/input/devices should report VirtIO input devices.

Manual end-to-end verification

  1. Launch semu:

    make ENABLE_INPUT_DEBUG=1 check

    If SDL is available, semu should open a window for input testing. With ENABLE_INPUT_DEBUG=1, semu also prints the host input events translated by the SDL backend, including:

    • keyboard press/release events
    • mouse button press/release events
    • mouse motion coordinates
    • wheel deltas
    • queue-full drop messages
  2. In the guest, verify that the VirtIO-Input devices are present:

    ls /dev/input/event*
    cat /proc/bus/input/devices
  3. Optionally inspect the guest event stream directly:

    hexdump /dev/input/event1
  4. Focus the SDL window and:

    • type keys
    • click mouse buttons
    • move the mouse
    • scroll the wheel

You should see the translated host-side events in semu's debug output, and new records should appear in the guest input event stream.

Automated testing

  • Adds .ci/test-vinput.sh as a smoke test
  • Verifies:
    • /dev/input/event* exists
    • /proc/bus/input/devices reports VirtIO input devices
  • Runs in GitHub Actions on both Linux and macOS
  • Uses SDL_VIDEODRIVER=offscreen in CI so the guest-visible device
    enumeration path can be validated without requiring a visible desktop
    window

The current automated test covers device enumeration in the guest. It does not yet validate the actual host->guest input injection path, which is still verified manually.


Summary by cubic

Implements virtio-input keyboard and mouse over VirtIO‑MMIO with an SDL‑backed window so host input reaches guest evdev. The window runs on the main thread; the emulator runs in a worker and uses a coalesced wake pipe plus bounded per‑device SPSC queues for fast IRQs and clean shutdown (including gdbstub and window close).

  • New Features

    • Separate keyboard and mouse; key press/release (EV_REP advertised, host autorepeat dropped), mouse buttons, relative motion (REL_X/REL_Y), and wheel (REL_WHEEL/REL_HWHEEL). Click window to grab; Ctrl+Alt+G releases.
    • Linux-required config selectors (ID_NAME, ID_SERIAL, ID_DEVIDS, PROP_BITS, EV_BITS, ABS_INFO); unknown selectors return size=0. Drains STATUSQ for LED updates (no host LED control).
    • Capabilities match the backend: keyboard EV_KEY bitmap from the SDL key map, INPUT_PROP_POINTER, and relative axes only.
    • SDL/main thread translates host input into bounded per-device SPSC queues; the emulator thread drains events, validates virtqueue bounds/descriptors, coalesces wakeups via a self‑pipe, and clears per‑device queues on reset. Window‑close requests stop the emulator cleanly.
    • Optional input debug logging via ENABLE_INPUT_DEBUG=1 prints injected events and queue‑drop messages.
  • Dependencies

    • Build gated on SDL2; if sdl2-config is missing, virtio-input is disabled. CI installs SDL2, exports SDL_VIDEODRIVER=offscreen, adds .ci/test-vinput.sh on Linux/macOS, and extends macOS timeouts.
    • Guest Linux enables CONFIG_INPUT_EVDEV and CONFIG_VIRTIO_INPUT; the device tree adds two VirtIO‑MMIO input nodes with separate IRQs. CI forces fresh Image/rootfs.cpio downloads to avoid stale artifacts.

Written for commit 5c24de0. Summary will update on new commits.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

@Mes0903
Copy link
Copy Markdown
Collaborator Author

Mes0903 commented Mar 22, 2026

The virtio-input feature added in this branch slightly increases boot time: SDL initialization is attempted on every run, and two additional virtio device objects are initialized even in the headless environment used by the netdev test. As a result, the netdev CI test now times out.

The netdev test script runs two network-device tests back to back: on macOS, it exercises both user-mode networking and vmnet, each with a 900-second per-statement expect timeout. However, based on observation, this test sometimes finishes in about 10 minutes, while at other times it exceeds the previous 20-minute limit. In other words, the current setting is still insufficient in slower cases to cover both tests when the guest is slow to configure the network interface or complete the ping exchange.

Although I previously made optimizations for the headless-mode case, they still do not seem to be sufficient. Therefore, for now, I have added a temporary commit to extend the timeout to 30 minutes, so that we can stay focused on the vinput implementation first.

cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

@Mes0903 Mes0903 force-pushed the vinput-pr branch 2 times, most recently from 3784aec to de65ee8 Compare March 23, 2026 17:21
cubic-dev-ai[bot]

This comment was marked as resolved.

@Mes0903
Copy link
Copy Markdown
Collaborator Author

Mes0903 commented Mar 23, 2026

The macOS netdev CI test timed out again. From the log, it appears to be stuck at Starting klogd: OK. This makes me suspect that there may be a condition in the macOS environment that can cause the emulator to hang. For now, I have extended the timeout to two hours so I can determine whether this is really just a timeout issue, or whether there is an actual bug that only occurs on macOS.

@Mes0903 Mes0903 force-pushed the vinput-pr branch 4 times, most recently from f0d5e7f to fbeb8ee Compare March 24, 2026 12:54
@Mes0903
Copy link
Copy Markdown
Collaborator Author

Mes0903 commented Mar 24, 2026

While testing #123 on my own remote, I found that the sound test also experienced timeouts. At the same time, this PR has also shown timeout behavior. These timeouts have occurred in both the netdev and sound tests, but they appear to be random and are difficult to reproduce.

After extending the timeout to two hours, I found that the jobs were not consistently hanging, but rather sometimes just running much more slowly. The difference usually starts to appear only after the init process has been launched, and the gap between the fast and slow cases can be quite large.

For example, the virtio-input test just took 31 minutes, whereas it normally only takes about 5 minutes, such as in this run. The codebase did not change at all between these two runs. Taking the EXT4-fs (vda): mounted filesystem message as an observation point, the former recorded a timestamp of [ 108.175012], while the latter recorded [ 22.946067], which is far smaller.

Based on this, I believe the problem is most likely with the GitHub runner. The fact that #123 also experienced timeouts is a key reason for this conclusion, because #123 does not change the overall architecture at all, it only fixes a value inside virtio-fs, which should not have any impact on execution time.

@sysprog21 sysprog21 deleted a comment from Mes0903 Apr 23, 2026
@sysprog21 sysprog21 deleted a comment from cubic-dev-ai Bot Apr 23, 2026
@sysprog21 sysprog21 deleted a comment from Mes0903 Apr 23, 2026
jserv

This comment was marked as resolved.

Comment thread .github/workflows/main.yml Outdated
Comment thread virtio-input.c Outdated
Comment thread main.c Outdated
Comment thread main.c Outdated
Comment thread common.h Outdated
@shengwen-tw

This comment was marked as resolved.

jserv

This comment was marked as resolved.

Update guest kernel configuration to enable virtio-input.

Co-authored-by: Shengwen Cheng <shengwen1997.tw@gmail.com>
@Mes0903 Mes0903 force-pushed the vinput-pr branch 2 times, most recently from 71a4803 to c725884 Compare April 24, 2026 21:53
@shengwen-tw shengwen-tw self-requested a review April 25, 2026 08:53
Copy link
Copy Markdown
Collaborator

@jserv jserv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drop "DO NOT MERGE" commits. Then, I will merge this PR.

Mes0903 and others added 5 commits April 25, 2026 23:26
Add virtio-input keyboard and mouse devices and connect them to the SDL
frontend so host input can reach the guest.

Introducing window-driven input requires more than just another virtio
device. Once SDL owns the frontend, the emulator needs a threading model
that keeps video handling on the process main thread, lets the emulator
continue running while the window loop is active, and still allows a
window-close request to wake a blocked SMP 'poll(-1)' path. The same
shutdown path also needs to work in both normal execution and
'gdbstub'-driven execution.

There are a few ways to structure this. One option is to keep the older
execution model and bolt the SDL loop onto it. In practice that is a
poor fit. The official SDL documentation [1] notes that most graphics
backends are not thread-safe and that SDL video functions should only be
called from the application's main thread. That matters especially on
macOS, and a plain shared close flag is still not enough to unblock an
idle emulator. The more workable structure is to reserve the original
main thread for the window loop and move 'semu_run()' /
'semu_run_debug()' to a worker thread, with an explicit wakeup path
from the frontend back into the emulator.

This commit takes that approach. The main thread now stays in the window
backend when one is present, while the emulator runs on a worker thread.
Shutdown is completed with a wake pipe so closing the window can break a
blocked SMP poll and let both sides unwind cleanly.

At the device level, this adds virtio-input keyboard and mouse support,
including MMIO state, configuration handling, event injection, and
integration with the existing virtio-mmio framework. The advertised
capabilities are tightened to match what the frontend can actually
generate. The mouse is modeled as a relative pointing device with
'REL_X'/'REL_Y' and wheel axes rather than an absolute pointer.

On the frontend side, SDL input is translated into Linux input events
for keyboard, mouse buttons, relative mouse motion, and wheel scrolling.
Repeated SDL keydown events are dropped once 'EV_REP' is advertised so
the guest remains responsible for key-repeat timing. Mouse clicks also
enter SDL relative mouse mode, and 'Ctrl+Alt+G' releases that grab again
so continuous relative motion can be delivered to the guest.

At this stage the SDL thread still calls the virtio-input update helpers
directly, so shared device state remains protected by
'virtio_input_mutex'. A later commit moves that ownership boundary onto
the emulator thread.

The guest-side enumeration path is covered by a new virtio-input smoke
test in CI, and manual testing still confirms end-to-end event delivery
through the guest evdev interfaces.

[1]: https://wiki.libsdl.org/SDL3/FAQDevelopment

Co-authored-by: Shengwen Cheng <shengwen1997.tw@gmail.com>
Exclude the SMP stamp file, full root filesystem image, and the
buildroot, linux, and rootfs working directories so they do not
appear as untracked noise.
Set SDL_VIDEODRIVER=offscreen in common.sh so that SDL initialises
without a display server, which is required for headless CI runs.
Manual testing of virtio-input currently relies on guest-side tools such
as 'hexdump /dev/input/eventX'. That confirms that input reaches the
guest, but it is not very convenient when checking how host input is
turned into virtio-input events in real time. Relative mouse motion and
wheel input are especially tedious to inspect from the raw evdev stream
alone.

Add a build-time input debug switch for the virtio-input event injection
path. When 'ENABLE_INPUT_DEBUG=1' is set in the Makefile, the build
defines 'SEMU_INPUT_DEBUG' and enables verbose logging in the
virtio-input update helpers.

With the flag enabled, the virtio-input update path prints the events
that are about to be injected into the guest. The output includes:

- keyboard events, including the Linux input key code and evdev value
  (release, press, or repeat)
- mouse button events, including the Linux input button code and
  pressed state
- relative mouse motion events, including dx/dy deltas
- mouse wheel events, including dx/dy deltas
- drop messages when an input event cannot be delivered because the
  guest has not supplied buffers to 'EVENTQ'
The current virtio-input path gives the SDL thread direct access to
guest-visible virtio-input state. SDL input is translated on the main
thread and then immediately forwarded into the virtio-input update
helpers, which write EVENTQ, advance the used ring, and set
'InterruptStatus'. At the same time, the emulator thread polls
'virtio_input_irq_pending()' from 'emu_tick_peripherals()', which puts
this shared state on a hot path.

That ownership model creates an awkward tradeoff. Without
synchronization, the SDL thread and the emulator thread can race on the
same 'virtio_input_state_t'. With 'virtio_input_mutex', the hot path has
to pay for cross-thread locking just to observe whether an interrupt is
pending.

This commit moves the ownership boundary earlier. SDL input is still
collected and translated on the main thread, but it is now stored as
bounded host event records instead of updating guest memory, virtqueues,
or virtio registers directly. 'emu_tick_peripherals()' drains those
records on the emulator thread and dispatches them into the existing
'virtio_input_update_*()' helpers. Guest-visible virtio-input state is
therefore owned by the emulator thread, while the SDL thread only owns
host event collection.

This change also simplifies the interrupt pending path.
'virtio_input_irq_pending()' now only checks 'InterruptStatus', and the
vinput-internal mutex is removed because descriptor processing, queue
updates, status queue draining, MMIO accesses, and other guest-visible
state transitions all run on the emulator thread.

At the interface level, the new event queue carries keyboard, mouse
button, mouse motion, and mouse wheel input in a host-centric form. This
keeps the split between the window side and the emulator side explicit
while preserving the existing internal virtio-input event injection
helpers.
@Mes0903
Copy link
Copy Markdown
Collaborator Author

Mes0903 commented Apr 25, 2026

The current CI failure is expected because the Image on the main blob branch is still the old one, and CI does not rebuild the kernel Image.

I've manually rebuilt the kernel locally with the current config and tested it by hand. The virtio-input functionality works correctly with the rebuilt Image.

@jserv jserv merged commit ecf0afa into sysprog21:master Apr 25, 2026
@jserv
Copy link
Copy Markdown
Collaborator

jserv commented Apr 25, 2026

Thank @Mes0903 for contributing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants