[LoongArch] Fix lossy IPI events and fine-grained Virtio IRQ deassertion

[enkerewpo]

Affected upstream snapshots

• hvisor: syswonder/hvisor, branch dev, commit 9dbacc423100c1deb2e7d3be6d1bd6bef06e520c
• hvisor-tool: syswonder/hvisor-tool, branch main, commit e4f6931dc9d26e9f748eacd37f3b982fd66778b0
• Architecture/platform: LoongArch64, Loongson 3A6000 public board with 7A2000
• Related hvisor-tool issue: [LoongArch] Fix Virtio MMIO ACK semantics and console RX data loss hvisor-tool#109

cc @li041

Summary

The current LoongArch path has several coupled interrupt-state problems. The common issue is that software events, guest IPI actions, guest GINTC HWI bits, Virtio device status, and the backend clear request are not represented as one-to-one state transitions.

The result can be lost hvisor events, imprecise guest IPI delivery, global clearing of unrelated guest HWIs, random Virtio console stalls, and a root serial console that appears to work through early polling while its real UART interrupt is not connected.

This issue describes the complete chain because fixing only one layer can hide the failure without making the protocol correct.

Problems confirmed in the current dev branch

1. arch_prepare_send_event() removes target events

src/arch/loongarch64/ipi.rs#L360-L367 repeatedly calls fetch_event(cpu_id) before a new event is queued. fetch_event() is a destructive pop_front() in src/event.rs#L53-L70.

The generic send path calls arch_prepare_send_event(), then enqueues the new event, then sends the IPI (src/event.rs#L177-L192). Therefore the LoongArch preparation hook does not wait for the target to process earlier work. It processes nothing and discards every queued wakeup, shutdown, Virtio, IVC, or guest-IPI event it pops.

The physical IPI handler later calls reset_ipi(), which clears all action bits and enables all bits again (src/arch/loongarch64/trap.rs#L1270-L1292). This can also clear an action unrelated to the hvisor event doorbell.

2. Physical hvisor events and guest Linux IPI actions are conflated

There are two separate state domains:

• a physical IPI action used only as a doorbell so a pCPU drains hvisor software events;
• the guest-visible LoongArch IPI status/action bits used by Root Linux SMP.

The current implementation uses shared per-core MMIO windows for local status/enable/clear (src/arch/loongarch64/ipi.rs#L73-L85, #L242-L293). On the tested board this made hvisor four-core bring-up timing-dependent; replacing local status/enable/clear accesses with local IOCSR accesses, as LoongArch Linux does, made CPU0-3 initialization repeatable.

For trapped guest IPI sends, the action bits must be kept as a pending bitmap and cleared by the guest's exact IPI_CLEAR mask. Flushing an unrelated hvisor event queue is not an implementation of IOCSR_IPI_SEND_BLOCKING.

The virtualization boundary should follow the LoongArch KVM state machine:

• atomically OR a sent action into the target guest IPI status;
• assert the guest IPI line only on a zero-to-nonzero transition;
• atomically clear only the guest-provided action mask;
• deassert only when no pending action remains;
• make the trapped send visible before returning, but do not wait for the guest to execute a future clear, which can deadlock self-IPI paths.

References: native Linux LoongArch IPI path, KVM LoongArch IPI state machine.

3. GINTC injection and clear are global rather than per bit

The current HWI injection path writes one GINTC guest HWI value, enables a timer, and the clear path writes the injected field to zero (src/device/irqchip/ls7a2000/mod.rs#L113-L160). The timer interrupt also calls the same global clear helper (src/arch/loongarch64/trap.rs#L1295-L1300).

If Virtio console IRQ4 and block IRQ5 are pending together, acknowledging one device can clear both. The current hwis/hwip/hwic names in gintc.rs correspond to the three 8-bit GINTC fields. The injected guest pending field must be updated without overwriting the physical pending/control fields.

A correct implementation should maintain a per-target-pCPU guest-HWI asserted bitmap, update only the requested bit, and synchronize only the guest-visible pending field while preserving the other GINTC fields. Timer-based forced clearing should not be part of the normal Virtio acknowledgement protocol.

4. Hypercall 20 cannot identify the line to deassert

src/hypercall/mod.rs#L91-L100 handles HvClearInjectIrq by broadcasting IPI_EVENT_CLEAR_INJECT_IRQ to every running nonroot CPU. The call carries no zone or IRQ arguments.

The ABI should instead use the two existing hypercall arguments as (zone_id, irq_id). hvisor should:

1. require the caller to be Root;
2. validate that the target zone exists;
3. resolve the target vCPU for that zone and IRQ;
4. clear only that HWI bit;
5. clear all bits only during explicit zone reset/shutdown.

The hvisor-tool ioctl must be changed in the same release so userspace and the kernel driver pass the same (zone_id, irq_id) pair. See the linked hvisor-tool issue for the backend side.

5. Current hvisor-tool violates Virtio MMIO ACK semantics

At the affected main commit, hvisor-tool clears the LoongArch injection while reading InterruptStatus, uses logical !value instead of bitwise ~value for ACK, and consumes PTY data into trashbuf while no RX descriptor exists. These issues are tracked in the linked hvisor-tool issue.

The required end-to-end rule is:

• status read is side-effect free;
• InterruptACK clears only the acknowledged status bits;
• the (zone_id, irq_id) line is deasserted only when that device's remaining status becomes zero;
• no PTY input is consumed without an RX descriptor.

Reference: Virtio 1.2 MMIO InterruptStatus/InterruptACK.

6. Root UART0 DTS lacked the real interrupt route

The Root Linux DTS previously described CPU UART0 MMIO but did not enable LIOINTC or provide a valid interrupt parent/specifier. Linux reported IRQ index 0 not found and registered ttyS0 with IRQ0. Early console output still appeared because earlycon polls MMIO, but interactive input was slow and could overrun.

The validated route on the 3A6000 public board is:

UART0 0x1fe001e0
  -> LIOINTC source 10, level high
  -> LIOINTC int0
  -> CPUINTC HWI2
  -> Linux dynamic virq 36

The minimal DTS fix has already been merged separately as enkerewpo/linux-hvisor-loongarch64#1, commit cbe25de0. /proc/interrupts reported LIOINTC hwirq10 for ttyS0 and the count increased with input.

Reference: Linux LoongArch IRQ chip model.

Proposed hvisor implementation

Event doorbell

• Keep a FIFO per pCPU; never remove an event from a sender-side prepare hook.
• Track whether the target's physical event doorbell is armed.
• Send the physical doorbell only when transitioning from unarmed/empty to armed/nonempty.
• Drain events on the target and recheck the queue while changing the armed state so an enqueue cannot be left without a doorbell.
• Reserve one physical IPI action bit for this doorbell and clear only that bit in the handler.

Guest IPI

• Keep an atomic pending action bitmap per target vCPU.
• OR on send, AND with the complement of the exact clear mask on guest clear.
• Assert on 0 -> nonzero; deassert on nonzero -> 0.
• Reset status, enable state, and mailboxes when a zone is reused.

Guest HWI/GINTC and Virtio

• Keep a per-pCPU guest-HWI asserted bitmap.
• Implement set_guest_irq_line(target_cpu, irq, asserted) as a per-bit operation.
• Synchronize only the guest-visible GINTC pending field and preserve the physical pending/control fields.
• Change hypercall 20 to (zone_id, irq_id) deassert semantics.
• Coordinate the ABI change with hvisor-tool's status/ACK state machine.

Validation performed with the reference implementation

The reference implementation is based on the vPLIC-oriented li041/hvisor@8b9ecde branch and syswonder/hvisor-tool@e4f6931. It is a validated design reference, not a statement that the hvisor patch applies to current dev without rebasing after the build/Kconfig refactor.

Two complete board cycles were run, with Root, nonroot, hvisor-tool, Virtio console, and Virtio block all active before stress and cyclictest:

Test	Cycle 1	Cycle 2 after physical reboot
hvisor pCPU initialization	CPU0-3 PASS	CPU0-3 PASS
Root SMP	CPU0-1 online	CPU0-1 online
nonroot Linux	CPU2 online	CPU2 online
UART0	ttyS0 IRQ36	ttyS0 IRQ36
Virtio console	600/600 over 600 s	180/180 over 180 s
Virtio block	37,498 loops over 600 s	11,286 loops over 180 s
IRQ4/IRQ5 independent ACK	0x0c -> 0x08 -> 0x00	PASS
Root cyclictest	CPU0 2/2/16 us; CPU1 2/2/3 us	CPU0 2/2/75 us; CPU1 2/2/5 us
nonroot cyclictest	1/1/5 us	1/1/5 us

No target panic, BUG, warning, RCU stall, Virtio error, or I/O error was found in the final runs. The full report records the remaining test limits, including only two cold cycles, nonzero but stable Root spurious ERR counts, and no nonroot multi-vCPU test.

Reference report, patches, and logs

• Full report, patches, build evidence, and clean logs
  SHA-256: 3cbb0f6693bf088e71a4917ed30328b6a81754ca5f6ffc90b11d935d1b04be70
• Report and fix patches only
  SHA-256: a9a09ec1d8b27f933a0aaeef720903af54d2e309f4290e751298b6d700f1d971

The archives were scanned for access keys, private keys, passwords, and common token formats before publication. The full archive includes exact base commits, Linux mail-format patches, clean before/after logs, and patch-apply checks.

Demo video

a video with:

1. root linux booted with CPU0 + CPU1 multicore
2. mounted UART0 interrupt with fast response (top refresh), and no terminal overrun happens again
3. succesfully booted nonroot linux on CPU2 with virtio-console + virtio-blk
4. no random stuck on virito-console after several screen -r and can be verified after reboot multiple times

hvisor-loongarch-virtio-demo-github.mp4