How to use Remote I/O

[JLValdivieso]

I am currently working on implementing Remote I/O capabilities within a RISC-V architecture. Could you provide any reference examples or guidance on the integration process?

[joaopeixoto13]

Hi @JLValdivieso , you can take a look at the VirtIO demo in bao-demos, since the demo also supports the qemu-riscv64-virt platform.

[joaopeixoto13]

Hi @JLValdivieso
If you have any questions about VirtIO or how to use it in your target application, please ping me, and I’ll be happy to help.

[joaopeixoto13]

But just to give you a brief overview: our Remote I/O is our implementation of VirtIO request/response forwarding between the frontend VM and its associated backend VM in Bao. We currently support a wide range of VirtIO backends, all running on Linux — meaning you always need a Linux VM (aka Linux I/O or Linux service VM). You can find the supported backends here: https://github.com/bao-project/bao-virtio-dm/tree/main.

As an example configuration, take a look at this demo. It features three guests (one FreeRTOS and two Linux VMs) and one Linux I/O VM holding all the VirtIO backends. The two Linux guests share the network and storage devices, while the FreeRTOS guest accesses a virtual console.

The configuration is quite straightforward: you specify whether a device is a backend or frontend, link each device with its frontend/backend counterpart, and assign an interrupt and a static shared memory region for passing/sharing I/O buffers between the frontend and backend. In the backend VM, you only need to add a bao-io-dispatcher node (example here) with a description of the shared memory region and the interrupt used by each backend device.

[JLValdivieso]

Hi @joaopeixoto13

Many thanks for your support and for the introduction to your Remote I/O implementation. I am still setting up the demo and learning more about your approach. I intend to use this implementation in future work on a platform based on the RISC-V CVA6 core, I currently have a stable version of my platform where I can integrate your work. I will attempt to replicate your results and will let you know if I have any questions.

Regarding the shared memory region: I have been reading your paper, I have some questions about the specific mechanism used to copy data between the frontend and the shared memory region. To the best of my knowledge, your implementation relies on a Linux-specific mechanism. However, how would this function if we were using Zephyr RTOS as the frontend?

In an RTOS or bare-metal application, we lack the swiotlb mechanism. Could the data copy be performed directly in these cases, or would a direct copy introduce security concerns?

[joaopeixoto13]

Hi @JLValdivieso ,

Thanks for the update, and great to hear that you’re working on integrating Bao on a CVA6-based platform. Feel free to reach out if you run into any issues.

Regarding your question about the shared memory mechanism: you are absolutely right that for Linux frontend guests we rely on the swiotlb mechanism. This is mainly because Linux may place buffers in memory regions that are not directly accessible or suitable for sharing with the backend, so swiotlb provides a safe way to bounce those buffers into a shared region.

However, in the case of RTOS or bare-metal frontends, things are usually much simpler. These systems typically have direct control over memory placement, so you are free to place the I/O buffers directly inside the shared memory region used by the VirtIO queues. In those cases, you don’t need a swiotlb-like mechanism.

For example, in our FreeRTOS virtual console demo we implemented a very simple bare-metal VirtIO console driver that follows exactly this approach. You can take a look here:
https://github.com/bao-project/bao-baremetal-guest/blob/85e770f3031741954270cdb69457f98b9fbf9c19/src/core/virtio_console.c

In that implementation we introduce the concept of a memory pool inside the shared memory region associated with each virtqueue (in this case, one for TX and one for RX). Each time the frontend wants to print something, it allocates memory for the I/O buffer from the next available position in that pool and then notifies the backend. The relevant part is here:
https://github.com/bao-project/bao-baremetal-guest/blob/master/src/core/virtio_console.c#L240

Since the backend has access to the same shared memory region, and the descriptor table contains the address and size of the buffer, it can directly read the data and process the request.

So in short:

• Linux frontends typically rely on swiotlb for safe buffer sharing.
• RTOS/bare-metal frontends can usually place buffers directly in the shared memory region, which simplifies the implementation considerably.

Hope this helps clarify things, and let me know if you have further questions while experimenting with it.

[JLValdivieso]

Thank you for the quick response. I will let you know once I have it running on my platform or any doubt. Let's keep in touch.

Best regards

[JLValdivieso]

Hi @joaopeixoto13,

I hope you are doing well. I have a question regarding the VirtIO devices in your implementation. Based on the YAML configuration files, I noticed that each Linux frontend utilizes two VirtIO devices: one for virtio-net and another for virtio-block.

Could you please clarify the specific purpose of the virtio-block device in this context? Specifically, I am unsure if it is a necessary requirement for the correct configuration of the Remote I/O mechanism.

[joaopeixoto13]

Hi @JLValdivieso

Thanks for your question!

The YAML configuration files (for example: https://github.com/bao-project/bao-demos/blob/master/demos/virtio/setup/config-virtio-demo-vm2.yaml) are used to configure the VirtIO backend devices themselves. In these files, we specify the type of VirtIO device (e.g., virtio-net, virtio-block, virtio-console), the device ID (which must match the one defined in the Device Tree), the MMIO address range, and other device-specific parameters.

Regarding the virtio-block device, it is a VirtIO device that provides a virtual block storage interface to the guest. In practice, it behaves like a disk device from the guest OS perspective. This allows the guest (in this case Linux) to access a block device backed by a file or storage resource (eMMC or SD Card) on the host side, which can then be used to mount a filesystem and store persistent data.

In our demos, the purpose of the virtio-block device is simply to allow the Linux guest to have access to non-volatile storage (e.g., a root filesystem or additional disk image). It is not strictly required for the Remote I/O mechanism itself. Remote I/O only provides a way to forward requests and notifications between the frontend and backend devices.

With this in mind, depending on the specific demo or setup you are using, the virtio-block device may therefore be optional. If your experiment does not require the guest to access a filesystem or persistent storage, you can typically remove it and only define the devices you want.

If you’d like, feel free to share the demo or configuration you are working with, and I can help check whether the block device is needed in that particular case.

[JLValdivieso]

Thanks again for your quick response.

I am currently working with the qemu-riscv64-virt demo and am now replicating the work on a RISC-V platform. Now that I understand the purpose of virtio-block, I believe I can skip its implementation for my specific use case, due to I only need to use the network communication between VMs.

[JLValdivieso]

Hi @joaopeixoto13,

I wanted to share an update on my progress with the Remote I/O implementation. I have successfully integrated it on a dual-core RISCV platform, running Linux on each core. After several debugging sessions, the setup is now working as expected.

At this point, I have a follow-up question. Have you ever explored using Zephyr as a frontend while leveraging Remote I/O to share the same Ethernet interface with Linux? I believe this approach could be relevant for mixed-criticality systems. In particular, it might help mitigate the latency introduced when copying data on the Linux side via SWIOTLB (in my case, I observed up to a 10× increase in latency when using the Ethernet interface).

My idea is to offload more time-critical tasks to Zephyr while maintaining Linux for less critical workloads. I plan to experiment with this approach in my research, and I was wondering if you have come across similar work or have any insights on this setup.

Best regards,

[joaopeixoto13]

Hi @JLValdivieso

Glad to hear you’ve got the setup fully working!

Regarding the latency you observed with SWIOTLB on Linux, this is something we’ve already looked into. We actually have some interesting results using TSN on Linux that help mitigate part of this overhead. If you’re curious, you can check out this paper.

On your idea of offloading more time-critical tasks to Zephyr, that makes a lot of sense. If the goal is to achieve more deterministic and lower-latency networking, moving in that direction is a solid approach. Architecturally, this would imply shifting the VirtIO backend from Linux to Zephyr, so that Zephyr owns the fast path while Linux operates through a virtualized interface.

We already have some related work around VirtIO in a baremetal context that might be useful as a reference.
Even though this example focuses on a console frontend, it includes useful abstractions you can reuse, such as Virtqueues, MMIO interface, and memory management.

Overall, this is a very interesting direction and aligns well with Bao's mixed-criticality system design. It would be great to see how your experiments evolve, so feel free to move forward with this approach, and don’t hesitate to reach out along the way. We’re happy to support you whenever questions come up.