virtio: add virtio framework in NuttX #9297

CV-Bowen · 2023-05-16T16:04:20Z

Summary

This PR add the basic virtio framework for NuttX. Implement the virtio-mmio transport layer and four basic virtio-drivers.
And the new virtio-mmio transport layer support both leagcy and new version.

virtio devics/drivers match and probe/remove mechanism;
virtio mmio transport layer based on OpenAmp;
virtio-serial driver based on new virtio framework;
virtio-rng driver based on new virtio framework;
virtio-net driver based on new virtio framework (IOB Offload implementation);
virtio-blk driver based on new virtio framework;
Remove the old virtio mmio framework, the old framework only support mmio transport layer, and the new framwork support more transport layer and this commit has implemented all the old virtio drivers;
Refresh the the qemu-arm64 and qemu-riscv virtio related configs, and update its README.txt;

Compared with previous virtio-mmio-net:

|                        | master/-c | master/-s | this/-c | this/-s |
| :--------------------: | :-------: | :-------: | :-----: | :-----: |
| qemu-armv8a:netnsh     |  539Mbps  |  524Mbps  | 906Mbps | 715Mbps |
| qemu-armv8a:netnsh_smp |  401Mbps  |  437Mbps  | 583Mbps | 505Mbps |
| rv-virt:netnsh         |  487Mbps  |  512Mbps  | 760Mbps | 634Mbps |
| rv-virt:netnsh_smp     |  387Mbps  |  455Mbps  | 447Mbps | 502Mbps |
| rv-virt:netnsh64       |  602Mbps  |  595Mbps  | 881Mbps | 769Mbps |
| rv-virt:netnsh64_smp   |  414Mbps  |  515Mbps  | 491Mbps | 525Mbps |
| rv-virt:knetnsh64      |  515Mbps  |  457Mbps  | 606Mbps | 540Mbps |
| rv-virt:knetnsh64_smp  |  308Mbps  |  389Mbps  | 415Mbps | 474Mbps |

Note: Both CONFIG_IOB_NBUFFERS=64, using iperf command, all in Mbits/sec
Tested in QEMU 7.2.2

Impact

The qemu-arm64 and qemu-rv board

Testing

Local test with qmeu-arm64, qemu-rv board

test virtio-net with iperf, ping, telnet;
test virtio-blk with dd command;
test virtio-rng with rand example;
test virtio-serial with linux telnet;

btashton · 2023-05-17T05:10:57Z

I am very interested in this PR, but do not let my review hold this up since I likely wont have time to give it a detailed review for a few days.

masayuki2009 · 2023-05-17T13:51:57Z

@CV-Bowen @xiaoxiang781216

Remove the old virtio mmio framework, the old framework only support mmio transport layer, and the new framwork support more transport layer and this commit has implemented all the old virtio drivers;

I'd like to keep my old drivers for my study. So, I propose adding a config option to select which driver to use.

CV-Bowen · 2023-05-27T10:57:54Z

#9406 has been merged, rebase to last.

xiaoxiang781216 · 2023-05-27T17:26:11Z

@CV-Bowen @xiaoxiang781216

Remove the old virtio mmio framework, the old framework only support mmio transport layer, and the new framwork support more transport layer and this commit has implemented all the old virtio drivers;

I'd like to keep my old drivers for my study. So, I propose adding a config option to select which driver to use.

@masayuki2009 here is some my thought:

Since the public interface is changed, keeping both implementations complicate the initialization of board which use virtio and confuse the developer who doesn't familiar with the history.
The new implementation has more generic design and better performance, we can improve the implementation and support more virtio devices together.
We can always restore your version for studying and comparing at any time with git help.

masayuki2009 · 2023-05-29T00:09:20Z

@CV-Bowen @xiaoxiang781216

Hmm, as long as I tried this PR rv-virt:netnsh64 seems OK but at least rv-virt:knetnsh64, rv-virt:knetnsh64_smp and qemu-armv8a:netnsh_smp have performance issues with iperf.

xiaoxiang781216 · 2023-05-29T06:40:56Z

@CV-Bowen @xiaoxiang781216

Hmm, as long as I tried this PR rv-virt:netnsh64 seems OK but at least rv-virt:knetnsh64, rv-virt:knetnsh64_smp and qemu-armv8a:netnsh_smp have performance issues with iperf.

@wengzhe will take a look. Could you share your perf number on qemu-armv8a:netnsh_smp?

masayuki2009 · 2023-05-29T07:02:03Z

@wengzhe will take a look. Could you share your perf number on qemu-armv8a:netnsh_smp?

+ /home/ishikawa/opensource/QEMU/qemu-7.1/build/qemu-system-aarch64 -cpu cortex-a53 -smp 4 -nographic -machine virt,virtualization=on,gic-version=3 -chardev stdio,id=con,mux=on -serial chardev:con -global virtio-mmio.force-legacy=false -netdev user,id=u1,hostfwd=tcp:127.0.0.1:10023-10.0.2.15:23,hostfwd=tcp:127.0.0.1:15001-10.0.2.15:5001 -device virtio-net-device,netdev=u1,bus=virtio-mmio-bus.0 -mon chardev=con,mode=readline -kernel ./nuttx
telnetd [7:100]

NuttShell (NSH) NuttX-10.2.0
nsh> uname -a
NuttX  10.2.0 b3d2a32e1d May 29 2023 15:55:41 arm64 qemu-armv8a
nsh> ps
  PID GROUP CPU PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK           STACK   USED  FILLED COMMAND
    0     0   0   0 FIFO     Kthread N-- Assigned           0000000000000000 008144 000976  11.9%  CPU0 IDLE
    1     1   1   0 FIFO     Kthread N-- Running            0000000000000000 008144 000320   3.9%  CPU1 IDLE
    2     2   2   0 FIFO     Kthread N-- Running            0000000000000000 008144 000320   3.9%  CPU2 IDLE
    3     3   3   0 FIFO     Kthread N-- Running            0000000000000000 008144 000080   0.9%  CPU3 IDLE
    4     4 --- 192 RR       Kthread --- Waiting  Semaphore 0000000000000000 008096 000512   6.3%  hpwork 0x402da9e8
    5     5 --- 100 RR       Kthread --- Waiting  Semaphore 0000000000000000 008096 000544   6.7%  lpwork 0x402da9b8
    6     6   0 100 RR       Task    --- Running            0000000000000000 008112 002448  30.1%  nsh_main
    7     7 --- 100 RR       Task    --- Waiting  Semaphore 0000000000020000 008112 001296  15.9%  telnetd
nsh> iperf -c 43.31.76.242 -i 1 -t 5
     IP: 10.0.2.15

 mode=tcp-client sip=10.0.2.15:5001,dip=43.31.76.242:5001, interval=1, time=5

           Interval         Transfer         Bandwidth

   0.00-   1.01 sec   30326784 Bytes  240.21 Mbits/sec
   1.01-   2.02 sec          0 Bytes    0.00 Mbits/sec
   2.02-   3.03 sec          0 Bytes    0.00 Mbits/sec
   3.03-   4.04 sec          0 Bytes    0.00 Mbits/sec
   4.04-   5.05 sec          0 Bytes    0.00 Mbits/sec
   0.00-   5.05 sec   30326784 Bytes   48.04 Mbits/sec

CV-Bowen · 2023-06-04T08:57:43Z

@masayuki2009 Thanks, we figure out the reason and update the patch, new test results are also updated, could you take a look?

arch/arm64/src/qemu/Make.defs

arch/risc-v/src/qemu-rv/Make.defs

drivers/virtio/virtio-serial.c

masayuki2009 · 2023-06-04T23:53:18Z

@masayuki2009 Thanks, we figure out the reason and update the patch, new test results are also updated, could you take a look?

@CV-Bowen
Hmm, rv-virt:knetnsh64 and rv-virt:knetnsh64_smp still have problems.

CV-Bowen · 2023-06-05T03:00:46Z

@masayuki2009 Sorry, I'm not familiar with kernal mode, I see there is no iperf command after starting the rv-virt:knetnsh64 (Follow the README.txt).
Could you show me more detail steps to run the built-in command in kernal mode.

NuttShell (NSH) NuttX-12.1.0
nsh> 
nsh> 
nsh> 
nsh> ?
help usage:  help [-v] [<cmd>]

    .         break     df        free      kill      nslookup  set       uname     
    [         cat       dmesg     get       ls        printf    sleep     umount    
    ?         cd        echo      help      mkdir     ps        source    unset     
    alias     cp        env       hexdump   mkrd      put       test      uptime    
    unalias   cmp       exec      ifconfig  mount     pwd       time      usleep    
    arp       dirname   exit      ifdown    mv        rm        true      wget      
    basename  dd        false     ifup      nfsmount  rmdir     truncate  xd        
nsh>

masayuki2009 · 2023-06-05T03:46:24Z

@masayuki2009 Sorry, I'm not familiar with kernal mode, I see there is no iperf command after starting the rv-virt:knetnsh64 (Follow the README.txt).
Could you show me more detail steps to run the built-in command in kernal mode.

@CV-Bowen

In the CONFIG_BUILD_KERNEL mode, iperf command is built as a separate ELF file so the command can not be seen in the help.

nsh> ps
  PID GROUP PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK           STACK   USED  FILLED COMMAND
    0     0   0 FIFO     Kthread N-- Ready              0000000000000000 003024 001344  44.4%  Idle Task
    1     1 224 RR       Kthread --- Waiting  Semaphore 0000000000000000 001952 000408  20.9%  hpwork 0x802017d8
    2     2 100 RR       Kthread --- Waiting  Semaphore 0000000000000000 001952 000408  20.9%  lpwork 0x80201808
    3     3 100 RR       Task    --- Running            0000000000000000 002976 001720  57.7%  /system/bin/init
    4     4 100 RR       Task    --- Waiting  Semaphore 0000000000020000 001968 000648  32.9%  telnetd
nsh> free
                   total       used       free    largest  nused  nfree
        Kmem:    2023776     295840    1727936    1712112     93      8
        Page:    4194304    1277952    2916352    2916352
nsh> help
help usage:  help [-v] [<cmd>]

    .         break     df        free      kill      nslookup  set       uname
    [         cat       dmesg     get       ls        printf    sleep     umount
    ?         cd        echo      help      mkdir     ps        source    unset
    alias     cp        env       hexdump   mkrd      put       test      uptime
    unalias   cmp       exec      ifconfig  mount     pwd       time      usleep
    arp       dirname   exit      ifdown    mv        rm        true      wget
    basename  dd        false     ifup      nfsmount  rmdir     truncate  xd
nsh> iperf
ERROR: should specific client/server mode
USAGE: iperf [-sua] [-c <ip|cpu>] [-p <port>] [-i <interval>] [-t <time>] [--local <path>] [--rpmsg <name>]
iperf command:
  -c, --client=<ip>    run in client mode, connecting to <host>
  -s, --server         run in server mode
  -u, --udp            use UDP rather than TCP
  --local=[<path>]     use local socket
  --rpmsg=<name>       use RPMsg socket
  -p, --port=<port>    server port to listen on/connect to
  -i, --interval=<interval> seconds between periodic bandwidth reports
  -t, --time=<time>    time in seconds to transmit for (default 10 secs)
  -a, --abort          abort running iperf
nsh>

CV-Bowen · 2023-06-07T08:24:24Z

@masayuki2009 I use the wrong virtio mmio intterupt number in S-mode in nuttx/boards/risc-v/qemu-rv/rv-virt/src/qemu_rv_appinit.c, this leads some problems in rv-virt:knetnsh64 and rv-virt:knetnsh64_smp and has been fixed now. Could take a look again.

masayuki2009 · 2023-06-08T01:16:10Z

@masayuki2009 Thanks, we figure out the reason and update the patch, new test results are also updated, could you take a look?

@CV-Bowen
Now rv-virt:knetnsh64_smp works but it seems that the performance is not good compared to the latest master.

With this PR,

NuttShell (NSH) NuttX-10.2.0
nsh> uname -a
NuttX  10.2.0 73ba318ed8 Jun  8 2023 09:37:54 risc-v rv-virt
nsh> ps
  PID GROUP CPU PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK           STACK   USED  FILLED COMMAND
    0     0   0   0 FIFO     Kthread N-- Assigned           0000000000000000 003024 001592  52.6%  CPU0 IDLE
    1     1   1   0 FIFO     Kthread N-- Running            0000000000000000 003024 000584  19.3%  CPU1 IDLE
    2     2   2   0 FIFO     Kthread N-- Running            0000000000000000 003024 000584  19.3%  CPU2 IDLE
    3     3   3   0 FIFO     Kthread N-- Running            0000000000000000 003024 000584  19.3%  CPU3 IDLE
    4     4 --- 224 RR       Kthread --- Waiting  Semaphore 0000000000000000 001952 000408  20.9%  hpwork 0x80205988
    5     5 --- 100 RR       Kthread --- Waiting  Semaphore 0000000000000000 001952 000408  20.9%  lpwork 0x802059b8
    6     6   0 100 RR       Task    --- Running            0000000000000000 002976 001720  57.7%  /system/bin/init
    7     7 --- 100 RR       Task    --- Waiting  Semaphore 0000000000020000 001968 000648  32.9%  telnetd
nsh> free
                   total       used       free    largest  nused  nfree
        Kmem:    1945936     115968    1829968    1826800     83      5
        Page:    4194304    1277952    2916352    2916352
nsh> iperf -s -i 1
     IP: 10.0.2.15

 mode=tcp-server sip=10.0.2.15:5001,dip=0.0.0.0:5001, interval=1, time=0
accept: 10.0.2.2:42966

           Interval         Transfer         Bandwidth

   0.00-   1.01 sec   21740920 Bytes  172.21 Mbits/sec
   1.01-   2.02 sec   17514024 Bytes  138.72 Mbits/sec
   2.02-   3.03 sec   17508692 Bytes  138.68 Mbits/sec
   3.03-   4.04 sec   17411632 Bytes  137.91 Mbits/sec
   4.04-   5.05 sec   17481728 Bytes  138.47 Mbits/sec

With the latest master,

NuttShell (NSH) NuttX-12.0.0
nsh> uname -a
NuttX  12.0.0 b89215a11f Jun  8 2023 09:46:10 risc-v rv-virt
nsh> ps
  PID GROUP CPU PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK           STACK   USED  FILLED COMMAND
    0     0   0   0 FIFO     Kthread N-- Assigned           0000000000000000 003024 001592  52.6%  CPU0 IDLE
    1     1   1   0 FIFO     Kthread N-- Running            0000000000000000 003024 000584  19.3%  CPU1 IDLE
    2     2   2   0 FIFO     Kthread N-- Running            0000000000000000 003024 000584  19.3%  CPU2 IDLE
    3     3   3   0 FIFO     Kthread N-- Running            0000000000000000 003024 000584  19.3%  CPU3 IDLE
    4     4 --- 224 RR       Kthread --- Waiting  Semaphore 0000000000000000 001952 000408  20.9%  hpwork 0x80205778
    5     5 --- 100 RR       Kthread --- Waiting  Semaphore 0000000000000000 001952 000408  20.9%  lpwork 0x802057a8
    6     6   0 100 RR       Task    --- Running            0000000000000000 002976 001720  57.7%  /system/bin/init
    7     7 --- 100 RR       Task    --- Waiting  Semaphore 0000000000020000 001968 000648  32.9%  telnetd
nsh> free
                   total       used       free    largest  nused  nfree
        Kmem:    1945936      38496    1907440    1906672     92      3
        Page:    4194304    1277952    2916352    2916352
nsh> iperf -s -i 1
     IP: 10.0.2.15

 mode=tcp-server sip=10.0.2.15:5001,dip=0.0.0.0:5001, interval=1, time=0
accept: 10.0.2.2:49160

           Interval         Transfer         Bandwidth

   0.00-   1.02 sec   25213764 Bytes  197.76 Mbits/sec
   1.02-   2.03 sec   25088640 Bytes  198.72 Mbits/sec
   2.03-   3.04 sec   25368960 Bytes  200.94 Mbits/sec
   3.04-   4.05 sec   25217120 Bytes  199.74 Mbits/sec
   4.05-   5.06 sec   25193760 Bytes  199.55 Mbits/sec

masayuki2009 · 2023-07-12T10:45:26Z

@CV-Bowen
I tried 3116245 with QEMU-7.2.4 instead of QEMU-7.1.1 but I still have performance issues with rv-virt:netnsh_smp and rv-virt:netnsh64_smp. Could you tell me what compiler you are using for RISC-V?

CV-Bowen · 2023-07-12T10:50:19Z

@CV-Bowen
I tried 3116245 with QEMU-7.2.4 instead of QEMU-7.1.1 but I still have performance issues with rv-virt:netnsh_smp and rv-virt:netnsh64_smp. Could you tell me what compiler you are using for RISC-V?

@wengzhe Could you answer this question?

wengzhe · 2023-07-12T13:03:15Z

@CV-Bowen
I tried 3116245 with QEMU-7.2.4 instead of QEMU-7.1.1 but I still have performance issues with rv-virt:netnsh_smp and rv-virt:netnsh64_smp. Could you tell me what compiler you are using for RISC-V?

@wengzhe Could you answer this question?

Bandwidth data from us are normally collected from my PC:

Host: Ubuntu 22.04.1 x86_64 with Core i7 12700
QEMU: 7.2.2 compiled by gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0

> riscv64-unknown-elf-gcc --version
riscv64-unknown-elf-gcc () 10.2.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

> aarch64-none-elf-gcc --version
aarch64-none-elf-gcc (GNU Toolchain for the Arm Architecture 11.2-2022.02 (arm-11.14)) 11.2.1 20220111
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Note:

The riscv64 gcc 10.2.0 is from apt: gcc-riscv64-unknown-elf/jammy,now 10.2.0-0ubuntu1 amd64
It seems that rv-virt:netnsh_smp is also compiled by riscv64 gcc

CV-Bowen · 2023-07-30T07:08:30Z

@masayuki2009 I found the root cause of error: kvm run failed Function not implemented and it has been fixed now.
The root cause is the non-aligned copy from the virtio mmio config region, which does not match the standard:
Virtio 1.0 ~ Virtio 1.2 Spec 4.2.2.2 Driver Requirements: MMIO Device Register Layout

The driver MUST only use 32 bit wide and aligned reads and writes to access the control registers described
in table 4.1. For the device-specific configuration space, the driver MUST use 8 bit wide accesses for 8 bit
wide fields, 16 bit wide and aligned accesses for 16 bit wide fields and 32 bit wide and aligned accesses for
32 and 64 bit wide fields.

Follow Linux does, I change the virtio mmio read/write config to aligned copy and add macro virtio_read_config_member() and virtio_write_config_member() to directly read/write the config member. Could you have a look?

drivers/virtio/virtio-mmio.c

masayuki2009 · 2023-07-31T00:33:37Z

@masayuki2009 I found the root cause of error: kvm run failed Function not implemented and it has been fixed now.

@CV-Bowen
I confirmed the issue has been fixed.
However, I noticed that iperf performance with qemu-armv8a:netnsh is not good as before.
Could you confirm the latest performance and update commit messages as well?

CV-Bowen · 2023-08-02T13:41:11Z

@masayuki2009 The iperf performance is related to the compiler version.

GCC 12.3

https://developer.arm.com/-/media/Files/downloads/gnu/12.3.rel1/binrel/arm-gnu-toolchain-12.3.rel1-x86_64-arm-none-linux-gnueabihf.tar.xz
Master

wangbowen@ubuntu:/data/project/code/apache$ ~/Software/qemu/qemu-7.2.4/build/qemu-system-aarch64 -cpu cortex-a53 -nographic \
> -machine virt,virtualization=on,gic-version=3 \
> -chardev stdio,id=con,mux=on -serial chardev:con \
> -global virtio-mmio.force-legacy=false \
> -netdev user,id=u1,hostfwd=tcp:127.0.0.1:10023-10.0.2.15:23,hostfwd=tcp:127.0.0.1:15001-10.0.2.15:5001,hostfwd=udp:127.0.0.1:15001-10.0.2.15:5001 \
> -device virtio-net-device,netdev=u1,bus=virtio-mmio-bus.0 \
> -mon chardev=con,mode=readline -kernel ./arm64_netnsh_master_gcc12.3 
telnetd [4:100]

NuttShell (NSH) NuttX-12.2.1
nsh> 
nsh> 
nsh> iperf -s -i 1
     IP: 10.0.2.15

 mode=tcp-server sip=10.0.2.15:5001,dip=0.0.0.0:5001, interval=1, time=0
accept: 10.0.2.2:58472

           Interval         Transfer         Bandwidth

   0.00-   1.01 sec   43075840 Bytes  341.19 Mbits/sec
   1.01-   2.02 sec   43239360 Bytes  342.49 Mbits/sec
   2.02-   3.03 sec   43391200 Bytes  343.69 Mbits/sec
   3.03-   4.04 sec   43297760 Bytes  342.95 Mbits/sec
   4.04-   5.05 sec   43309440 Bytes  343.05 Mbits/sec
closed by the peer: 10.0.2.2:58472
iperf exit
nsh> QEMU: Terminated

This PR

wangbowen@ubuntu:/data/project/code/apache$ ~/Software/qemu/qemu-7.2.4/build/qemu-system-aarch64 -cpu cortex-a53 -nographic -machine virt,virtualization=on,gic-version=3 -chardev stdio,id=con,mux=on -serial chardev:con -global virtio-mmio.force-legacy=false -netdev user,id=u1,hostfwd=tcp:127.0.0.1:10023-10.0.2.15:23,hostfwd=tcp:127.0.0.1:15001-10.0.2.15:5001,hostfwd=udp:127.0.0.1:15001-10.0.2.15:5001 -device virtio-net-device,netdev=u1,bus=virtio-mmio-bus.0 -mon chardev=con,mode=readline -kernel ./arm64_netnsh_gcc12.3 
telnetd [4:100]

NuttShell (NSH) NuttX-12.2.1
nsh> iperf -s -i 1
     IP: 10.0.2.15

 mode=tcp-server sip=10.0.2.15:5001,dip=0.0.0.0:5001, interval=1, time=0
accept: 10.0.2.2:59352

           Interval         Transfer         Bandwidth

   0.00-   1.01 sec   53257880 Bytes  421.84 Mbits/sec
   1.01-   2.02 sec   53728000 Bytes  425.57 Mbits/sec
   2.02-   3.03 sec   53996640 Bytes  427.70 Mbits/sec
   3.03-   4.04 sec   53929480 Bytes  427.16 Mbits/sec
   4.04-   5.05 sec   52216900 Bytes  413.60 Mbits/sec
closed by the peer: 10.0.2.2:59352
iperf exit
nsh> QEMU: Terminated
wangbowen@ubuntu:/data/project/code/apache$ aarch64-none-elf-gcc --version
aarch64-none-elf-gcc (Arm GNU Toolchain 12.3.Rel1 (Build arm-12.35)) 12.3.1 20230626
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

wangbowen@ubuntu:/data/project/code/apache$

GCC 12.2

https://developer.arm.com/-/media/Files/downloads/gnu/12.2.rel1/binrel/arm-gnu-toolchain-12.2.rel1-x86_64-aarch64-none-elf.tar.xz
Master

wangbowen@ubuntu:/data/project/code/apache$ ~/Software/qemu/qemu-7.2.4/build/qemu-system-aarch64 -cpu cortex-a53 -nographic \
> -machine virt,virtualization=on,gic-version=3 \
> -chardev stdio,id=con,mux=on -serial chardev:con \
> -global virtio-mmio.force-legacy=false \
> -netdev user,id=u1,hostfwd=tcp:127.0.0.1:10023-10.0.2.15:23,hostfwd=tcp:127.0.0.1:15001-10.0.2.15:5001,hostfwd=udp:127.0.0.1:15001-10.0.2.15:5001 \
> -device virtio-net-device,netdev=u1,bus=virtio-mmio-bus.0 \
> -mon chardev=con,mode=readline -kernel ./arm64_netnsh_master_gcc12.2
telnetd [4:100]

NuttShell (NSH) NuttX-12.2.1
nsh> 
nsh> iperf -s -i 1
     IP: 10.0.2.15

 mode=tcp-server sip=10.0.2.15:5001,dip=0.0.0.0:5001, interval=1, time=0
accept: 10.0.2.2:43158

           Interval         Transfer         Bandwidth

   0.00-   1.01 sec   40880000 Bytes  323.80 Mbits/sec
   1.01-   2.02 sec   40693120 Bytes  322.32 Mbits/sec
   2.02-   3.03 sec   40517920 Bytes  320.93 Mbits/sec
   3.03-   4.04 sec   40482880 Bytes  320.66 Mbits/sec
   4.04-   5.05 sec   41136960 Bytes  325.84 Mbits/sec
closed by the peer: 10.0.2.2:43158
iperf exit
nsh> QEMU: Terminated

This PR

wangbowen@ubuntu:/data/project/code/apache$ ~/Software/qemu/qemu-7.2.4/build/qemu-system-aarch64 -cpu cortex-a53 -nographic -machine virt,virtualization=on,gic-version=3 -chardev stdio,id=con,mux=on -serial chardev:con -global virtio-mmio.force-legacy=false -netdev user,id=u1,hostfwd=tcp:127.0.0.1:10023-10.0.2.15:23,hostfwd=tcp:127.0.0.1:15001-10.0.2.15:5001,hostfwd=udp:127.0.0.1:15001-10.0.2.15:5001 -device virtio-net-device,netdev=u1,bus=virtio-mmio-bus.0 -mon chardev=con,mode=readline -kernel ./arm64_netnsh_gcc12.2
telnetd [4:100]

NuttShell (NSH) NuttX-12.2.1
nsh> iperf -s -i 1
     IP: 10.0.2.15

 mode=tcp-server sip=10.0.2.15:5001,dip=0.0.0.0:5001, interval=1, time=0
accept: 10.0.2.2:55610

           Interval         Transfer         Bandwidth

   0.00-   1.01 sec   52955660 Bytes  419.45 Mbits/sec
   1.01-   2.02 sec   52149740 Bytes  413.07 Mbits/sec
   2.02-   3.03 sec   51813940 Bytes  410.41 Mbits/sec
   3.03-   4.04 sec   51881100 Bytes  410.94 Mbits/sec
   4.04-   5.05 sec   52049000 Bytes  412.27 Mbits/sec
closed by the peer: 10.0.2.2:55610
iperf exit
nsh> QEMU: Terminated
wangbowen@ubuntu:/data/project/code/apache$ aarch64-none-elf-
aarch64-none-elf-addr2line      aarch64-none-elf-gcc            aarch64-none-elf-gcov-tool      aarch64-none-elf-nm
aarch64-none-elf-ar             aarch64-none-elf-gcc-11.2.1     aarch64-none-elf-gdb            aarch64-none-elf-objcopy
aarch64-none-elf-as             aarch64-none-elf-gcc-12.2.1     aarch64-none-elf-gdb-add-index  aarch64-none-elf-objdump
aarch64-none-elf-c++            aarch64-none-elf-gcc-ar         aarch64-none-elf-gfortran       aarch64-none-elf-ranlib
aarch64-none-elf-c++filt        aarch64-none-elf-gcc-nm         aarch64-none-elf-gprof          aarch64-none-elf-readelf
aarch64-none-elf-cpp            aarch64-none-elf-gcc-ranlib     aarch64-none-elf-ld             aarch64-none-elf-size
aarch64-none-elf-elfedit        aarch64-none-elf-gcov           aarch64-none-elf-ld.bfd         aarch64-none-elf-strings
aarch64-none-elf-g++            aarch64-none-elf-gcov-dump      aarch64-none-elf-lto-dump       aarch64-none-elf-strip
wangbowen@ubuntu:/data/project/code/apache$ aarch64-none-elf-gcc -v
Using built-in specs.
COLLECT_GCC=aarch64-none-elf-gcc
COLLECT_LTO_WRAPPER=/home/wangbowen/Software/aarch64-gcc/bin/../libexec/gcc/aarch64-none-elf/12.2.1/lto-wrapper
Target: aarch64-none-elf
Configured with: /data/jenkins/workspace/GNU-toolchain/arm-12/src/gcc/configure --target=aarch64-none-elf --prefix=/data/jenkins/workspace/GNU-toolchain/arm-12/build-aarch64-none-elf/install --with-gmp=/data/jenkins/workspace/GNU-toolchain/arm-12/build-aarch64-none-elf/host-tools --with-mpfr=/data/jenkins/workspace/GNU-toolchain/arm-12/build-aarch64-none-elf/host-tools --with-mpc=/data/jenkins/workspace/GNU-toolchain/arm-12/build-aarch64-none-elf/host-tools --with-isl=/data/jenkins/workspace/GNU-toolchain/arm-12/build-aarch64-none-elf/host-tools --disable-shared --disable-nls --disable-threads --disable-tls --enable-checking=release --enable-languages=c,c++,fortran --with-newlib --with-gnu-as --with-gnu-ld --with-sysroot=/data/jenkins/workspace/GNU-toolchain/arm-12/build-aarch64-none-elf/install/aarch64-none-elf --with-pkgversion='Arm GNU Toolchain 12.2.Rel1 (Build arm-12.24)' --with-bugurl=https://bugs.linaro.org/
Thread model: single
Supported LTO compression algorithms: zlib
gcc version 12.2.1 20221205 (Arm GNU Toolchain 12.2.Rel1 (Build arm-12.24)) 
wangbowen@ubuntu:/data/project/code/apache$

GCC 11.2

https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-aarch64-none-elf.tar.xz
Master

wangbowen@ubuntu:/data/project/code/apache$ ~/Software/qemu/qemu-7.2.4/build/qemu-system-aarch64 -cpu cortex-a53 -nographic \
> -machine virt,virtualization=on,gic-version=3 \
> -chardev stdio,id=con,mux=on -serial chardev:con \
> -global virtio-mmio.force-legacy=false \
> -netdev user,id=u1,hostfwd=tcp:127.0.0.1:10023-10.0.2.15:23,hostfwd=tcp:127.0.0.1:15001-10.0.2.15:5001,hostfwd=udp:127.0.0.1:15001-10.0.2.15:5001 \
> -device virtio-net-device,netdev=u1,bus=virtio-mmio-bus.0 \
> -mon chardev=con,mode=readline -kernel ./arm64_netnsh_master_gcc11.2 
telnetd [4:100]

NuttShell (NSH) NuttX-12.2.1
nsh> 
nsh> iperf -s -i 1
     IP: 10.0.2.15

 mode=tcp-server sip=10.0.2.15:5001,dip=0.0.0.0:5001, interval=1, time=0
accept: 10.0.2.2:60002

           Interval         Transfer         Bandwidth

   0.00-   1.01 sec   37013920 Bytes  293.18 Mbits/sec
   1.01-   2.02 sec   38380480 Bytes  304.00 Mbits/sec
   2.02-   3.03 sec   38508960 Bytes  305.02 Mbits/sec
   3.03-   4.04 sec   38216960 Bytes  302.71 Mbits/sec
   4.04-   5.05 sec   38614080 Bytes  305.85 Mbits/sec
closed by the peer: 10.0.2.2:60002
iperf exit
nsh> QEMU: Terminated

This PR

wangbowen@ubuntu:/data/project/code/apache$ ~/Software/qemu/qemu-7.2.4/build/qemu-system-aarch64 -cpu cortex-a53 -nographic -machine virt,virtualization=on,gic-version=3 -chardev stdio,id=con,mux=on -serial chardev:con -global virtio-mmio.force-legacy=false -netdev user,id=u1,hostfwd=tcp:127.0.0.1:10023-10.0.2.15:23,hostfwd=tcp:127.0.0.1:15001-10.0.2.15:5001,hostfwd=udp:127.0.0.1:15001-10.0.2.15:5001 -device virtio-net-device,netdev=u1,bus=virtio-mmio-bus.0 -mon chardev=con,mode=readline -kernel ./arm64_netnsh_gcc11.2 
telnetd [4:100]

NuttShell (NSH) NuttX-12.2.1
nsh> iperf -s -i 1
     IP: 10.0.2.15

 mode=tcp-server sip=10.0.2.15:5001,dip=0.0.0.0:5001, interval=1, time=0
accept: 10.0.2.2:55980

           Interval         Transfer         Bandwidth

   0.00-   1.01 sec   46676200 Bytes  369.71 Mbits/sec
   1.01-   2.02 sec   46709780 Bytes  369.98 Mbits/sec
   2.02-   3.03 sec   46609040 Bytes  369.18 Mbits/sec
   3.03-   4.04 sec   46508300 Bytes  368.38 Mbits/sec
   4.04-   5.05 sec   47179900 Bytes  373.70 Mbits/sec
closed by the peer: 10.0.2.2:55980
iperf exit
nsh> QEMU: Terminated
wangbowen@ubuntu:/data/project/code/apache$ aarch64-none-elf-gcc -v
Using built-in specs.
COLLECT_GCC=aarch64-none-elf-gcc
COLLECT_LTO_WRAPPER=/data/software/gcc/gcc-arm-11.2-2022.02-x86_64-aarch64-none-elf/bin/../libexec/gcc/aarch64-none-elf/11.2.1/lto-wrapper
Target: aarch64-none-elf
Configured with: /data/jenkins/workspace/GNU-toolchain/arm-11/src/gcc/configure --target=aarch64-none-elf --prefix=/data/jenkins/workspace/GNU-toolchain/arm-11/build-aarch64-none-elf/install// --with-gmp=/data/jenkins/workspace/GNU-toolchain/arm-11/build-aarch64-none-elf/host-tools --with-mpfr=/data/jenkins/workspace/GNU-toolchain/arm-11/build-aarch64-none-elf/host-tools --with-mpc=/data/jenkins/workspace/GNU-toolchain/arm-11/build-aarch64-none-elf/host-tools --with-isl=/data/jenkins/workspace/GNU-toolchain/arm-11/build-aarch64-none-elf/host-tools --disable-shared --disable-nls --disable-threads --disable-tls --enable-checking=release --enable-languages=c,c++,fortran --with-newlib --with-pkgversion='GNU Toolchain for the Arm Architecture 11.2-2022.02 (arm-11.14)' --with-bugurl=https://bugs.linaro.org/
Thread model: single
Supported LTO compression algorithms: zlib
gcc version 11.2.1 20220111 (GNU Toolchain for the Arm Architecture 11.2-2022.02 (arm-11.14)) 
wangbowen@ubuntu:/data/project/code/apache$

xiaoxiang781216 · 2023-08-03T07:18:45Z

@masayuki2009 do you have more comment?

masayuki2009 · 2023-08-03T08:08:52Z

@masayuki2009 do you have more comment?

@xiaoxiang781216
Let me check the latest PR tomorrow.

masayuki2009 · 2023-08-04T08:57:29Z

@CV-Bowen
I noticed that iperf numbers with qemu-armv8a:netnsh were much improved this time.
However, the numbers with rv-virt:knetnsh64_smp are worse than before.

To compare your changes easily on the github whenever you modify your code and rebase this PR,
please

rebase the PR first and force push
then modify your PR and force push again

CV-Bowen · 2023-08-04T11:17:36Z

@masayuki2009 OK, but actually, I only modified the virtio mmio config read/write code, and not did changes to virtio-net or somethins else to improve the iperf performance.

CV-Bowen · 2023-08-08T03:07:54Z

@masayuki2009 I will update the iperf test data based on gcc 12.2 and qemu 7.2.4 with this PR, but the performance degradation in some configs may be fixed later.

CV-Bowen · 2023-08-08T03:09:12Z

Only rebase to the master.

xiaoxiang781216 · 2023-08-08T13:20:57Z

@masayuki2009 OK, but actually, I only modified the virtio mmio config read/write code, and not did changes to virtio-net or somethins else to improve the iperf performance.

@masayuki2009 how about your opinion? Most config run fast than before, only RISCV SMP in kernel mode is a little bit slow. Since QEMU is an emulation, the slow may due to the inefficient RISCV implementation, not reflect the performance on real device.

masayuki2009 · 2023-11-14T06:01:28Z

@CV-Bowen
I noticed that rv-virt with the new VirtIO framework does not work with QEMU-8.1.2.

CV-Bowen · 2023-11-14T06:15:47Z

@masayuki2009 OK, I will try QEMU-8.1.2 later today.

CV-Bowen · 2023-11-15T09:25:45Z

@masayuki2009 I tried the config rv-virtio netnsh and netnsh_smp in QEMU-8.1.2, netnsh work but config netnsh_smp stuck at an atomic instruction:

And both configs work well in QEMU-8.0.5. I will figure out what happened in QEMU-8.1.2.

masayuki2009 · 2023-11-15T12:27:32Z

@CV-Bowen
Can we use the RISC-V atomic operations for the I/O space?

CV-Bowen · 2023-11-16T03:45:30Z

@masayuki2009 I'm not sure, but I decide to change the metal_read/write operation to nuttx implemented instead openamp libmetal implemented. Nuttx implemented operation not use the atomic operation, and can fix this issue.

CV-Bowen marked this pull request as draft May 16, 2023 16:05

xiaoxiang781216 requested review from masayuki2009 and btashton May 16, 2023 16:57

This was referenced May 20, 2023

openamp: decoupling the transport layer and virtio device layer #9348

Merged

openamp: virtio.h add version field in device id table struct #9406

Merged

CV-Bowen force-pushed the virtio branch from 2b3e191 to 42d1fa0 Compare May 27, 2023 07:43

CV-Bowen marked this pull request as ready for review May 27, 2023 07:56

CV-Bowen force-pushed the virtio branch 2 times, most recently from 38dba38 to b3d2a32 Compare May 27, 2023 10:56

xiaoxiang781216 mentioned this pull request May 27, 2023

fdt: Add initial FDT support and procfs for userspace export #9204

Merged

CV-Bowen force-pushed the virtio branch 2 times, most recently from ee53de9 to 8e88378 Compare June 4, 2023 08:55

xiaoxiang781216 reviewed Jun 4, 2023

View reviewed changes

arch/arm64/src/qemu/Make.defs Show resolved Hide resolved

arch/risc-v/src/qemu-rv/Make.defs Show resolved Hide resolved

drivers/virtio/virtio-serial.c Outdated Show resolved Hide resolved

CV-Bowen force-pushed the virtio branch 2 times, most recently from a90ae82 to 73ba318 Compare June 7, 2023 08:20

wengzhe mentioned this pull request Jun 14, 2023

netdev/upper: Optimize on quota related operations #9543

Merged

CV-Bowen force-pushed the virtio branch from 73ba318 to cb67cb3 Compare June 17, 2023 05:42

CV-Bowen force-pushed the virtio branch 3 times, most recently from c37a50e to 1578f29 Compare July 30, 2023 06:55

xiaoxiang781216 reviewed Jul 30, 2023

View reviewed changes

drivers/virtio/virtio-mmio.c Outdated Show resolved Hide resolved

drivers/virtio/virtio-mmio.c Show resolved Hide resolved

drivers/virtio/virtio-mmio.c Show resolved Hide resolved

drivers/virtio/virtio-mmio.c Show resolved Hide resolved

drivers/virtio/virtio-mmio.c Show resolved Hide resolved

CV-Bowen force-pushed the virtio branch from 1578f29 to 296d00c Compare August 2, 2023 13:41

xiaoxiang781216 approved these changes Aug 2, 2023

View reviewed changes

jerpelea approved these changes Aug 2, 2023

View reviewed changes

CV-Bowen force-pushed the virtio branch from 296d00c to cf214d7 Compare August 8, 2023 03:08

xiaoxiang781216 merged commit 9aa57b6 into apache:master Aug 9, 2023
26 checks passed

jerpelea added this to To-Add in Release Notes - 12.3.0 Sep 26, 2023

jerpelea moved this from To-Add to done in Release Notes - 12.3.0 Oct 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

virtio: add virtio framework in NuttX #9297

virtio: add virtio framework in NuttX #9297

CV-Bowen commented May 16, 2023 •

edited

btashton commented May 17, 2023

masayuki2009 commented May 17, 2023 •

edited

CV-Bowen commented May 27, 2023

xiaoxiang781216 commented May 27, 2023

masayuki2009 commented May 29, 2023

xiaoxiang781216 commented May 29, 2023

masayuki2009 commented May 29, 2023

CV-Bowen commented Jun 4, 2023 •

edited

masayuki2009 commented Jun 4, 2023

CV-Bowen commented Jun 5, 2023

masayuki2009 commented Jun 5, 2023

CV-Bowen commented Jun 7, 2023

masayuki2009 commented Jun 8, 2023

masayuki2009 commented Jul 12, 2023

CV-Bowen commented Jul 12, 2023 •

edited

wengzhe commented Jul 12, 2023

CV-Bowen commented Jul 30, 2023 •

edited

masayuki2009 commented Jul 31, 2023

CV-Bowen commented Aug 2, 2023

xiaoxiang781216 commented Aug 3, 2023

masayuki2009 commented Aug 3, 2023

masayuki2009 commented Aug 4, 2023 •

edited

CV-Bowen commented Aug 4, 2023

CV-Bowen commented Aug 8, 2023

CV-Bowen commented Aug 8, 2023

xiaoxiang781216 commented Aug 8, 2023

masayuki2009 commented Nov 14, 2023

CV-Bowen commented Nov 14, 2023

CV-Bowen commented Nov 15, 2023

masayuki2009 commented Nov 15, 2023

CV-Bowen commented Nov 16, 2023

virtio: add virtio framework in NuttX #9297

virtio: add virtio framework in NuttX #9297

Conversation

CV-Bowen commented May 16, 2023 • edited

Summary

Impact

Testing

btashton commented May 17, 2023

masayuki2009 commented May 17, 2023 • edited

CV-Bowen commented May 27, 2023

xiaoxiang781216 commented May 27, 2023

masayuki2009 commented May 29, 2023

xiaoxiang781216 commented May 29, 2023

masayuki2009 commented May 29, 2023

CV-Bowen commented Jun 4, 2023 • edited

masayuki2009 commented Jun 4, 2023

CV-Bowen commented Jun 5, 2023

masayuki2009 commented Jun 5, 2023

CV-Bowen commented Jun 7, 2023

masayuki2009 commented Jun 8, 2023

masayuki2009 commented Jul 12, 2023

CV-Bowen commented Jul 12, 2023 • edited

wengzhe commented Jul 12, 2023

CV-Bowen commented Jul 30, 2023 • edited

masayuki2009 commented Jul 31, 2023

CV-Bowen commented Aug 2, 2023

GCC 12.3

GCC 12.2

GCC 11.2

xiaoxiang781216 commented Aug 3, 2023

masayuki2009 commented Aug 3, 2023

masayuki2009 commented Aug 4, 2023 • edited

CV-Bowen commented Aug 4, 2023

CV-Bowen commented Aug 8, 2023

CV-Bowen commented Aug 8, 2023

xiaoxiang781216 commented Aug 8, 2023

masayuki2009 commented Nov 14, 2023

CV-Bowen commented Nov 14, 2023

CV-Bowen commented Nov 15, 2023

masayuki2009 commented Nov 15, 2023

CV-Bowen commented Nov 16, 2023

CV-Bowen commented May 16, 2023 •

edited

masayuki2009 commented May 17, 2023 •

edited

CV-Bowen commented Jun 4, 2023 •

edited

CV-Bowen commented Jul 12, 2023 •

edited

CV-Bowen commented Jul 30, 2023 •

edited

masayuki2009 commented Aug 4, 2023 •

edited