[RISCV] Discuss opprtunities to extend OpenVINO with RISCV support

[vladimir-dudnik]

Hello OpenVINO team,

Thanks you guys for amazing work you are doing, we’d like to ask your opinion on opportunity to add OpenVINO support for RISCV CPU ISA to enable DL inference on this promising new architecture. As you probably know, RISCV committee has ratified some time ago vector extension v1.0 (kind of SIMD technology but vector length may vary at runtime). At the moment there are publicly available tools which enable development for this new arch (although, there is no CPU with RVV 1.0 hardware implementation available at the market yet but it is expected they may appear at this year or beginning of the next year). So, for now there is gcc and clang compilers capable to generate RVV 1.0 code and there is QEMU emulator, allowing functional testing of such code.

Also, as we can see, there is some effort in community to bring up inference capabilities on RISCV architectures (https://github.com/tum-ei-eda/muriscv-nn, https://github.com/rvboards/ncnn and there are might be others). We think having OpenVINO API available for RISCV platforms would be beneficial for both OpenVINO and community.

So, we have tried those compilers with OpenVINO code base and successfully build OpenVINO and CPU plugin (with minor cmake files adjustments to enable new toolchain). Obviously, when CPU plugin compiled for new CPU architecture, all x86 specific optimization turned off and only ngraph reference code is actually compiled. But this makes OpenVINO work on RISCV, we successfully passed OpenVINO conformance tests on RISCV QEMU (although performance of reference code is not good, expectable). That makes us think that ngraph reference code could be replaced one by one with RVV 1.0 optimized code, under conditional compilation, for example, allowing performance boost on RISCV CPUs.

What is you guys vision on this? Do you think it make sense to bring RISCV support to OpenVINO and if so, what should be the best way to implement this?
We also may need pay attention to the possibility for RISCV vendor specific extensions, in other words, RISCV committee ratified base RVV 1.0 ISA, which any RISCV CPU should be compatible with, and also allow vendor to extend RISCV ISA with some specific, optional extensions (for example DSP, or compressed instructions or whatever). It seems to support RISCV vendor adoption for OpenVINO we may need to think about some way to enable this kind of differentiation across CPU vendors. Something like specify not only 'CPU' as execution device, but also have ability to enable or disable some vendor extensions, to allow portability on application level through different RISCV implementations/platforms.

Would be interesting to hear back your opinion on this
Thanks

[yury-gorbachev]

Hi Vladimir,
Thank you for your interest. This feature would be definitely interesting for us.
Recently we have been discussing option of using same CPU plugin for ARM target (currently we have separate plugin) and it does look doable without major implementation showstoppers.

On technical level as part of CPU plugin there are C++ implementations (which are not ngraph reference, but actually used in production) and there are no x86 specific extensions used in them. Then there are also oneDNN primitives which are working through separate executor and those are x86 specific.
So, on a high level, what could be done is to add RISC-V specific executor that implements primitives that are not covered by c++ ones and also ensure that c++ primitives compile. Latter is very unlikely problematic. For former you can use some library from what you have referred in your original message, just wrap it properly. I think there was also csi-nn2 library that implements primitives for RISC-V. This is something that we see feasible to do for ARM with use of ARM compute, so we are exploring this direction.
One concern is that we do not want to have ifdefs all over plugin and would like to have some things separated in target specific folder. This is implementation detail and we can discuss how this could be done when you decide to actually implement this feature.
Overall, we are definitely open to support you and learn RISC-V specifics throughout the process.
Perhaps @dmitry-gorokhov & @ilya-lavrenov could comment more or refer to developer documentation that describes arch in details.

Thanks,
Yury

[vladimir-dudnik]

Completely agree that bunch of #ifdefs is not good choice, and also interested in what you say regarding moving different CPU specific code in one general CPU plugin, the idea looks quite logical and straightforward. It should allow to reuse a lot of common code and keep really HW specific implementation as separate components of that general CPU plugin.
So thinking on RISCV support it would be useful for us to keep look at this draft implementation (when it will be available in some branch) to avoid overlapping work or some mismatches.
Thanks and look forward to see how such general CPU plugin may look like,
Vladimir

[dmitry-gorokhov]

Hi @vladimir-dudnik

Just to add a little bit more details to Yury's message. What we are proposing for ARM is shown on the diagram below. The major idea is to extend existing modular structure of CPU plugin with new blocks required for ARM (and other HW architectures) support.
First extension point in the proposed design is specific device opset (dialect in MLIR terminology). We don't believe there will be a huge necessity to have specific device operations, most of them likely will be reused from Intel CPU opset. However we want to give full flexibility in terms of new architectures support while keeping required changes as isolated as possible. Separate device specific transformation component is introduced to support than point. We believe there will be more or less the same story as for opset: most of the transformation can be shared with existing Intel CPU transformation pipeline, while adding device specific ones is still an option.

Moving closer to kernel level we propose to extend existing Executor concept which is designed to separate an operation and its implementation abstractions. Each operation may have several executors and this is exactly the place there ARM/RISCV specifics can be encapsulated. It also assumes some base classes in hierarchy, like ACL executor designed to contain all specifics related to ARM compute library. Some can be applied for csi-nn2 library or any other.
Finally here we already can offer one of the major reuse points since 73 of 82 operations in CPU opset have optimized C++ implementations that can be cross-compiled for specific device mostly w/o any modifications. Given that new CPU architecture support can be bounded with rest operations implementation activity .

With that design we tried to maximize amount of logic that can be reused between the HW architectures. In this way all major plugin level features like dynamic shapes support, graph level optimizations, memory reuse optimization, opset coverage, graph serialization and others stays common.
We expect to provide draft of proposed design within the next couple of weeks, so you will able to check how it fits your expectations.
Any discussions and suggestions are very appreciated.

[vladimir-dudnik]

Hi @dmitry-gorokhov this is an excellent explanation of proposed architecture, thanks a lot, it really helpful, and I believe, should strenghten OpenVINO positioning as cross-platform inference API.
Regards,
Vladimir

[vladimir-dudnik]

@vladimir-dudnik @vinograd47 are you interested in participating in merge of ARM + Intel CPU plugin? Or just want to use the same approach for RISC-V? External help is very appreciated here

Hi @ilya-lavrenov for the first place we were looking for RISCV opportunities. We need some time to think whether and how much we can help with ARM executor.
How do you see in general third-party HW support in OpenVINO? With this proposal it seems should be possible to have support for any third-party CPU architecture by default, with using C++ implementation and particular CPU vendor (or community, if interested) can add specific optimizations, if necessary, by providing optimized executor within OpenVINO contrib module?

[ilya-lavrenov]

Exactly, C++ impls are generic and can be reused across all CPUs, while it's allowed to add backend specific kernels optimized with native vector or other instructions.
For RISC-V specifically, if developers want to add kernels from any backend library:

• RISC-V opset and transformations to better map to backend primitives
• RISC-V's backend executor
• Snippets and JIT code if applicable

[ilya-lavrenov]

Hi @vladimir-dudnik and @vinograd47
Here is a branch with Intel CPU plugin, which can be compiled for ARM, ARM64, RISCV (using this toolchain) and for WebAssembly.
We are going to merge it into master in mean time, so you can start looking at it.

We are considering integration of kernels from XNNPACK, because it supports kernels for WebAssembly to run OpenVINO in browsers, as well as on RISCV. Do you think for your purposes XNNPACK integration can also be useful?

[vinograd47]

Hi @ilya-lavrenov!

Here is a branch with Intel CPU plugin, which can be compiled for ARM, ARM64, RISCV (using this toolchain) and for WebAssembly.
We are going to merge it into master in mean time, so you can start looking at it.

Great news! We will take a look at new changes.

We are considering integration of kernels from XNNPACK, because it supports kernels for WebAssembly to run OpenVINO in browsers, as well as on RISCV. Do you think for your purposes XNNPACK integration can also be useful?

As we can see from quick look at this library, right now it has only initial support for RISC-V. It provides RISC-V optimizations for small set of primitimes (hswish, abs, neg, sqr), which are not very common in models. We will continue to track their progress on RISC-V, but for now it doesn't look so useful for RISC-V.