[microTVM][RISCV] Tensorization for conv_2d_nchw_int8 with RVV extension #14836

katebern-grovety · 2023-05-12T09:33:53Z

Dot product tensorization with RISC-V V-extension intrinsic for conv_2d_nchw_int8. Using llvm intrinsic.
Testing with AOT test runner on Spike.

tvm-bot · 2023-05-12T09:33:57Z

Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.

cc @alanmacd, @gromero, @leandron, @mehrdadh, @mkatanbaf _{See #10317 for details}

_{Generated by tvm-bot}

PhilippvK · 2023-07-25T11:21:47Z

python/tvm/topi/riscv_cpu/conv2d_int8.py

+
+    # 2) Check target.
+    current_target = target.Target.current(allow_none=False)
+    has_attr = "+v" in current_target.mattr


Could you please look up if your schedules would also be supported by the sub-extensions zve32x, zve32f, zve64x, zve64f, zve64d? If this is the case, please extend this check to be aware of these mattr flags.

PhilippvK · 2023-07-25T11:29:02Z

python/tvm/topi/riscv_cpu/tensor_intrin.py

+#ifdef __cplusplus
+extern "C"
+#endif
+int32_t dot_uint8_int8_int32_body(uint8_t* data, int8_t* kernel, int32_t* output) {{


Do you by any chance have some benchmark results on how this performs compared to the RVV autovectorizer which is provided by RISCV LLVM/GCC?

PhilippvK · 2023-07-25T11:33:24Z

tests/python/relay/aot/riscv.mk

@@ -0,0 +1,85 @@
+# Licensed to the Apache Software Foundation (ASF) under one


I tried to upstream a patch like this, before and if was rejected because of the AOTTestRunner being removed at some point a favor of the MicroTVM Project API interface. (See #12534 (comment))

I actually have MicroTVM Project API template for Spike for a long time now (and use it regulary). I still need to clean it up before upstreaming: https://github.com/PhilippvK/microtvm-spike-template

I added Project API template for RISC-V with Spike based on CRT

Thank you for coming up with that change. Would it be possible to split this up into a different PR with separate tests etc.?

That would make reviewing the individual changes much easier.

Yes, sure. I'll create a new PR for Project API

PhilippvK · 2023-07-25T11:33:50Z

tests/python/relay/aot/riscv.mk

+
+TARGET_CFLAGS = --target=riscv64-unknown-linux-gnu -march=rv64gcv -static
+RUNNER = spike
+RUNNER_OPT = --isa=rv64gcv $(shell which pk)


I would like to have support for rv32gc here as well.

PhilippvK · 2023-07-25T11:34:28Z

tests/python/relay/aot/riscv.mk

+RUNNER = spike
+RUNNER_OPT = --isa=rv64gcv $(shell which pk)
+
+PKG_CFLAGS = ${PKG_COMPILE_OPTS} ${TARGET_CFLAGS} -O2 \


Some reason why using -O2 instead of -O3 or -Os?

No, there was no reason

PhilippvK · 2023-07-25T11:35:10Z

tests/python/relay/aot/riscv.mk

+DMLC_CORE=$(TVM_ROOT)/3rdparty/dmlc-core
+TOOLCHAIN_PATH=$(shell dirname $(shell which riscv64-unknown-linux-gnu-gcc))/..
+
+CC = clang-16


Clang-16 and RISC-V GCC could both be supported here?

PhilippvK · 2023-07-25T11:37:34Z

python/tvm/topi/riscv_cpu/conv2d_int8.py

+    has_attr = "+v" in current_target.mattr
+    is_arch_support = "v" in current_target.arch[2:]
+    if not is_arch_support and "march" in current_target.attrs:
+        is_arch_support = "v" in current_target.attrs["march"]


I was not aware that -march= is actually exposed to the TVM target. This is great because it would also work with non-llvm TVM targets.

PhilippvK · 2023-07-25T11:46:54Z

python/tvm/topi/riscv_cpu/conv2d_int8.py

+                s[kernel_vec].reorder(oc_chunk, oh, ic_chunk, ow, ic_block, oc_block)
+                oc_bn = cfg["tile_oc"].size[-1]
+                if oc_bn > 1:
+                    s[kernel_vec].vectorize(oc_block)


You are adding LLVM-exclusive schedule primitives but only testing them on TVMs C-backends (with tir.disable_vectorize=1). We eventually need to also test if your vector intrinsics does interfere with the auto-vectorization going on in LLVM. Testing this should be possible without compiling and running it for Spike, since errors would already show up during tvm.build().

…2d_nchw_int8

PhilippvK · 2023-08-16T15:20:41Z

apps/microtvm/riscv/template_project/CMakeLists.txt.template

+target_include_directories(tvm_model PRIVATE ${CMAKE_SOURCE_DIR}/include crt_config crt/include)
+target_compile_options(tvm_model PRIVATE -Wno-error=unused-variable -Wno-error=missing-braces -Wno-error=unused-const-variable -Wno-unused-variable)
+set_target_properties(tvm_model PROPERTIES LINKER_LANGUAGE C)
+target_link_libraries(main PRIVATE tvm_model)


I think you would need to add m (aka. libmath) here to support Softmax layers.

I'll add it.

PhilippvK · 2023-08-16T15:22:48Z

apps/microtvm/riscv/template_project/microtvm_api_server.py

+        spike_args = ["spike", f"--isa={march}", os.path.join(toolchain_path, target, "bin/pk")]
+        self._proc = subprocess.Popen(
+            spike_args + [self.BUILD_TARGET],
+            stdin=subprocess.PIPE,


I was wondering if you also run into the issue with the broken STDIN support in recent version of spike mentioned here: riscv-software-src/riscv-isa-sim#1400

Are you using an version of spike which does not have that problem or did you find another workaround?

On the latest version of spike, when using instruction counting, I got an error
An illegal instruction was executed!
now I'm using spike version 1.1.0 and this error does not occur on it.

PhilippvK · 2023-08-16T15:25:25Z

apps/microtvm/riscv/template_project/CMakeLists.txt.template

+set(CMAKE_CXX_STANDARD 11)
+set(CMAKE_C_COMPILER "clang")
+set(CMAKE_CXX_COMPILER "clang++")
+set(FLAGS "--target=${TARGET} --sysroot=${TOOLCHAIN_PATH}/sysroot --gcc-toolchain=${TOOLCHAIN_PATH} -march=${MARCH} -static")


I think we should also expose -mabi to the user.

I'll add it.

PhilippvK · 2023-08-16T15:28:22Z

apps/microtvm/riscv/template_project/microtvm_api_server.py

+        toolchain_path = options.get("toolchain_path")
+        target = options.get("target")
+        march = options.get("march")
+        spike_args = ["spike", f"--isa={march}", os.path.join(toolchain_path, target, "bin/pk")]


The --isa argument of spike will not always be the same as the -march used with clang. I would prefer having isa exposed as project option (defaulting to use the same value as march) to overwrite it if required.

I'll add isa as project option.

PhilippvK · 2023-08-16T15:31:55Z

apps/microtvm/riscv/template_project/microtvm_api_server.py

+                    default="rv64gc",
+                    help="Sets the value of target architecture.",
+                ),
+            ],


To compare how this performs on different vector length, we should make it configurable either by exposing vlen and elen as project options or by adding a generic spike_extra_args option to add any user defined arguments to the cmdline. See here: https://github.com/PhilippvK/microtvm-spike-template/blob/main/template_project/microtvm_api_server.py#L136-L149

katebern-grovety mentioned this pull request Jul 25, 2023

[CI][Docker] Add riscv-gnu-toolchain to ci_riscv #15403

Open

PhilippvK reviewed Jul 25, 2023

View reviewed changes

katebern-grovety added 9 commits August 16, 2023 12:12

Dot product tensorization with RISC-V V-extension intrinsics for conv…

4f8b271

…2d_nchw_int8

lint fix

686a41a

lint fix

4127e9f

lint fix

fe62428

lint fix, update comments

9bb118c

fix clang version

61ee1c3

Add Project API for RISC-V with spike

d2445fc

Add select implementation test case

7b01321

Delete riscv.mk

f7fb425

katebern-grovety force-pushed the rvv-intrin-dot-prod-int8 branch from b6db5aa to f7fb425 Compare August 16, 2023 09:25

PhilippvK reviewed Aug 16, 2023

View reviewed changes

move Project API changes to new PR

fe48a1b

katebern-grovety force-pushed the rvv-intrin-dot-prod-int8 branch from f15d664 to fe48a1b Compare August 21, 2023 06:52

cbalint13 mentioned this pull request Oct 21, 2023

[LLVM][RUNTIME] Add optional LLVM ORCJIT runtime executor #15964

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[microTVM][RISCV] Tensorization for conv_2d_nchw_int8 with RVV extension #14836

[microTVM][RISCV] Tensorization for conv_2d_nchw_int8 with RVV extension #14836

katebern-grovety commented May 12, 2023

tvm-bot commented May 12, 2023

PhilippvK Jul 25, 2023

PhilippvK Jul 25, 2023

PhilippvK Jul 25, 2023

katebern-grovety Aug 16, 2023

PhilippvK Aug 16, 2023

katebern-grovety Aug 17, 2023 •

edited

PhilippvK Jul 25, 2023

PhilippvK Jul 25, 2023

katebern-grovety Aug 16, 2023

PhilippvK Jul 25, 2023

PhilippvK Jul 25, 2023

PhilippvK Jul 25, 2023

PhilippvK Aug 16, 2023

katebern-grovety Aug 17, 2023

PhilippvK Aug 16, 2023

katebern-grovety Aug 17, 2023

PhilippvK Aug 16, 2023

katebern-grovety Aug 17, 2023

PhilippvK Aug 16, 2023

katebern-grovety Aug 17, 2023

PhilippvK Aug 16, 2023

		@@ -0,0 +1,85 @@
		# Licensed to the Apache Software Foundation (ASF) under one

[microTVM][RISCV] Tensorization for conv_2d_nchw_int8 with RVV extension #14836

Are you sure you want to change the base?

[microTVM][RISCV] Tensorization for conv_2d_nchw_int8 with RVV extension #14836

Conversation

katebern-grovety commented May 12, 2023

tvm-bot commented May 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

katebern-grovety Aug 17, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

katebern-grovety Aug 17, 2023 •

edited