muti-core-boom-simutor can run multi-thread coremark?

[qqjinger]

Hi,
I have generated a SmallBoomConfig-debug with 2 cores configuration.At the same time, 1-thread coremark.riscv and 2-thread coremark.riscv are generated.
1-thread coremark.riscv can be run correctly on simulator SmallBoomConfig-debug and get right coremark. However, when 2-thread coremark.riscv run on SmallBoomConfig-debug , a error occurs, whose pc=fffffffffffffffe. Obviously, pc is out of control. So whether sonicboom does support run multi-thread riscv program？ Or should I modify boom code somewhere?

[jerryz123]

How are you running compiling and running multithreaded coremark? In general SonicBOOM supports multithreaded applications.

[qqjinger]

1. my method to compile multithread coremark
As we all known, gcc should use "-lpthread -DMULTITHREAD=N -DUSE_PTHREAD" to get multithreaded applications. But my gcc can not find -lpthread when adding this option. So I add two more option "--sysroot=riscv-tools-install/sysroot/lib/" and "-Wl,-rpath-link riscv-tools-install/sysroot/lib/" to specify the path of pthread lib and other multithread needed libraries. After these effort, the multithreaded coremark.riscv is generated. But there are still two warnning as below and I don't know whether it is normal.
2 warnning appeared in coremark compiling are as below:
../riscv64_multi_thread/core_portme.c: In function 'core_start_parallel':
../riscv64_multi_thread/core_portme.c:248:16: warning: implicit declaration of function 'pthread_create' [-Wimplicit-function-declaration]
248 | return (ee_u8)pthread_create(&(res->port.thread),NULL,iterate,(void *)res);
| ^~~~~~~~~~~~~~
../riscv64_multi_thread/core_portme.c: In function 'core_stop_parallel':
../riscv64_multi_thread/core_portme.c:252:16: warning: implicit declaration of function 'pthread_join'; did you mean 'pthread_atfork'? [-Wimplicit-function-declaration]
252 | return (ee_u8)pthread_join(res->port.thread,&retval);
| ^~~~~~~~~~~~
| pthread_atfork

2. my method to run multithread coremark
I run multithreaded coremark.riscv same as single-thread coremark.riscv, and the brief cmd is as below:
./simulator-chipyard-SmallBoomConfig-debug pk coremark.riscv 0x0 0x0 0x66 1

[jerryz123]

I don't believe pk supports pthreads. What does spike -p2 pk coremark.riscv 0x0 0x0 0x66 1 return

[qqjinger]

my info returned is as below:
1. 1thread-coremark
bbl loader
2K performance run parameters for coremark.
CoreMark Size : 666
Total ticks : 367544
Total time (secs): 0.000368
Coremark/Mhz : 2.720763
ERROR! Must execute for at least 10 secs for a valid result!
Iterations : 1
Compiler version : GCC9.2.0
Compiler flags : -O2
Memory location : Please put data memory location here
(e.g. code in flash, data on heap etc)
seedcrc : 0xe9f5
[0]crclist : 0xe714
[0]crcmatrix : 0x1fd7
[0]crcstate : 0x8e3a
[0]crcfinal : 0xe714
Errors detected

2. 2thread-coremark
   bbl loader
   z 0000000000000000 ra 000000000001220a sp 000000007f7e9980 gp 00000000000212b0
   tp 0000000000000000 t0 0000000000000000 t1 0000000000000008 t2 0000000000021690
   s0 0000000000000000 s1 000000007f7e9a90 a0 000000007f7e9a0c a1 0000000000000000
   a2 0000000000011484 a3 000000007f7e99a0 a4 0000000000000007 a5 000000007f7e9a90
   a6 0000000000000299 a7 0000000000000001 s2 000000007f7e9a90 s3 000000007f7e9b18
   s4 00000000000007d0 s5 0000000000000000 s6 0000000000000000 s7 0000000000000000
   s8 0000000000000000 s9 0000000000000000 sA 0000000000000000 sB 0000000000000000
   t3 ffffffffffffffff t4 0000000000000007 t5 000000000000002c t6 0000000000000294
   pc fffffffffffffffe va fffffffffffffffe insn ffffffff sr 8000000200046020
   User fetch segfault @ 0xfffffffffffffffe

I have a question. When using spike, spike provide simulation env? In your cmd, no boom simulator is specified.

[jerryz123]

Spike is the RISC-V functional ISA simulator.
In this case, Spike displays the same error that you saw when running on BOOM. This indicates that this is a software issue, rather than a hardware issue.

[qqjinger]

So what should I do to run multithreaded coremark? In general, how can we run multithreaded applications with simulator-chipyard-SmallBoomConfig-debug? Do you have any suggestions？Thanks！

[jerryz123]

You have to compile coremark in baremetal mode (no pk), and then edit the baremetal initialization code to boot up all harts. https://github.com/riscv-boom/riscv-coremark/tree/master/riscv64-baremetal contains files for compiling coremark in baremetal. That repo also contains a script to do this.
You should look at crt.S for the line where only hart0 is allowed to execute the main program. You should modify this to allow other harts through.

[qqjinger]

I'm sorry to bother you. After generating coremark.bare.riscv which has modified crt.S, what method should I do to run this coremark. Previously, using cmd "boom_simulator pk coremark.riscv 0x0 0x0 0x66 1", the riscv application can be run. But without pk, how can I run the riscv application?

[jerryz123]

You can just run directly spike coremark.bare.riscv

[qqjinger]

As you pointed, spike is a RISCV-V functional ISA simulator, not real boom simulator. So, Whether is there a method to run multi-thread application in boom's multi-core simulator? I want to see the coremark difference between single-thread on single boom core and multi-threads on multi boom cores.

[jerryz123]

If a binary can be executed with spike -p2 binary.riscv, then it can be executed by a multicore boom config as well.

[qqjinger]

I have tried to run coremark.bare.riscv on spike with cmd "spike coremark.bare.riscv". No error occurs but "Total time" and "Iterations/Sec" can not be printed. Cmd log is as follow：
2K performance run parameters for coremark.
CoreMark Size : 666
Total ticks : 14703689829
Total time (secs): %f
Iterations/Sec : %f
Iterations : 40000
Compiler version : GCC9.2.0
Compiler flags : -O2 -mcmodel=medany -static -std=gnu99 -fno-common -nostdlib -nostartfiles -lm -lgcc -T ../riscv64-baremetal/link.ld
Memory location : Please put data memory location here
(e.g. code in flash, data on heap etc)
seedcrc : 0xe9f5
[0]crclist : 0xe714
[0]crcmatrix : 0x1fd7
[0]crcstate : 0x8e3a
[0]crcfinal : 0x25b5
Correct operation validated. See README.md for run and reporting rules.
*** FAILED *** (tohost = 1337)

Using coremark codes you pointed out yesterday，I can easily get two riscv files, "coremark.riscv" and "coremark.bare.riscv". Cmd "spike pk coremark.riscv" can print float "Total time (secs)" and "Iterations/Sec" value, but  cmd "spike coremark.bare.riscv" can not print float result as below report.
I notice that all inst pc in "coremark.bare.riscv" are larger than 8000_0000. From the perspective of file size, "coremark.bare.riscv" is 24KB and "coremark.riscv" is 149KB. I don't know whether it is normal or not for baremetal riscv far less than standard riscv.
So what should I do to print correct "Total time (secs)" and "Iterations/Sec" when using "coremark.bare.riscv"?  Thank you very much!

[qqjinger]

Hi guys,
After much efforts, single-thread-baremetal-coremark.riscv run normally on spike and boom simulator. At the same time, multi-thread baremetal coremark application can be generated without errors, but a new problem occurs. Running 2-thread-baremetal-coremark.riscv on 2-core-boom-simulator shows the following error message:
Assertion failed: 'A' channel carries PutPartial type unsupported by manager (connected at BusWrapper.scala:333:18)
at Monitor.scala:44 assert(cond, message)
I dont know what changes lead to this problem. I have modified crt.S file by changing core num from 1 to 2, corresponding to file's line 125. Can someone give me some suggestions? Thanks!

[jerryz123]

Can you post, or email to me, a waveform from this run?

[qqjinger]

Hi，
Here is my vcd file, "svn1_1core_bare_lpthread1.vcd", which is generated by 2-threads-bare-coremark.riscv running on boom-2-cores-simulator. This vcd file can be obtained from my github repository，"https://github.com/qqjinger/fileShare".
My compiler options to compile coremark is as follow:
riscv64-unknown-elf-gcc -O2 -mcmodel=medany -std=gnu99 -fno-common -nostdlib -nostartfiles -lm -lgcc -T
../riscv64-baremetal-multi-thread/link.ld --sysroot=riscv-tools-install/sysroot -I../riscv64-baremetal-multi-thread
-I. -DFLAGS_STR=""-O2 -mcmodel=medany -std=gnu99 -fno-common -nostdlib -nostartfiles -lm -lgcc -T
../riscv64-baremetal-multi-thread/link.ld --sysroot=riscv-tools-install/sysroot -DMULTITHREAD=2 -DUSE_PTHREAD
-lpthread -Wl,-rpath-link riscv-tools-install/sysroot/lib/ "" -DITERATIONS=0 -DMULTITHREAD=2 -DUSE_PTHREAD
core_list_join.c core_main.c core_matrix.c core_state.c core_util.c ../riscv64-baremetal-multi-thread/core_portme.c
../riscv64-baremetal-multi-thread/syscalls.c ../riscv64-baremetal-multi-thread/crt.S -o ./coremark_multi_thread.bare.riscv
-lpthread -Wl,-rpath-link riscv-tools-install/sysroot/lib/
The above cmd is to generate 2-thread-baremetal-coremark.riscv. No error occurs but two warnings exist, you can see it as follow.
../riscv64-baremetal-multi-thread/core_portme.c: In function 'core_start_parallel':
../riscv64-baremetal-multi-thread/core_portme.c:248:16: warning: implicit declaration of function 'pthread_create' [-Wimplicit-function-declaration]
248 | return (ee_u8)pthread_create(&(res->port.thread),NULL,iterate,(void *)res);
| ^~~~~~~~~~~~~~
../riscv64-baremetal-multi-thread/core_portme.c: In function 'core_stop_parallel':
../riscv64-baremetal-multi-thread/core_portme.c:252:16: warning: implicit declaration of function 'pthread_join'; did you mean 'pthread_atfork'? [-Wimplicit-function-declaration]
252 | return (ee_u8)pthread_join(res->port.thread,&retval);
| ^~~~~~~~~~~~
| pthread_atfork

  There is a strange phenomenon that baremetal with "-DMULTITHREAD=1" option works well in 1-core-boom-simulator while baremetal with "-DMULTITHREAD=1 -lpthread" option 
reports error messages. Spike reports the same log when using above corresponding options.

[qqjinger]

I also upload the riscv and objdump files of 1-thread-with-lpthread and 1-thread-without-lpthread coremark. You can download all these files from my repository, "https://github.com/qqjinger/fileShare".
cmd "spike/boom_simulator+coremark_1thread_NOlpthread.bare.riscv" performs well and reports correct coremark.
cmd "spike/boom_simulator+coremark_1thread_lpthread.bare.riscv" reports error logs.
1) spike reports errors as below:
terminate called after throwing an instance of 'trap_store_access_fault'
2) boom simulator reports error as below:
Assertion failed: 'A' channel carries PutPartial type unsupported by manager (connected at BusWrapper.scala:333:18)
at Monitor.scala:44 assert(cond, message)

[jerryz123]

Can you verify this binary works on a dual-core rocket design first? It looks like there is something wrong with the binary ELF that is causing the binary loader to attempt to write to a non-existent memory address (0x7fff_f000)

[qqjinger]

1. About wrong address
   I also found that the address was wrong. But it is strange！When not using "-lpthread" compiler option, .text.init section of elf file in section headers and program headers are both 0x8000_0000，which is equal to "Entry point address". When using "-lpthread" compiler option, .text.init section of elf file in section headers is correct 0x8000_0000，but .text.init section of elf file in program headers turn into a wrong address 0x7fff_f000 unexpectedly. I have tired to change addr by PHDR cmd in link.ld file, but it only change physical addr and does not influence virtual address.
   Do you have some suggestions? Thank you very much.

2. For coremark changeing to multi-thread
   For multi-thread, I changed crt.S file from 1 to 2 as below:

1. originally:
   #get core id
   csrr a0, mhartid
   #for now, assume only 1 core
   li a1,1
   1:bgeu a0,a1,1b
2. changed:
   #get core id
   csrr a0, mhartid
   #for now, assume only 2 core
   li a1,2
   1:bgeu a0,a1,1b
   I am not sure whether this is enough for chaning to multi-thread.

[qqjinger]

Hi,
Currently, I have known that spike and pk do not support multi-threaded application. So running multi-threaded application on bare-mode is the unique way. Because multi-thread application need task segmentation, multi thread switching, stack saving and other functions, so should I realize all this functions in syscall.c and crt.S?
I haven't done multithreading before. If I need to implement all these, I think it may be a big project.

[wkxfudan]

I have a same problem when I run coremark.bare.riscv on the boom. I just run the normal coremark for 1 iteration. And I get the log with no Total time and Iteration/Sec

$ ./simulator-chipyard-MediumBoomConfig-debug -v coremark.vcd ~/git/chipyard/software/coremark/riscv-coremark/coremark.bare.riscv
This emulator compiled with JTAG Remote Bitbang client. To enable, use +jtag_rbb_enable=1.
Listening on port 38393
[UART] UART0 is here (stdin/stdout).
2K performance run parameters for coremark.
CoreMark Size    : 666
Total ticks      : 270707
Total time (secs): %f
Iterations/Sec   : %f
ERROR! Must execute for at least 10 secs for a valid result!
Iterations       : 1
Compiler version : GCC9.2.0
Compiler flags   : -O2 -mcmodel=medany -static -std=gnu99 -fno-common -nostdlib -nostartfiles -lm -lgcc -T ../riscv64-baremetal/link.ld   
Memory location  : Please put data memory location here
                        (e.g. code in flash, data on heap etc)
seedcrc          : 0xe9f5
[0]crclist       : 0xe714
[0]crcmatrix     : 0x1fd7
[0]crcstate      : 0x8e3a
[0]crcfinal      : 0xe714
Errors detected

May be I think it has the same problem as bellow. Has the author of this issue solved the problem?

I have tried to run coremark.bare.riscv on spike with cmd "spike coremark.bare.riscv". No error occurs but "Total time" and "Iterations/Sec" can not be printed. Cmd log is as follow： 2K performance run parameters for coremark. CoreMark Size : 666 Total ticks : 14703689829 Total time (secs): %f Iterations/Sec : %f Iterations : 40000 Compiler version : GCC9.2.0 Compiler flags : -O2 -mcmodel=medany -static -std=gnu99 -fno-common -nostdlib -nostartfiles -lm -lgcc -T ../riscv64-baremetal/link.ld Memory location : Please put data memory location here (e.g. code in flash, data on heap etc) seedcrc : 0xe9f5 [0]crclist : 0xe714 [0]crcmatrix : 0x1fd7 [0]crcstate : 0x8e3a [0]crcfinal : 0x25b5 Correct operation validated. See README.md for run and reporting rules. *** FAILED *** (tohost = 1337)

Using coremark codes you pointed out yesterday，I can easily get two riscv files, "coremark.riscv" and "coremark.bare.riscv". Cmd "spike pk coremark.riscv" can print float "Total time (secs)" and "Iterations/Sec" value, but  cmd "spike coremark.bare.riscv" can not print float result as below report.
I notice that all inst pc in "coremark.bare.riscv" are larger than 8000_0000. From the perspective of file size, "coremark.bare.riscv" is 24KB and "coremark.riscv" is 149KB. I don't know whether it is normal or not for baremetal riscv far less than standard riscv.
So what should I do to print correct "Total time (secs)" and "Iterations/Sec" when using "coremark.bare.riscv"?  Thank you very much!