New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spike can't simulate simple program, progmem.c from icicle repository #192

Open
afiskon opened this Issue Apr 5, 2018 · 9 comments

Comments

Projects
None yet
3 participants
@afiskon
Copy link

afiskon commented Apr 5, 2018

Hello everyone,

I compiled icicle RISC-V core for ICE40 FPGA https://github.com/grahamedgecombe/icicle It works perfectly with BlackIce II development board.

Now I would like to simulate the program (progmem.c) before uploading it next time:

riscv64-unknown-elf-gcc -march=rv32i -mabi=ilp32 -Wall -Wextra -pedantic -DFREQ=24000000 -O2   -c -o progmem.o progmem.c
riscv64-unknown-elf-gcc -march=rv32i -mabi=ilp32 -Wl,-Tprogmem.lds -ffreestanding -nostartfiles -o progmem progmem.o start.o

Here is how I execute spike:

spike -d --isa=rv32i --pc=0x00000000 -m0x0:0x100000 progmem

Right after the start pc points to 0x1000 despite --pc flag:

: pc 0
0x0000000000001000

And naturally 0x1000 doesn't contain the code I would like to execute:

: mem 0x1000
0x0202859300000297
: mem 0x0000  
0x0302829300000297

Here are a few first instructions of the program:

$ riscv64-unknown-elf-objdump -d progmem
...
00000000 <start>:
   0:	00000297          	auipc	t0,0x0
   4:	03028293          	addi	t0,t0,48 # 30 <bss_start>
   8:	00000317          	auipc	t1,0x0
   c:	12830313          	addi	t1,t1,296 # 130 <bss_end>
  10:	00628863          	beq	t0,t1,20 <clear_bss_done>

As you can see the right data is stored by 0x0000 address but since spike ignored the --pc flag I can't simulate the program.

riscv-isa-sim was compiled from master branch (2dbcb01) using Arch Linux AUR package: https://aur.archlinux.org/packages/riscv-isa-sim-git

$ uname -a
Linux e733 4.15.12-1-ARCH #1 SMP PREEMPT Wed Mar 21 15:14:56 UTC 2018 x86_64 GNU/Linux
@evancox10

This comment has been minimized.

Copy link
Contributor

evancox10 commented Apr 8, 2018

Hi afiskon,
The --pc flag is not directly used as the reset vector. Instead, there is a hardcoded reset vector at 0x1000 that is always executed first. I've described this in more detail in #145.

Hope this helps.
-Evan

@afiskon

This comment has been minimized.

Copy link

afiskon commented Apr 9, 2018

Hello @evancox10

Alright, but this is not what --help says:

--pc=<address>        Override ELF entry point
@afiskon

This comment has been minimized.

Copy link

afiskon commented Apr 9, 2018

Also even without --pc flag spike still uses 0x1000 entry point. This is wrong since readelf clearly says that entry point of my executable is zero:

Entry point address:               0x0

This is clearly a bug since there is no way I can simulate the program with spike.

Here is a compiled binary if it will help somehow: https://afiskon.ru/s/9e/27c092dd19_progmem

@evancox10

This comment has been minimized.

Copy link
Contributor

evancox10 commented Apr 12, 2018

Hi afiskon,
As I described in the issue I linked to, even though spike is hardcoded to start at 0x1000, once it's done executing the 8 instructions there it will jump to either 1) the --pc value, if specified, or 2) the entry point defined in the ELF file. So in that sense --pc does work, it just doesn't quite work the way the help makes it sound.

In practice it's not the execution of this "boot loader" that causes problems — as I said, it's only 8 instructions— but rather its location in the memory map. If you want to place your program at 0x0, it must be smaller than 4 KiB to avoid growing into this bootrom space. This is obviously somewhat limiting. You will either need to keep your program small, or place it after the boot ROM.

By the way, having a fixed "first execution address" pretty similar to howe real HW works. For example, in Cortex-M systems, the cpu always first looks at addresses 0 and 4, then uses the data it finds there as the start PC and stack pointer, respectively. So those addresses are reserved and not available for actual program instructions.

Spike takes a slightly different approach: in keeping with the RISC-V goal of "doing things in SW that you can do in SW", it just starts executing the code at 0x1000. It's the job of the code there to jump to the user's entry point. So yes, a program that collides with this boot ROM will not simulate correctly and is not compatible with Spike. Again though, you will almost always have a similar limitation in real HW.

I agree the documentation is wrong/confusing. This small bootloader and boot sequence is not documented anywhere, and I only know about it from digging through the code.

If you want to change this behavior, it's not very hard. I'm going from memory here, but I believe if you just comment out the call to make_dtb(), then the bootrom will not be placed in the memory map. Then, you can change the macro defining the start location to whatever you like, including the command line --pc value or what have you. There are more details on this in #145, although the implementation details there could be out of date by now.

-Evan

@afiskon

This comment has been minimized.

Copy link

afiskon commented Apr 12, 2018

@evancox10 thank you a lot for explaining. It makes sense now!

I'm still having difficulties with emulating progmem.c from the icicle repository though and I don't quite understand why.

I simplified the program:

#include <stdint.h>

#define LEDS        *((volatile uint32_t *) 0x00010000)
//#define UART_BAUD   *((volatile uint32_t *) 0x00020000)
//#define UART_STATUS *((volatile uint32_t *) 0x00020004)
//#define UART_DATA   *((volatile  int32_t *) 0x00020008)
//#define MTIME       *((volatile uint64_t *) 0x00030000)
//#define MTIMECMP    *((volatile uint64_t *) 0x00030008)

#define UART_STATUS_TX_READY 0x1
#define UART_STATUS_RX_READY 0x2

#define BAUD_RATE 9600
/*
static void uart_puts(const char *str) {
    char c;
    while ((c = *str++)) {
        while (!(UART_STATUS & UART_STATUS_TX_READY));
        UART_DATA = c;
    }
}
*/
static inline uint32_t rdcycle(void) {
    uint32_t cycle;
    asm volatile ("rdcycle %0" : "=r"(cycle));
    return cycle;
}

int main() {
//    UART_BAUD = FREQ / BAUD_RATE;
    uint8_t leds = 0xAA;
    LEDS = leds;

    for (;;) {
 //       uart_puts("Hello, world!\r\n");
        leds = ~leds;
        LEDS = leds;

        uint32_t start = rdcycle();
        while ((rdcycle() - start) <= FREQ);
    }
}

Basically now it only changes one byte on LEDS address. I also changed -O2 to -O1 in the Makefile since spike has difficulties with -O2 (see below).

If I compile the code and execute spike -l -m0x0:0x30000 ./progmem 2>&1 | tee 1.log everything works almost as expected. Though there are exceptions in the log:

core   0: 0x0000000000000030 (0x000107b7) lui     a5, 0x10
core   0: 0x0000000000000034 (0x0aa00713) li      a4, 170
core   0: 0x0000000000000038 (0x00e7a023) sw      a4, 0(a5)
core   0: exception trap_store_access_fault, epc 0x0000000000000038
core   0:           tval 0x0000000000010000

I don't understand why there is an issue with accessing 0x10000. I allocated the memory using -m flag. If I change progmem.lds to explicitly mark this area as rwx, it doesn't change anything.

Here is another issue. If I restore original Makefile (with -O2) spike returns the following error:

ERROR: invalid load from debug module: 8 bytes at 0x0000000000000168
terminate called after throwing an instance of 'trap_load_access_fault'

Once again it's very strange. The program shouldn't access 8 bytes @ 0x168 since it ends @ 0x16c:

 160:	40e787b3          	sub	a5,a5,a4
 164:	fef6fce3          	bleu	a5,a3,15c <main+0x2c>
 168:	fe5ff06f          	j	14c <main+0x1c>

And even if it does the whole 0x0 ... 0x20000 area should be rwx.

Ok, let's return -O1 and modify progmem.c like this:

// #define LEDS        *((volatile uint32_t *) 0x00010000)
uint32_t LEDS;

And run spike:

$ spike -l -m0x0:0x30000 ./progmem 2>&1 | tee 2.log
ERROR: invalid load from debug module: 8 bytes at 0x0000000000000150
terminate called after throwing an instance of 'trap_load_access_fault'

Once again something is not right with access rights. If I execute readelf -a progmem I see:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
...
  [ 3] .bss              NOBITS          00000150 001150 000004 00  WA  0   0  4
...
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),

...
Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  LOAD           0x001000 0x00000000 0x00000000 0x00150 0x00154 RWE 0x1000

...
Symbol table '.symtab' contains 14 entries:
   Num:    Value  Size Type    Bind   Vis      Ndx Name
...
    11: 00000030    28 FUNC    GLOBAL DEFAULT    1 main
    12: 00000150     4 OBJECT  GLOBAL DEFAULT    3 LEDS

It looks like everything is OK with access rights to LEDS but for some reason spike doesn't understand it.

@afiskon afiskon changed the title --pc flag seems to be broken Spike can't emulate simple program, progmem.c from icicle repository Apr 12, 2018

@afiskon afiskon changed the title Spike can't emulate simple program, progmem.c from icicle repository Spike can't simulate simple program, progmem.c from icicle repository Apr 12, 2018

@afiskon

This comment has been minimized.

Copy link

afiskon commented Apr 12, 2018

In fact I can reproduce it on even simpler example:

$ cat test.c 
#include <stdio.h>

int main() {
  printf("Hello!\n");
}

$ riscv64-unknown-elf-gcc -march=rv32i -mabi=ilp32 -Wall -Wextra -pedantic test.c -o test
$ spike ./test
ERROR: invalid load from debug module: 8 bytes at 0x0000000000011e00
terminate called after throwing an instance of 'trap_load_access_fault'

$ pacman -Qi | egrep -A1 'Name\s+: riscv'
Name            : riscv-fesvr-git
Version         : r255.1a1edf1-1
--
Name            : riscv-isa-sim-git
Version         : r892.2dbcb01-1
--
Name            : riscv64-unknown-elf-binutils
Version         : 2.29.1-1
--
Name            : riscv64-unknown-elf-gcc
Version         : 7.3.0-1

$ uname -a
Linux e733 4.15.12-1-ARCH #1 SMP PREEMPT Wed Mar 21 15:14:56 UTC 2018 x86_64 GNU/Linux
@mwachs5

This comment has been minimized.

Copy link
Contributor

mwachs5 commented Apr 12, 2018

@afiskon

This comment has been minimized.

Copy link

afiskon commented Apr 13, 2018

@mwachs5 OK, but is it documented?

@afiskon

This comment has been minimized.

Copy link

afiskon commented Apr 13, 2018

If I modify progmem.lds like this:

MEMORY {
    bram (rwx) : ORIGIN = 0x00020000, LENGTH = 0x00020000
}

It doesn't change much for progmem.c:

ERROR: invalid store to debug module: 8 bytes at 0x0000000000020000
terminate called after throwing an instance of 'trap_store_access_fault'

Also test.c doesn't use addresses <= 0x1000 so I believe this is not the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment