Building cheriot-safe for Artix-A7 100T

[Thingybob8055]

Hello,

I have been attempting to build the core for the Arty-A7 100T. I did get it to build but I wanted to point something out and also would like some guidance for getting CHERIot compiled software running on the core.

Software

I am using Vivado 2022.2

Commit used

31ddb47

Steps I used to build

1. I created a new project in Vivado targeting the Arty-A7 100T
2. For source directories I added: All directories in
   a. All directories from src excluding msft_cheri_tb
   b. rtl/ from cheriot-ibex folder
   c. vendor/lowrisc_ip from cheriot-ibex
3. The top file for me was automatically detected which is: msftDvIp_cheri_arty7_fpga.sv
4. I added the constraint file from this reo

During synthesis and implementatin

When I was synthesising using the Vivado (running as a GUI), I encountered this error:

[Place 30-574] Poor placement for routing between an IO pin and BUFG. If this sub optimal condition is acceptable for this design, you may use the CLOCK_DEDICATED_ROUTE constraint in the .xdc file to demote this message to a WARNING. However, the use of this override is highly discouraged. These examples can be used directly in the .xdc file to override this clock rule.

      < set_property CLOCK_DEDICATED_ROUTE FALSE [get_nets sck0_i_IBUF] >
      sck0_i_IBUF_inst (IBUF.O) is locked to IOB_X1Y114
       and sck0_i_IBUF_BUFG_inst (BUFG.I) is provisionally placed by clockplacer on BUFGCTRL_X0Y31

So I added this line to the constraint file in the repo:

set_property CLOCK_DEDICATED_ROUTE FALSE [get_nets sck0_i_IBUF]

After adding the line, I managed to build and generate the bitstream.

My questions

1. I when I uploaded the bitstream, LED4 was blinking and it is labeled alive_o, is this a watchdog LED and does it indicate the system is working correctly?
2. I did compile the LLVM project (cheriot branch) for the compiler. Which produced the clangd binary. How do I go about running software on this core (on the Arty-A7 100T).
3. Was the constraint file change fine to use?

Thanks!

[davidchisnall]

I did compile the LLVM project (cheriot branch) for the compiler. Which produced the clangd binary. How do I go about running software on this core (on the Arty-A7 100T).

We need to update the build instructions in the RTOS repo, because the CHERIoT compiler is now maintained here.

The easiest way of making sure that you have an up-to-date developer environment is to use the dev container, which includes the latest toolchain and the Sail and Ibex simulators.

The RTOS includes a board description file (ibex-safe-simulator.json) that is tested with the Verilator simulator (see: https://cheriot.org/jekyll/update/2023/10/17/ibex-now-in-dev-container.html). The FPGA build should have the same memory layout, so firmware images built with that should work on the FPGA as well (I have not yet tested this). @kliuMsft can hopefully explain how to load it.

It's possible that the clock speed will be different and so you may need to tweak this line.

[Thingybob8055]

Hello,

Thanks for the reply!
After compiling C code, what is the process of getting the binary running on the core? Do I need to use openOCD to upload the compiled binary to the FPGA (running the cheriot)

[kliuMsft]

@Thingybob8055 I updated a couple more scripts and a simple README under build/. However it looks like you are ahead of me - which is great. I believe just by putting firmware images in the right place during synthesis, they are included in the bitfile. Please give it a try. You can also use openocd/jtag to load image but that would require some scripting (to talk to the backdoor memory access port in FPGA through JTAG).

[Thingybob8055]

Thanks! I will try it out. I assume the compiler produces the firmware images.
Do you have an example code or how to compile using the compiler? Or can brief me how (with an example LED blink code if its no trouble)

[kliuMsft]

Here's the answer to your earlier questions,

1. it's ok to add the command you suggested to the constraints. It probably doesn't matter yet (unless you are using SPI interfacd), since the signal is only used as SPI clock
2. The "alive" signal is actually driven by a hardware counter that runs on the main sysclk, so the blinking LED means the clock is alive.

I will try to work out a software build example. Meanwhile if you can try the cheri_sanity or coremark example in the sim/verilator directory it would be good. The examples don't drive LEDs but they do print to UART, which is connected to the USB port on A7 and if things work out you should be able to see printouts on a serial terminal like PuTTY or TeraTerm

[kliuMsft]

@Thingybob8055 I have added an (hopefully complete) example under sim/verilator/examples/hello_world. The example simply prints out a message on uart and blinks 2 LEDs, however it contains all the necessary startup code that you can use to start developing bare-metal the CHERIoT way.. Please let me know how it works out.

[Thingybob8055]

Hello,
I was away for a few days. I will to work on this and let you know how it goes!
@kliuMsft To compile the C code for the hello word example, do I need to setup the dev container linked above in this issue post? So that I have the compiler for CHERIoT
And for now, can I just use the two .vhx file in the firmware folder to get the example running?

[kliuMsft]

Yes the vhx in veriator/examples should work - I'd suggest trying the hello_world first since it also should blink LED. You can use either the dev container or locally installed toolchain (in which case the build_fpga_test.sh in hello_world can serve as an example)

[Thingybob8055]

Hello,
The hello world example provided is working, I saw a Hello World message in the serial monitor and also the LEDs are blinking!
I would like to setup the compiler locally.
Are these instructions now up to date: https://github.com/microsoft/cheriot-rtos/blob/main/docs/GettingStarted.md
After building, the compiler (and LSP) binaries will probably me put inside a build folder in the repo root. Will I need to edit build_fpga_test.sh to find the correct location of these binaries?
And if so can you provide an example?
Or can I simply run the binary from the terminal like this to compile:
-target riscv32-unknown-unknown -mcpu=cheriot -mabi=cheriot -mxcheri-rvc -mrelax -Oz -nostdlib -DCHERIOT

[Thingybob8055]

I followed the build instructions for the compiler (the same one linked above)
@kliuMsft I was trying to run the build_fpga_test.sh script, and I added a line:
CLANG=<path-to-compiled-clang-13>
When running the script, it kept saying that it can't find the folder ./csrc_cheri and startup.S not found. So I ran the following commands manually:

<path-to>/clang-13 -target riscv32-unknown-unknown -mcpu=cheriot -mabi=cheriot -mxcheri-rvc -mrelax -Oz -nostdlib -DCHERIOT -c csrc_cheri/startup.S

<path-to>/clang-13 -target riscv32-unknown-unknown -mcpu=cheriot -mabi=cheriot -mxcheri-rvc -mrelax -Oz -nostdlib -DCHERIOT -I./csrc_cheri/ -Tlink_fpga_test.ld -o hello_world.elf hello_world.c startup.o csrc_cheri/util.c csrc_cheri/cstart.c

I managed to get a hello_world.elf file. But if I run objcopy (one installed by default on Ubuntu 22.04), I get this error:

objcopy -O binary -S hello_world.elf hello_world.bin
objcopy: Unable to recognise the format of the input file `hello_world.elf'

Is this the correct way to convert the elf file to a .bin or do I need another tool/specific version of the tool?
And how would I next convert the .bin to a .hex?
Thanks!

EDIT: I used the objcopy binary provided here, and it worked: https://github.com/lowRISC/lowrisc-toolchains/releases
Is this okay?
And I ran ./bin2hex.pl hello_world.bin > cpu0_iram.vhx
So have I compiled and generated the hex correctly after all these steps?
Well, do I use the cpu0_irom.vhx that you have provided, as the steps only produced the iram hex file and not the irom

[kliuMsft]

Yes you will have to massage the file paths in build_fpga_test.sh a little bit to get things to work. you can use either the obj_copy comes with the LLVM or GCC toos. Your steps look good - great job. cpu0_irom0.vhx just jumps to iram, so it can be reused.

[Thingybob8055]

I tried manually compiling a firmware hex file using my steps and it did work!
But it is a bit of a hassle to manually re-generate the bitstream to get a binary running on the core (it takes a while!)
If its no trouble, can you show how to upload a binary via openocd to the core?
Thanks!

[kliuMsft]

This might take a little time, as I need to remember how JTAG loading works for this board. Also this way you may not be able to use the same usb/jtag connection to load both bitfile and firmware image. Possibly another usb2jtag (FH232) is needed.

[davidchisnall]

The other approach that I’ve been pondering is to provide an IROM loader that reads the firmware into IRAM over the UART and then jumps to it. That might be easier than using JTAG. Writing 256 KiBs at 115,200 baud will be a lot quicker than reprogramming the FPGA.

[kliuMsft]

That's definitely a valid approach. Either way you probably want to program the bit file to flash or micro SD, so that you can reboot/reload bit file by just resetting the board.

[Thingybob8055]

Generating the bit file from Vivado takes quite a long time actually and can be a bit of a wait, if I change just one line in the C programs for example. Which is why I was wondering if I can use openocd to program the core. Just like from here: https://github.com/lowrisc/ibex-demo-system
I do have USB to JTAG modules available if that is needed. Or as davidchisnall said, over UART would also be nice.
Let me know if you have updated the instructions to do this.
If you need me to test this as well, I will be happy to do so.
Thanks a lot for the support so far!!

[atrieu]

I managed to make the hello_world example in build/ sim/verilator/examples/hello_world/firmware work on my Arty A7, though when I try with the hello_world example from cheriot-rtos, it prints random symbols instead. It works correctly with the cheriot_ibex_safe_sim simulator though. Any clue what I'm doing wrong ?

Edit: fix my brainfart

[kliuMsft]

@atrieu, just to confirm, if you use the firmware image in sim/verilator/examples/hello_world/firmwrare on FPGA board (not in sim), do you see the correct UART printout?

[atrieu]

@atrieu, just to confirm, if you use the firmware image in sim/verilator/examples/hello_world/firmwrare on FPGA board (not in sim), do you see the correct UART printout?

Yes, this one works. That's what I meant by build/ initially, but I brainfarted.

[kliuMsft]

In this case I suspect the example in cheri-rtos may need an update. @davidchisnall, I recall that you had to tweak the uart settings to get it work in rtos?

[davidchisnall]

Hi, I have a BSP for the A7 that I hope works, but have not got an A7 to test it with. I will post it to a branch this morning for feedback.

[davidchisnall]

Try this branch:

https://github.com/microsoft/cheriot-rtos/tree/arty-a7

You will need to set board to ibex-arty-a7-100 during the configure step when building.

This should work with the UART and exposes an LED driver for two of the LEDs.

[davidchisnall]

Note: Resetting still doesn't work, but if someone can confirm that the UART driver works then I will add something to load firmware over the UART that can run in IROM.

[atrieu]

@davidchisnall Thanks, it worked on my board !
It printed the following over the UART

             Hello world compartment: Hello world
                                                 Error handler: Thread exit CSP=0x20040930 (v:1 0x20040730-0x20040930 l:0x200 o:0x0 p: - RWcgml -- ---)

For the record, I executed the following commands in the dev container in VSCode:

$ cd cheriot-rtos/examples/01.hello_world/
$ xmake config --sdk=/cheriot-tools/ --board=ibex-arty-a7-100
[...]
$ xmake
[...]
$ cd build/cheriot/cheriot/release/
$ ../../../../../../scripts/run-ibex-safe-sim.sh hello_world

Then I copied the produced firmware/ and used the build_arty_a7 script from here.

[davidchisnall]

Hurray!

That is what should happen. Can you try adding this to the example:

Somewhere at the top:

#include <platform-leds.hh>

Somewhere around the hello world line:

auto leds = MMIO_CAPABILITY(LEDs, gpio_led0);
leds->enable_all();
bool on = true;
while (true)
{
  if (on)
  {
    leds->led_off(0);
    leds->led_on(1);
  }
  else
  {
     leds->led_on(0);
     leds->led_off(1);
  }
  on = !on;
  Timeout t{100};
  thread_sleep(&t);
}

This should make two of the LEDs alternate on/off states.

[atrieu]

I had to include a bit more, but it works ! LD5 and LD6 are alternating and LD4 is flickering.

Complete hello.cc

#include <platform-gpio-leds.hh>
#include <thread.h>
#include <compartment.h>
#include <debug.hh>
#include <fail-simulator-on-error.h>

/// Expose debugging features unconditionally for this compartment.
using Debug = ConditionalDebug<true, "Hello world compartment">;

/// Thread entry point.
void __cheri_compartment("hello") say_hello()
{
	auto leds = MMIO_CAPABILITY(LEDs, gpio_led0);
	leds->enable_all();
	bool on = true;
	Debug::log("Hello world");
	while (true)
	{
		if (on)
		{
			leds->led_off(0);
			leds->led_on(1);
		}
		else
		{
			leds->led_on(0);
			leds->led_off(1);
		}
		on = !on;
		Timeout t{100};
		thread_sleep(&t);
	}
	// Print hello world, along with the compartment's name to the default UART.
}

[Thingybob8055]

@davidchisnall Hi
I read through the previous commit you have made. Do you currently have instructions to load compiled code via UART?
And in order to do so, do I need to re-synthesise the core with the newly committed cpu_irom.vhx file?
Thanks!

[davidchisnall]

Hi,

I wrote a few instructions here. Basically, rerun the build_arty_a7 script with the committed vhx (if you don't like running random binaries, rebuild the vhx from here). When it boots, it will then ask for firmware. I use minicom for talking to the board. Minicom (-c on to get pretty colours) should be set to 115200 8N1 (which, I think, is the default these days). You can then use paste file (meta-z y - meta is escape by default, at least for me) to send it a firmware image.

If you are building CHERIoT RTOS, I've added a little bit of board support code there. Use --board=ibex-arty-a7-100 to compile for this config.

David

[atrieu]

Not sure if this is the right place to ask, but are CHERI exception causes fully implemented ?

I have tried playing around with it for testing. Basically, I have set up mtcc so that my handler is called:

void exn_handler() {
    volatile unsigned int mcause;
    volatile unsigned int mtval;
    char buf[11];

    __asm__ __volatile__("csrr %0, mcause": "=r" (mcause));
    __asm__ __volatile__("csrr %0, mtval": "=r" (mtval));

    uart_puts("Hello from the exception handler!");

    uart_print_string("Exception cause: 0x");
    write_nibbles(mcause, buf);
    uart_puts(buf);

    if (mcause == 0x1C) {
        unsigned int cap_exn_cause = mtval & 0x1F;
        uart_print_string("Capability exception cause: 0x");
        write_nibbles(cap_exn_cause, buf);
        uart_puts(buf);
    }

    while (1);

    return;
}

Then, I trigger an exception with

    cspecialr        ca0, mtdc
    ccleartag        ca0, ca0
    clw              x0, 0(ca0)

I get the following

Hello from the exception handler!
Exception cause: 0x1C
Capability exception cause: 0x0

while I would expect to find a tag violation with code 0x2.
Clearing the permissions with candperm ca0, ca0, x0 instead of the tag also results in a 0x0 code, while cspecialw pcc, ca0 results in a 0x18 permit access system registers violation code instead.

This happens on both the simulator and my Arty A7.

[davidchisnall]

Hi @atrieu,

There is currently a bug in exception priorities in Ibex. See this comment for the work-around that we use in the software stack. You should still get CHERI exceptions at the right point, but they will be the wrong numbers. This is tracked in microsoft/cheriot-ibex#11 and will hopefully be fixed soon. We're currently not using the CHERI exception causes for anything other than debugging, so it hasn't been a high priority.

You might be interested in talking to Tom Melham at Oxford. His students have been working on verifying the Ibex.

[kliuMsft]

Hi @atrieu,

Any way you can attach your ibex trace for the exception handler execution? As David pointed we haven't fully implemented exception causes yet, but even the existing RTL is supposed to report code 2 in this case (I have a very similar test case). So I am a little bit puzzled as why you are seeing this. Thanks.

[atrieu]

Hi @kliuMsft,

Sorry for the late answer, I was traveling. You can find the full simulator trace here: https://gist.github.com/atrieu/16ef5aa869676d37c79174068ee70453

I'm guessing the interesting part is here https://gist.github.com/atrieu/16ef5aa869676d37c79174068ee70453#file-cheriot_cap_exn_trace-log-L4586-L4596

              0	     19669	2004004a	feb5055b	CH.ccleartag	c10,c10	 x10:0x00000000+0x17e3e0000 x10=0x00000000+0x07e3e0000
              0	     19670	2004004e	00052003	-->clw	x0,0(c10)	 x10:0x00000000+0x07e3e0000  x0=0x00000000 PA:0x00000000 load:0x00000000
              0	     19674	20040090	    7179	c.CH.cinc16csp	c2,-48	  x2:0x20080000+0x17e3e0000  x2=0x2007ffd0+0x17e3e0000
              0	     19677	20040092	    f406	c.CH.cscsp	c1,40(c2)	  x2:0x2007ffd0+0x17e3e0000  x1:0x20040046+0x15ebe0000 PA:0x2007fff8 store:0x120040046+0x15ebe0000
              0	     19679	20040094	    f022	c.CH.cscsp	c8,32(c2)	  x2:0x2007ffd0+0x17e3e0000  x8:0x00000000+0x000000000 PA:0x2007fff0 store:0x000000000+0x000000000
              0	     19682	20040096	    ec26	c.CH.cscsp	c9,24(c2)	  x2:0x2007ffd0+0x17e3e0000  x9:0x00000000+0x000000000 PA:0x2007ffe8 store:0x000000000+0x000000000
              0	     19683	20040098	0051155b	CH.cincaddrimm	c10,c2,5	  x2:0x2007ffd0+0x17e3e0000 x10=0x2007ffd5+0x17e3e0000
              0	     19684	2004009c	00b5245b	CH.csetboundsimm	c8,c10,11	 x10:0x2007ffd5+0x17e3e0000  x8=0x2007ffd5+0x17e03c1d5
              0	     19685	200400a0	34202573	csrrs	x10,mcause,x0	  x0:0x00000000 x10=0x0000001c
              0	     19686	200400a4	    ca2a	c.swsp	x10,20(c2)	  x2:0x2007ffd0+0x17e3e0000 x10:0x0000001c PA:0x2007ffe4 store:0x00000001c
              0	     19687	200400a6	34302573	csrrs	x10,mtval,x0	  x0:0x00000000 x10=0x00000000

It shows that reading mcause gives 0x1c and reading mtval gives 0x0.

[kliuMsft]

Hi @atrieu,

Thanks! There is indeed a bug in the cheriot-ibex reporting which caused mtval not updating for a rv32 load/stores. I pushed a fix in the cheriot-ibex repo, would really appreciate if you can give it a try.

[atrieu]

Hi @kliuMsft,

Can confirm it works, both in simulator in a locally built devcontainer, and on the fpga board!

[kliuMsft]

@atrieu, that's great, thanks for confirming!

[davidchisnall]

Thanks. I've kicked off a rebuild of the devcontainer, which should pick up this fix.

[davidchisnall]

I believe this can be closed, we now have some getting started instructions for the A7 and it all seems to be working again after the recent breakage.