xtensa: add qemu_xtensa_dc233c board for MMU testing #61864

dcpleung · 2023-08-24T18:04:18Z

This adds a new board qemu_xtensa_dc233c so that we can test MMU on Xtensa using QEMU, and various other changes needed for this to work.

Needs:

Toolchain: xtensa: add xtensa_dc233c sdk-ng#697
HAL: zephyr: remove intel_s1000 and add dc233c hal_xtensa#22

zephyrbot · 2023-08-24T18:05:24Z

The following west manifest projects have been modified in this Pull Request:

Name	Old Revision	New Revision	Diff
hal_xtensa	zephyrproject-rtos/hal_xtensa@`41a631d`	zephyrproject-rtos/hal_xtensa@`86b7ddf` (master)	zephyrproject-rtos/hal_xtensa@41a631d4..86b7ddf9

Note: This message is automatically posted and updated by the Manifest GitHub Action.

nashif · 2023-08-24T20:55:14Z

@jcmvbkbc FYI

andyross

This looks amazing; I'm going to give this a shot ASAP with the call0 work (which I think I have fleshed out now -- all the pieces are there and make sense and I just need to fix all the goofs, which is going to be a ton easier on qemu than it has been on the Mediatek board, can I just say?)

Some notes and nitpicks if you have time, but my vote would be to merge this fast and fix it later as needed.

andyross · 2023-08-24T23:57:47Z

soc/xtensa/dc233c/include/_soc_inthandlers.h

@@ -0,0 +1,279 @@
+/*
+ * THIS FILE WAS AUTOMATICALLY GENERATED.  DO NOT EDIT.


There's a patch in the mt8195 PR that will automatically invoke the script at build time, might be worth trying that if you have time.

andyross · 2023-08-25T13:58:23Z

arch/xtensa/core/xtensa-asm2.c

-
-	__asm__ volatile("rsr.vaddrstatus %0" : "=r"(vaddrstatus));
-	__asm__ volatile("rsr.vaddr0 %0" : "=r"(vaddr0));
-	__asm__ volatile("rsr.vaddr1 %0" : "=r"(vaddr1));


Cool. I bumped into this handler in my transcription and couldn't figure out why this was happening here vs. just in the assembly entry hooks. Nice to see it isn't needed for something I didn't understand. :)

These actually are neat because you can see which address is at fault, especially fault on data as EXCVADDR points to the code but not the data address.

EXCVADDR points to the code but not the data address.

This is not true. EXCVADDR points to the faulting data address in most of the exceptions, epc1 points to the faulting instruction.
vaddrstatus/vaddr* were introduced in the very recent MMU version that can work with cores having more than one load/store unit, MMUv3 (the one in dc233c) doesn't have it.

Weird... maybe there is something wrong with the xt-ocd and GDB I have been using on the hardware. Will need to double check.

andyross · 2023-08-25T14:00:40Z

arch/xtensa/core/xtensa_mmu.c

+ * i.e. vecbase, so that we need to reserve those
+ * space or else k_mem_map() would be mapping those,
+ * resulting in faults.
+ */


Mentioned this in the other PR, but I really think this particular code belongs in the SOC layer, as that's where memory map responsibility is and we can be a TON more prescriptive (e.g. "reserve the 4k vector region area at the start of hpsram" or whatever instead of "linker scripts usually...").

andyross · 2023-08-25T14:02:56Z

soc/xtensa/dc233c/include/xtensa-dc233c.ld

+    _WindowVectors_text_start = ABSOLUTE(.);
+    KEEP (*(.WindowVectors.text))
+    _WindowVectors_text_end = ABSOLUTE(.);
+  } >sram0_0_seg :sram0_0_phdr


Likewise there's a patch in the mt8195 series which will auto-generate all this vector nonsense for you from the offsets in the core-isa.h file, putting it in a file you can just include that emits to just one memory region. Saves a ton of boilerplate.

andyross · 2023-08-25T14:04:03Z

soc/xtensa/dc233c/include/xtensa-dc233c.ld

+  .debug.xt.callgraph 0 :
+  {
+    KEEP (*(.debug.xt.callgraph .debug.xt.callgraph.* .gnu.linkonce.xt.callgraph.*))
+  }


Having just recopied this same code too, we really should have an "xt-toolchain-debug.ld" include or something rather than continuing to cut and past stuff that quite frankly I don't have any understanding of at all.

andyross · 2023-08-25T14:06:01Z

soc/xtensa/dc233c/linker.ld

+ *
+ * Linker script for the Xtensa platform.
+ */
+#include <xtensa-dc233c.ld>


Irrelevant nitpick, but I don't see the value to putting the linker script in a separate file included from the official linker script file? If it was shared with other SOCs or had some configurability, sure, but this seems like just creating a new file for no reason?

I am just copying from sample_controller here. My plan is to see if we can use dc233c for qemu_xtensa, so we can get rid of sample_controller. At that point, I will need to clean all these up and refactor. That's going to take some time to unify some of the common linker script stuff for all Xtensa platforms.

andyross · 2023-08-25T14:07:58Z

soc/xtensa/dc233c/mmu.c

+	xtensa_itlb_entry_write_sync(
+		Z_XTENSA_PTE(vecbase, Z_XTENSA_KERNEL_RING,
+			     Z_XTENSA_MMU_X | Z_XTENSA_MMU_CACHED_WT),
+		Z_XTENSA_TLB_ENTRY((uint32_t)vecbase, 4));


I recognize there's some SOC-specific code paths with doing TLB setup, but this really seems like an arch layer thing to me. Maybe we could generate pin settings for vectors based on a simpler API where the SOC just labels its vector region for the arch layer?

The issue here is that all 4KB instruction TLBs are auto-refills and cannot be pinned. With default config, way 4 TLB covers 1MB, way 5 128MB, and way 6 512MB. We cannot simply pin a whole 1MB region in instr TLB in the common arch layer because that will make the whole region executable, and is a security risk (e.g. RODATA being executable). So this needs to be done on a platform or SoC basis as the linker script decides where other stuff locates, as it is highly dependent on the memory layout.

Adds a CPU binding for the Xtensa LX3 core. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

Turns out not all MMU enabled Xtensa cores have vaddrstatus, vaddr0 and vaddr1. And there does not seem to be a way to determine whether they are available. So remove them from the exception printout for now. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

MMU needs to be initialized before going in to C, so z_xtensa_mmu_init() is called in crt1.S before call to z_cstart(). Note that this is the default case and crt1.S can be disabled if board and SoC desire to do so. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

This adds code to deal with TLB misses as these comes as level 1 interrupts. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

This removes the identity map of the first 512MB in TLB way 6. Or else it would interfere with mapped entries resulting in double mapped TLB exception. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

During MMU initialization, we clear TLB way 6 to remove all identity mapping. Depending on CPU configuration, there are certain number of entries per way. So use the number from core-isa.h to clear enough entries instead of hard-coded number 8. Specifying an entry number outside of permitted range may result in CPU reacting in weird way so better to avoid that. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

Simply using __data_start and __data_end is not enough as it leaves out kobject regions which is supposed to be near .data section. So use _image_ram_start and _image_ram_end instead to enclose data, bss and various kobject regions (among others). Signed-off-by: Daniel Leung <daniel.leung@intel.com>

If CONFIG_XTENSA_RPO_CACHE is not enabled, it can be assumed that memory is not double mapped in hardware for cached and uncached access. So we can specify those regions to have cache via TLB. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

rsync after writing MISC0..3 or EXCSAVE1..7 registers is needed before reading them. Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>

Instead of only handling data TLB misses for VECBASE, change it to handle all data TLB misses in the double exception handler. It is because we may encounter data TLB misses when trying to preload page table entries inside user exception handler. Signed-off-by: Daniel Leung <daniel.leung@intel.com> Signed-off-by: Anas Nashif <anas.nashif@intel.com> Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>

This adds arch_reserved_pages_update() which is called in k_mem_manage_init() to reserve some physical pages so they are not re-mapped. This is due to Zephyr's linker scripts for Xtensa which often puts something before z_mapped_start (aka .text, for example, vecbase). That space needs to be reserved or else k_mem_map() would be mapping those that could result in faults. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

...and move it to xtensa_mmu_priv.h. This would allow the SoC layer to use the RING number if needed. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

...VECBASE during MMU initialization. This is to make sure that we can use the TLB miss handling in the exception vector after we have moved back the VECBASE during MMU initialization. Or else we would be forever stuck in ITLB miss because the exception vectors are not in TLB and we cannot populate the TLB because those vectors are not in TLB. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

This adds a function arch_xtensa_mmu_post_init() which can be implemented on the SoC layer to perform additional MMU initialization steps if necessary. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

The Xtensa HAL has been updated with the following changes: * Removed intel_s1000 overlay files as it is no longer needed. * Fix uninitialized variable warning in xthal_v2p() * Add config and overlay files for dc233c core so we can use QEMU with this core to test MMU code. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

This adds SoC and board configs to support the dc233c core that is available on QEMU. This core has more features than sample_controller, such as MMU support. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

nashif · 2023-08-26T23:26:53Z

merging as this is blocking testing of new toolchains in the new zephyr sdk.

zephyrbot added manifest manifest-hal_xtensa DNM This PR should not be merged (Do Not Merge) labels Aug 24, 2023

dcpleung force-pushed the xtensa/dc233c branch 2 times, most recently from 6e13867 to 112307b Compare August 24, 2023 18:56

dcpleung added the area: Xtensa Xtensa Architecture label Aug 24, 2023

dcpleung force-pushed the xtensa/dc233c branch from 112307b to 6c4bc4b Compare August 24, 2023 20:35

dcpleung marked this pull request as ready for review August 24, 2023 23:55

dcpleung requested review from nordic-krch, andyross, galak and nashif as code owners August 24, 2023 23:56

zephyrbot added area: Devicetree Binding PR modifies or adds a Device Tree binding area: Logging labels Aug 24, 2023

zephyrbot assigned dcpleung Aug 24, 2023

dcpleung requested a review from ceolin August 25, 2023 00:25

andyross previously approved these changes Aug 25, 2023

View reviewed changes

dcpleung and others added 11 commits August 26, 2023 15:19

dts: cpu: add cdns,tensilica-xtensa-lx3

5fff38f

Adds a CPU binding for the Xtensa LX3 core. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

xtensa: mmu: handle TLB misses during user exception

01df1dd

This adds code to deal with TLB misses as these comes as level 1 interrupts. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

xtensa: mmu: no longer identity map the first 512MB

17fa8d6

This removes the identity map of the first 512MB in TLB way 6. Or else it would interfere with mapped entries resulting in double mapped TLB exception. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

xtensa: Add missing synchronization

83e2434

rsync after writing MISC0..3 or EXCSAVE1..7 registers is needed before reading them. Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>

dcpleung added 5 commits August 26, 2023 15:19

xtensa: mmu: rename MMU_KERNEL_RING to Z_XTENSA_KERNEL_RING...

22a4947

...and move it to xtensa_mmu_priv.h. This would allow the SoC layer to use the RING number if needed. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

xtensa: mmu: allow SoC to do additional MMU init steps

f4c710f

This adds a function arch_xtensa_mmu_post_init() which can be implemented on the SoC layer to perform additional MMU initialization steps if necessary. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

xtensa: add support for dc233c SoC for QEMU

afb56ac

This adds SoC and board configs to support the dc233c core that is available on QEMU. This core has more features than sample_controller, such as MMU support. Signed-off-by: Daniel Leung <daniel.leung@intel.com>

nashif dismissed andyross’s stale review via afb56ac August 26, 2023 15:19

nashif force-pushed the xtensa/dc233c branch from 6c4bc4b to afb56ac Compare August 26, 2023 15:19

zephyrbot removed the DNM This PR should not be merged (Do Not Merge) label Aug 26, 2023

nashif approved these changes Aug 26, 2023

View reviewed changes

nashif merged commit 169f505 into zephyrproject-rtos:main Aug 26, 2023
25 of 26 checks passed

dcpleung deleted the xtensa/dc233c branch August 28, 2023 16:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

xtensa: add qemu_xtensa_dc233c board for MMU testing #61864

xtensa: add qemu_xtensa_dc233c board for MMU testing #61864

dcpleung commented Aug 24, 2023 •

edited

zephyrbot commented Aug 24, 2023 •

edited

nashif commented Aug 24, 2023

andyross left a comment

andyross Aug 24, 2023

andyross Aug 25, 2023

dcpleung Aug 25, 2023 •

edited

jcmvbkbc Aug 25, 2023 •

edited

dcpleung Aug 25, 2023

andyross Aug 25, 2023

andyross Aug 25, 2023

andyross Aug 25, 2023

andyross Aug 25, 2023

dcpleung Aug 25, 2023

andyross Aug 25, 2023

dcpleung Aug 25, 2023

nashif commented Aug 26, 2023

		@@ -0,0 +1,279 @@
		/*
		* THIS FILE WAS AUTOMATICALLY GENERATED. DO NOT EDIT.

xtensa: add qemu_xtensa_dc233c board for MMU testing #61864

xtensa: add qemu_xtensa_dc233c board for MMU testing #61864

Conversation

dcpleung commented Aug 24, 2023 • edited

zephyrbot commented Aug 24, 2023 • edited

nashif commented Aug 24, 2023

andyross left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dcpleung Aug 25, 2023 • edited

Choose a reason for hiding this comment

jcmvbkbc Aug 25, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nashif commented Aug 26, 2023

dcpleung commented Aug 24, 2023 •

edited

zephyrbot commented Aug 24, 2023 •

edited

dcpleung Aug 25, 2023 •

edited

jcmvbkbc Aug 25, 2023 •

edited