optee-os Xilinx kv260 panic regression in 4.1.0

[mikkorapeli-linaro]

After update of our Trusted Substrate firmware (https://gitlab.com/Linaro/trustedsubstrate/meta-ts/ ARM System Ready compatible firmware with secure boot for a number of arm64 boards) from optee 4.0.0 to 4.1.0, Xilinx kv260 board panics with:

https://ledge.validation.linaro.org/scheduler/job/82020

ser2net KV260 01 port telnet,7341 speed s
�Xilinx Zynq MP First Stage Boot Loader 

Release 2022.2   Oct  7 2022  -  04:56:16
MultiBootOffset: 0x40
Reset Mode	:	System Reset
Platform: Silicon (4.0), Running on A53-0 (64-bit) Processor, Device Name: XCZUUNKNEG

QSPI 32 bit Boot Mode 

FlashID=0x20 0xBB 0x20

PMU Firmware 2022.2	Oct  7 2022   04:56:16
PMU_ROM Version: xpbr-v8.1.0-0
�D/TC:0   add_phys_mem:666 VCORE_UNPG_RX_PA type TEE_RAM_RX 0x60000000 size 0x000a4000
D/TC:0   add_phys_mem:666 VCORE_UNPG_RW_PA type TEE_RAM_RW 0x600a4000 size 0x0015c000
D/TC:0   add_phys_mem:666 ta_base type TA_RAM 0x60200000 size 0x0fe00000
D/TC:0   add_phys_mem:666 ROUNDDOWN(GIC_BASE + GICD_OFFSET, CORE_MMU_PGDIR_SIZE) type IO_SEC 0xf9000000 size 0x00200000
D/TC:0   add_phys_mem:666 ROUNDDOWN(GIC_BASE, CORE_MMU_PGDIR_SIZE) type IO_SEC 0xf9000000 size 0x00200000
D/TC:0   add_phys_mem:680 Physical mem map overlaps 0xf9000000
D/TC:0   add_phys_mem:666 ROUNDDOWN(CONSOLE_UART_BASE, CORE_MMU_PGDIR_SIZE) type IO_SEC 0xff000000 size 0x00200000
D/TC:0   add_phys_mem:666 TEE_SHMEM_START type NSEC_SHM 0x70000000 size 0x10000000
D/TC:0   add_va_space:706 type RES_VASPACE size 0x00a00000
D/TC:0   add_va_space:706 type SHM_VASPACE size 0x02000000
D/TC:0   dump_mmap_table:838 type TEE_RAM_RX   va 0x60000000..0x600a3fff pa 0x60000000..0x600a3fff size 0x000a4000 (smallpg)
D/TC:0   dump_mmap_table:838 type TEE_RAM_RW   va 0x600a4000..0x601fffff pa 0x600a4000..0x601fffff size 0x0015c000 (smallpg)
D/TC:0   dump_mmap_table:838 type SHM_VASPACE  va 0x60200000..0x621fffff pa 0x00000000..0x01ffffff size 0x02000000 (pgdir)
D/TC:0   dump_mmap_table:838 type RES_VASPACE  va 0x62200000..0x62bfffff pa 0x00000000..0x009fffff size 0x00a00000 (pgdir)
D/TC:0   dump_mmap_table:838 type TA_RAM       va 0x62c00000..0x729fffff pa 0x60200000..0x6fffffff size 0x0fe00000 (pgdir)
D/TC:0   dump_mmap_table:838 type NSEC_SHM     va 0x72a00000..0x829fffff pa 0x70000000..0x7fffffff size 0x10000000 (pgdir)
D/TC:0   dump_mmap_table:838 type IO_SEC       va 0x82a00000..0x82bfffff pa 0xf9000000..0xf91fffff size 0x00200000 (pgdir)
D/TC:0   dump_mmap_table:838 type IO_SEC       va 0x82c00000..0x82dfffff pa 0xff000000..0xff1fffff size 0x00200000 (pgdir)
D/TC:0   core_mmu_xlat_table_alloc:527 xlat tables used 1 / 5
D/TC:0   core_mmu_xlat_table_alloc:527 xlat tables used 2 / 5
D/TC:0   core_mmu_xlat_table_alloc:527 xlat tables used 3 / 5
I/TC: 
I/TC: OP-TEE version: 4.1.0-dev (gcc version 13.2.0 (GCC)) #1 Fri Jan 19 17:14:14 UTC 2024 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
D/TC:0 0 call_preinitcalls:21 level 2 mobj_mapped_shm_init()
D/TC:0 0 mobj_mapped_shm_init:470 Shared memory address range: 60200000, 62200000
D/TC:0 0 call_initcalls:40 level 1 register_time_source()
D/TC:0 0 call_initcalls:40 level 1 teecore_init_pub_ram()
D/TC:0 0 call_initcalls:40 level 3 check_ta_store()
D/TC:0 0 check_ta_store:454 TA store: "early TA"
D/TC:0 0 check_ta_store:454 TA store: "Secure Storage TA"
D/TC:0 0 check_ta_store:454 TA store: "REE"
D/TC:0 0 call_initcalls:40 level 3 early_ta_init()
D/TC:0 0 early_ta_init:56 Early TA fd02c9da-306c-48c7-a49c-bbd827ae86ee size 188835 (compressed, uncompressed 423776)
D/TC:0 0 call_initcalls:40 level 3 verify_pseudo_tas_conformance()
D/TC:0 0 call_initcalls:40 level 3 tee_cryp_init()
D/TC:0 0 call_initcalls:40 level 4 tee_fs_init_key_manager()
D/TC:0 0 call_initcalls:40 level 6 mobj_init()
D/TC:0 0 call_initcalls:40 level 6 default_mobj_init()
D/TC:0 0 call_finalcalls:59 level 1 init_multi_core_panic_handler()
I/TC: Primary CPU switching to normal world boot
PANIC at PC : 0x00000000fffed644

optee build configuration on top of meta-arm yocto recipes:

https://gitlab.com/Linaro/trustedsubstrate/meta-ts/-/blob/master/meta-trustedsubstrate/recipes-security/optee/optee-os-zynmp.inc?ref_type=heads

The panic is specific to Xilinx kv260 since Xilinx zcu102 without optee is working fine https://ledge.validation.linaro.org/scheduler/job/82021 and qemu with optee is working though there are no TAs like fTPM https://ledge.validation.linaro.org/scheduler/job/82016

On rockpi4b and synquacer boards with fTPM TA there is a separate RPMB/MMC related kernel crash when loading tpm_ftpm_tee kernel module but this is a kernel regression in 6.6.17 https://ledge.validation.linaro.org/scheduler/job/82017 https://ledge.validation.linaro.org/scheduler/job/82019

[jforissier]

Hi @mikkorapeli-linaro

I/TC: Primary CPU switching to normal world boot
PANIC at PC : 0x00000000fffed644

At this point we should be back from secure world. I wonder what piece of software logs the PANIC message, is it the Xilinx FSBL? Any way to know what causes the panic?
Does the exact same configuration work with OP-TEE 4.0?

[mikkorapeli-linaro]

Hi, this crash goes away if I downgrade trusted-firmware-a from 2.10 to 2.9 so the issue must be there not in optee.

[jforissier]

Hi, this crash goes away if I downgrade trusted-firmware-a from 2.10 to 2.9 so the issue must be there not in optee.

Interesting. Could be that the new version requires some build flags be set explicitly.