Kernel panic with CAAM on IMX6UL EVK?

[xfeng12]

when i run uboot->optee->linux，it throws this kernel panic:

[ 1.820000] caam 2140000.caam: failed to acquire DECO 0
[ 1.830000] caam 2140000.caam: Entropy delay = 3600
[ 1.850000] mmc1: new high speed SDIO card at address 0001
[ 1.880000] caam 2140000.caam: failed to acquire DECO 0
[ 1.890000] caam 2140000.caam: Entropy delay = 4000
[ 1.930000] caam 2140000.caam: failed to acquire DECO 0
[ 1.940000] caam 2140000.caam: Entropy delay = 4400
[ 1.980000] caam 2140000.caam: failed to acquire DECO 0
[ 1.980000] caam 2140000.caam: Entropy delay = 4800
[ 2.020000] caam 2140000.caam: failed to acquire DECO 0
[ 2.030000] caam 2140000.caam: Entropy delay = 5200
[ 2.070000] caam 2140000.caam: failed to acquire DECO 0
[ 2.070000] caam 2140000.caam: Entropy delay = 5600
[ 2.110000] caam 2140000.caam: failed to acquire DECO 0
[ 2.110000] caam 2140000.caam: Entropy delay = 6000
[ 2.150000] caam 2140000.caam: failed to acquire DECO 0
[ 2.160000] caam 2140000.caam: Entropy delay = 6400
[ 2.200000] caam 2140000.caam: failed to acquire DECO 0
[ 2.200000] caam 2140000.caam: Entropy delay = 6800
[ 2.240000] caam 2140000.caam: failed to acquire DECO 0
[ 2.250000] caam 2140000.caam: Entropy delay = 7200
[ 2.280000] caam 2140000.caam: failed to acquire DECO 0
[ 2.290000] caam 2140000.caam: Entropy delay = 7600
[ 2.330000] caam 2140000.caam: failed to acquire DECO 0
[ 2.330000] caam 2140000.caam: Entropy delay = 8000
[ 2.370000] caam 2140000.caam: failed to acquire DECO 0
[ 2.380000] caam 2140000.caam: Entropy delay = 8400
[ 2.410000] caam 2140000.caam: failed to acquire DECO 0
[ 2.420000] caam 2140000.caam: Entropy delay = 8800
[ 2.460000] caam 2140000.caam: failed to acquire DECO 0
[ 2.460000] caam 2140000.caam: Entropy delay = 9200
[ 2.500000] caam 2140000.caam: failed to acquire DECO 0
[ 2.510000] caam 2140000.caam: Entropy delay = 9600
[ 2.550000] caam 2140000.caam: failed to acquire DECO 0
[ 2.550000] caam 2140000.caam: Entropy delay = 10000
[ 2.590000] caam 2140000.caam: failed to acquire DECO 0
[ 2.590000] caam 2140000.caam: Entropy delay = 10400
[ 2.630000] caam 2140000.caam: failed to acquire DECO 0
[ 2.640000] caam 2140000.caam: Entropy delay = 10800
[ 2.680000] caam 2140000.caam: failed to acquire DECO 0
[ 2.680000] caam 2140000.caam: Entropy delay = 11200
[ 2.720000] caam 2140000.caam: failed to acquire DECO 0
[ 2.730000] caam 2140000.caam: Entropy delay = 11600
[ 2.760000] caam 2140000.caam: failed to acquire DECO 0
[ 2.770000] caam 2140000.caam: Entropy delay = 12000
[ 2.810000] caam 2140000.caam: failed to acquire DECO 0
[ 2.810000] caam 2140000.caam: Entropy delay = 12400
[ 2.850000] caam 2140000.caam: failed to acquire DECO 0
[ 2.860000] caam 2140000.caam: failed to instantiate RNG
[ 2.860000] ------------[ cut here ]------------
[ 2.870000] WARNING: CPU: 0 PID: 1 at mm/vmalloc.c:1465 __arm_iounmap+0x14/0x18()
[ 2.870000] Trying to vfree() nonexistent vm area (82031000)
[ 2.880000] Modules linked in:
[ 2.880000] CPU: 0 PID: 1 Comm: swapper Not tainted 4.1.15 #4
[ 2.890000] Hardware name: Freescale i.MX6 Ultralite (Device Tree)
[ 2.900000] [<80013f3c>] (unwind_backtrace) from [<80011ca4>] (show_stack+0x10/0x14)
[ 2.900000] [<80011ca4>] (show_stack) from [<8002b07c>] (warn_slowpath_common+0x74/0xac)
[ 2.910000] [<8002b07c>] (warn_slowpath_common) from [<8002b148>] (warn_slowpath_fmt+0x30/0x40)
[ 2.920000] [<8002b148>] (warn_slowpath_fmt) from [<8001a8c0>] (__arm_iounmap+0x14/0x18)
[ 2.930000] [<8001a8c0>] (__arm_iounmap) from [<803fb688>] (caam_remove+0x68/0x3dc)
[ 2.940000] [<803fb688>] (caam_remove) from [<803fc6c8>] (caam_probe+0xccc/0xf68)
[ 2.940000] [<803fc6c8>] (caam_probe) from [<8028c3a0>] (platform_drv_probe+0x48/0x98)
[ 2.950000] [<8028c3a0>] (platform_drv_probe) from [<8028aeb8>] (driver_probe_device+0x190/0x298)
[ 2.960000] [<8028aeb8>] (driver_probe_device) from [<8028b04c>] (__driver_attach+0x8c/0x90)
[ 2.970000] [<8028b04c>] (__driver_attach) from [<80289704>] (bus_for_each_dev+0x60/0x94)
[ 2.980000] [<80289704>] (bus_for_each_dev) from [<8028a600>] (bus_add_driver+0xd8/0x1cc)
[ 2.990000] [<8028a600>] (bus_add_driver) from [<8028b6b8>] (driver_register+0x78/0xf4)
[ 2.990000] [<8028b6b8>] (driver_register) from [<80009700>] (do_one_initcall+0x80/0x1d0)
[ 3.000000] [<80009700>] (do_one_initcall) from [<8074ed18>] (kernel_init_freeable+0xe8/0x1b0)
[ 3.010000] [<8074ed18>] (kernel_init_freeable) from [<8053e18c>] (kernel_init+0x8/0xe4)
[ 3.020000] [<8053e18c>] (kernel_init) from [<8000f448>] (ret_from_fork+0x14/0x2c)
[ 3.030000] ---[ end trace 6377d0690b443d12 ]---
[ 3.030000] caam: probe of 2140000.caam failed with error -11
[ 3.040000] Unable to handle kernel NULL pointer dereference at virtual address 00000004
[ 3.050000] pgd = 80004000
[ 3.050000] [00000004] *pgd=00000000
[ 3.050000] Internal error: Oops: 805 [#1] PREEMPT ARM
[ 3.050000] Modules linked in:
[ 3.050000] CPU: 0 PID: 1 Comm: swapper Tainted: G W 4.1.15 #4
[ 3.050000] Hardware name: Freescale i.MX6 Ultralite (Device Tree)
[ 3.050000] task: 8202c000 ti: 82030000 task.ti: 82030000
[ 3.050000] PC is at caam_sm_startup+0x7c/0x3b0
[ 3.050000] LR is at device_add+0x1d0/0x51c
[ 3.050000] pc : [<8040beec>] lr : [<80288714>] psr: a0000013
[ 3.050000] sp : 82031ec8 ip : 00000000 fp : 00000000
[ 3.050000] r10: 00000081 r9 : 80784b50 r8 : 831ae980
[ 3.050000] r7 : 8080a398 r6 : 831b5400 r5 : 00000000 r4 : 8317d240
[ 3.050000] r3 : 831b5410 r2 : 00000000 r1 : 00000001 r0 : 831b5400
[ 3.050000] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
[ 3.050000] Control: 10c53c7d Table: 80004059 DAC: 00000015
[ 3.050000] Process swapper (pid: 1, stack limit = 0x82030208)
[ 3.050000] Stack: (0x82031ec8 to 0x82032000)
[ 3.050000] 1ec0: 83e98410 8053dfa4 83f1e20c 80773580 00000000 8080a398
[ 3.050000] 1ee0: 8080a398 831ae980 80784b50 00000081 00000000 807735c4 00000000 80009700
[ 3.050000] 1f00: 00000000 8068ce08 8300ad80 8054ac28 8202a900 808882d4 80845000 00000000
[ 3.050000] 1f20: 00000000 60000053 00000000 8080f1f0 83fff5e5 83fff569 80722edc 800423dc
[ 3.050000] 1f40: 806ceb64 807226cc 00000006 00000006 8080f1c8 8077e644 00000006 8077e624
[ 3.050000] 1f60: 80845000 8074e588 80784b50 00000081 00000000 8074ed18 00000006 00000006
[ 3.050000] 1f80: 8074e588 80046b64 00000000 8053e184 00000000 00000000 00000000 00000000
[ 3.050000] 1fa0: 00000000 8053e18c 00000000 8000f448 00000000 00000000 00000000 00000000
[ 3.050000] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 3.050000] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000 fffffffd fbfffbff
[ 3.050000] [<8040beec>] (caam_sm_startup) from [<807735c4>] (caam_sm_init+0x44/0x5c)
[ 3.050000] [<807735c4>] (caam_sm_init) from [<80009700>] (do_one_initcall+0x80/0x1d0)
[ 3.050000] [<80009700>] (do_one_initcall) from [<8074ed18>] (kernel_init_freeable+0xe8/0x1b0)
[ 3.050000] [<8074ed18>] (kernel_init_freeable) from [<8053e18c>] (kernel_init+0x8/0xe4)
[ 3.050000] [<8053e18c>] (kernel_init) from [<8000f448>] (ret_from_fork+0x14/0x2c)
[ 3.050000] Code: 0a0000be e5846008 e2863010 e586405c (e5853004)
[ 3.270000] ---[ end trace 6377d0690b443d13 ]---
[ 3.280000] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[ 3.280000]
[ 3.280000] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[ 3.280000]

If i only boot uboot->kernel, it works fine.
Or when I disable caam in linux, then boot uboot->tee->linux, it also works fine. So it seems when i run tee, caam cannot be enabled in linux ?

I print the value of two caam regs in linux, one is CTPR, the other is SCFGR.
I find that when running tee, value of SCFGR is 0x0; when not running tee, value of SCFGR is 0x1.
Value of CTPR is always 0x1914200 when running or not running tee.

So I try to trace value of SCFGR in tee, I find that its value will be changed after mmu and cache enabled.
void core_init_mmu_regs(void)
{
.......
/* read value at the end of function */
DMSG("scfgr: 0x%x, comp_param: 0x%x.", read32(0x214000c), read32(0x2146FA8));
}
static int get_scfgr(void)
{
uint32_t scfgr;

scfgr = (uint32_t)phys_to_virt(0x214000c, MEM_AREA_IO_NSEC);
scfgr = *(volatile uint32_t *)scfgr;

return scfgr;

}

static void init_primary_helper(unsigned long pageable_part,
unsigned long nsec_entry, unsigned long fdt)
{
uint32_t scfgr;
/*
* Mask asynchronous exceptions before switch to the thread vector
* as the thread handler requires those to be masked while
* executing with the temporary stack. The thread subsystem also
* asserts that IRQ is blocked when using most if its functions.
*/
scfgr = get_scfgr();
thread_set_exceptions(THREAD_EXCP_ALL);
init_vfp_sec();

init_runtime(pageable_part);

IMSG("Initializing (%s), scfgr %x\n", core_v_str, scfgr);
    ......

}
The result shows:
DEBUG: TEE-CORE:core_init_mmu_regs:766: scfgr: 0x1, comp_param: 0x1914200.
INFO: TEE-CORE: Initializing (2.3.0-dev #33 Sat Mar 11 03:02:49 UTC 2017 arm), scfgr 0

Now I don't know how to debug this, could anyone help me?
Thanks!

[MrVan]

@xfeng12 when you try tee, needs to disable CAAM in linux side. Need some configuration to make caam workable in Linux side. I have not finialized my caam part to upstream. If you are in a rush to have a try,
configure the jobrings in optee to make it accessible in linux side, move the random number generator initialization from kernel to uboot, you could just use sec_init in uboot. then omit the trng initialization in kernel caam probe.

[xfeng12]

Hi MrVan,
Got it. Thanks very much for your answer.
I think i had to learn more about CAAM and then i will have a try.

[0xB0D]

Hi @MrVan

I'm seeing the same issue with CAAM as described here
u-boot -> Kernel directly is fine
u-boot -> OPTEE -> Kernel fails as described above.

You said you were working on patches - is there any progress on that ? I'm not entirely clear if you are suggesting both of these or just one ?

• Configure the jobrings in OPTEE
• Move the random number generator init to u-boot and skip in kernel ?

[0xB0D]

Poking about about I've managed to get this working.

There are two things that need to happen on MX7 at least for me

#1 Assign ownership of the job-ring registers to non-secure mode - prior to enabling trust zone
In u-boot I do this.

diff --git a/drivers/crypto/fsl/jr.c b/drivers/crypto/fsl/jr.c
index 986eabf..ad2bd43 100644
--- a/drivers/crypto/fsl/jr.c
+++ b/drivers/crypto/fsl/jr.c
@@ -573,6 +573,15 @@ int sec_init_idx(uint8_t sec_idx)
uint32_t liodn_ns;
uint32_t liodn_s;
#endif
+ uint32_t ns_toggle;
+
+ printf("JR0 ms=0x%08x ls=0x%08x\n",
+ sec->jrliodnr[0].ms,
+ sec->jrliodnr[0].ls);
+
+ printf("JR1 ms=0x%08x ls=0x%08x\n",
+ sec->jrliodnr[1].ms,
+ sec->jrliodnr[1].ls);

if (!(sec_idx < CONFIG_SYS_FSL_MAX_NUM_OF_SEC)) {
printf("SEC initialization failed\n");
@@ -621,6 +630,22 @@ int sec_init_idx(uint8_t sec_idx)
#endif
#endif

+ ns_toggle = sec_in32(&sec->jrliodnr[0].ms);
+ ns_toggle |= 0x08;
+ sec_out32(&sec->jrliodnr[0].ms, ns_toggle);
+
+ ns_toggle = sec_in32(&sec->jrliodnr[1].ms);
+ ns_toggle |= 0x08;
+ sec_out32(&sec->jrliodnr[1].ms, ns_toggle);
+
+ printf("JR0 ms=0x%08x ls=0x%08x\n",
+ sec->jrliodnr[0].ms,
+ sec->jrliodnr[0].ls);
+
+ printf("JR1 ms=0x%08x ls=0x%08x\n",
+ sec->jrliodnr[1].ms,
+ sec->jrliodnr[1].ls);
+

obvious you need to run sec_init()

#2 Skip RNG initialisation in Linux

diff --git a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c
index 98468b9..a79e11e 100644
--- a/drivers/crypto/caam/ctrl.c
+++ b/drivers/crypto/caam/ctrl.c
@@ -693,6 +693,7 @@ static int caam_probe(struct platform_device *pdev)
kick_trng(pdev, ent_delay);
ent_delay += 400;
}
+ #if 0
/*
* if instantiate_rng(...) fails, the loop will rerun
* and the kick_trng(...) function will modfiy the
@@ -708,6 +709,9 @@ static int caam_probe(struct platform_device *pdev)
* so don't hog the CPU
*/
cpu_relax();
+ #else
+ dev_info(dev, "Relying on bootloader CAAM RNG init\n");
+ #endif
} while ((ret == -EAGAIN) && (ent_delay < RTSDCTL_ENT_DLY_MAX));
if (ret) {
dev_err(dev, "failed to instantiate RNG");

The tricky thing here was working out the job-ring register's ownership needs to be set before OPTEE/TrustZone runs.

[Edited by admin to make patch readable]