Suspected problem in the Zephyr SDK 0.16.1 ARC 64 compiler or QEMU

[peter-mitsis]

Tests introduced as part of PR #59075 (Introducing Object Cores) have been failing in an unusual manner on the qemu_arc_hs6x board. After an otherwise successful test run, the obj_core_stats_api test fails with the following error message being reported.

Observed Error

TESTSUITE obj_core_stats_api succeeded
ERROR: Test 'test_obj_core_stats_deregister' assigned to test suite 'obj_core_stats_api' which doesn't exist

To duplicate

west build -p -b qemu_arc_hs6x ./tests/kernel/obj_core/obj_core_stats_api/ -t run

Zephyr SDK Version: 0.16.1
Host machine: Ubuntu 22.04

The adventure so far ...

This error message comes from the ztest routine ztest_verify_all_test_suites_ran() in subsys/testsuite/ztest/src/ztest_new.c after the call to ztest_find_test_suite(test->test_suite_name); returns NULL. For convenience, the relevant code is pasted below.

                for (test = _ztest_unit_test_list_start; test < _ztest_unit_test_list_end; ++test) {
                        suite = ztest_find_test_suite(test->test_suite_name);
                        if (suite == NULL) {
                                PRINT("ERROR: Test '%s' assigned to test suite '%s' which doesn't "
                                      "exist\n",
                                      test->name, test->test_suite_name);
                                all_tests_run = false;
                        }
                }

Adding printk("'%s'(%p) vs '%s'(%p)\n", name, name, node->name, node->name); within ztest_find_test_suite() shows the following when the test is executed ...

TESTSUITE obj_core_stats_api succeeded
''(0x804075d0) vs 'obj_core_stats_api'(0x80407900)
ERROR: Test 'test_obj_core_stats_deregister' assigned to test suite 'obj_core_stats_api' which doesn't exist
'obj_core_stats_api'(0x80407900) vs 'obj_core_stats_api'(0x80407900)
'obj_core_stats_api'(0x80407900) vs 'obj_core_stats_api'(0x80407900)
'obj_core_stats_api'(0x80407900) vs 'obj_core_stats_api'(0x80407900)
'obj_core_stats_api'(0x80407900) vs 'obj_core_stats_api'(0x80407900)
'obj_core_stats_api'(0x80407900) vs 'obj_core_stats_api'(0x80407900)
'obj_core_stats_api'(0x80407900) vs 'obj_core_stats_api'(0x80407900)

The address 0x804075d0 is of particular significance. It corresponds to _ztest_unit_test_list_end -- the address used in the comparison that terminates the for-loop that has been pasted above.

Getting down into the generated assembly, we can observe the following ...

    804020c4:   5e0a 1f80 8040 74b8     movl    r14,0x804074b8  #  Corresponds to _ztest_unit_test_list_start
    804020cc:   580a 0f80 8040 75d0     movl    r0,0x804075d0  # Corresponds to _ztest_unit_test_list_end
    804020d4:   0e53 100c               brlol   r14,r0,82       ;80402126 <ztest_verify_all_test_suites_ran+0x7e>
...
    80402126:   09a6 ffef               bl.d    -1628   ;80401ac8 <ztest_find_test_suite>
    8040212a:   1600 1840               ldl     r0,[r14]
...

I am interpreting the above as indicating that register R0 contains the parameter to ztest_find_test_suite() which on the first pass is _ztest_unit_test_list_end (something consistent with the previous observations). Today is admittedly the first day that I have tried my hand at parsing ARC64 assembly, so there are likely subtleties of which I am unaware. That being said, the instructions at addresses 0x80402126 and 0x8040212a to my eye look like candidates for being out-of-order (either being placed out of order by the compiler, or via QEMU being executed out of order).

Additional Note
Although I have note included the details here, I have observed that if I place a printk() statement before the call to ztest_find_test_suite(), that the tests pass.

[evgeniy-paltsev]

Hi @peter-mitsis,

I'll look into the issue, but may I firstly ask you to check (preferably with same Zephyr binary) if this is still an issue when you are using updated ARC QEMU from zephyrproject-rtos/sdk-ng#692.

There are lot's of fixes (including fix of buggy delay slot implementation which was causing multiple tricky runtime isssues).

You can download SDK with this update from artefacts (https://github.com/zephyrproject-rtos/sdk-ng/suites/14655986494/artifacts/831708523) or you can download only hosttools (https://github.com/zephyrproject-rtos/sdk-ng/suites/14655986494/artifacts/831708481)

Thanks!

[peter-mitsis]

@evgeniy-paltsev - I downloaded and installed the SDK as indicated and set ZEPHYR_SDK_INSTALL_DIR to point the installation and re-ran the test. With that particular SDK, it could not find QEMU, so I manually ran the zephyr.elf generated with the downloaded SDK on the 0.16.1 QEMU for arc64. In this hybrid setup, I observed the same original error.

[github-actions]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.

[tejlmand]

Fixed with Zephyr-sdk 0.16.3