tests: add Python CI + unit test framework#42
Merged
Conversation
Add a pure-Python test framework for QEMU-based functional testing, replacing the need for shell script test runners. Framework library (tests/functional/rtclaw_test/): - platform.py: PlatformConfig dataclass for esp32c3/esp32s3/vexpress-a9, QEMU binary resolution (Espressif fork first, then PATH fallback), and command-line builder - cmd.py: ConsoleBuffer with background stdout reader thread, wait_for_console_pattern(), exec_command(), ANSI escape stripping - testcase.py: RTClawQemuTest base class with setUp/tearDown lifecycle (flash image isolation via tempdir, SIGTERM/SIGKILL cleanup, restart_qemu() for persistence tests) Test suites: - test_boot.py: banner and shell prompt verification (all platforms) - test_shell.py: 8 shell command tests (/help, /log, /history, /clear, /remember + /memories, /ai_status) for ESP32 platforms - test_kv_persistence.py: KV data survives QEMU reboot via restart_qemu() - test_ai_online.py: AI boot connectivity (skipped without API key) Signed-off-by: Chao Liu <chao.liu.zevorn@gmail.com>
Cross-compile unit tests with the vexpress-a9 toolchain, link against the real RT-Thread OSAL, and run in QEMU with ARM semihosting for automated pass/fail via exit code. Framework (tests/unit/framework/): - test.h: minimal TEST_ASSERT macros and RUN_TEST/TEST_BEGIN/TEST_END - semihosting.h: ARM SYS_EXIT_EXTENDED via svc #0x123456 Test suites (22 tests total): - test_ai_memory.c: 11 tests — short-term memory (init, add, count, channel isolation, clear, JSON build) + long-term memory (init, save/load, delete, overflow, build_context) - test_gateway.c: 4 tests — init, register service, stats, overflow - test_tools.c: 7 tests — init, register, find, get_by_index, to_json, to_json_exclude, overflow Build integration: - meson.build: add tests/unit subdir producing libunittest.a - SConscript: RTCLAW_UNIT_TEST=1 env var adds -DRTCLAW_UNIT_TEST to CFLAGS, links libunittest.a with --start-group/--end-group - main.c: #ifdef RTCLAW_UNIT_TEST selects test runner or normal boot - run.py: Python orchestrator (Meson compile + SCons link + QEMU run) Usage: make test-unit Signed-off-by: Chao Liu <chao.liu.zevorn@gmail.com>
Remove the legacy shell-based test runner (tests/qemu/) and unify all test orchestration through the Python functional + unit test frameworks. Removed: - tests/qemu/common.sh (QEMU helpers, profile switching, log assertion) - tests/qemu/run-smoke.sh (shell smoke tests) - tests/qemu/run-online.sh (shell AI online tests) - tests/qemu/assert-log.sh (grep-based log assertion) - Makefile swarm-test target (script no longer exists) Added Makefile targets: - Quick targets (pre-built firmware): test-functional, test-boot, test-shell, test-persist, test-online, test-unit - CI pipeline targets (profile switch + build + test): test-smoke-esp32c3, test-smoke-esp32s3, test-smoke-vexpress, test-online-esp32c3, test-online-esp32s3 - run-esp32s3-qemu-flash (symmetric with C3) Signed-off-by: Chao Liu <chao.liu.zevorn@gmail.com>
Replace shell script invocations (tests/qemu/run-smoke.sh,
tests/qemu/run-online.sh) with Makefile targets that use the
new Python functional test framework.
Smoke: tests/qemu/run-smoke.sh → make test-smoke-{esp32c3,esp32s3,vexpress}
Online: tests/qemu/run-online.sh → make test-online-{esp32c3,esp32s3}
Signed-off-by: Chao Liu <chao.liu.zevorn@gmail.com>
zevorn
force-pushed
the
feat/test-framework
branch
2 times, most recently
from
ac65684 to
f9ad543
Compare
unittest discover needs -s to set the correct sys.path for the rtclaw_test package. Use discover -p <glob> instead of dotted module names which require __init__.py in parent directories. Smoke: -p 'test_[bsk]*.py' matches test_boot, test_shell, test_kv_* Boot: -p 'test_boot.py' Online: -p 'test_ai_online.py' Signed-off-by: Chao Liu <chao.liu.zevorn@gmail.com>
zevorn
force-pushed
the
feat/test-framework
branch
from
f9ad543 to
2ad2c19
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Functional tests (
tests/functional/): Python unittest framework that launches QEMU subprocess, interacts via stdin/stdout PIPE, and asserts console output patterns. Supports esp32c3-qemu, esp32s3-qemu, vexpress-a9-qemu platforms.test_boot.py: banner + shell prompt (2 tests)test_shell.py: /help, /log, /history, /clear, /remember, /ai_status (8 tests)test_kv_persistence.py: /remember survives QEMU reboot (1 test)test_ai_online.py: AI boot connectivity, skipped without API key (1 test)Unit tests (
tests/unit/): Cross-compiled C tests running on RT-Thread vexpress-a9 QEMU with ARM semihosting exit code for pass/fail.test_ai_memory.c: short-term + long-term memory (11 tests)test_gateway.c: service registry + stats (4 tests)test_tools.c: tool register/find/JSON (7 tests)Legacy removal: Deleted
tests/qemu/shell scripts (common.sh, run-smoke.sh, run-online.sh, assert-log.sh). All CI targets migrated to Python.Makefile targets:
test-unit,test-functional,test-boot,test-shell,test-persist,test-online, plus CI pipeline targets (test-smoke-*,test-online-*).34/34 tests passing (22 unit + 12 functional).
Test plan
make test-unit— 22/22 passedmake test-functional— 12/12 passedmake test-smoke-vexpress— boot test on vexpress-a9make test-smoke-esp32c3— full smoke on ESP32-C3 (requires demo profile rebuild)