This README is the canonical install reference. For online install (managed venv, newest API-compatible core/plugins; serialwrap pinned), see Quick Start. For offline environments or update procedures, see Managed Install and Update. After any install or update, run testpilot --verify-install to confirm health.
testpilot is the host runtime; test capabilities are provided by plugins.
Use testpilot list-plugins to see installed plugins and testpilot run <plugin>
to drive them.
The canonical project version is VERSION; release tags use vX.Y.Z.
testpilot --version also inventories every installed testpilot.plugins
entry point so operators can see the effective core/plugin/API combination:
TestPilot <core-version> (<source-ref>)
plugin wifi_llapi <plugin-version> (api <api-version>)
Plugin-based test automation framework for embedded device verification (prplOS / OpenWrt).
This repository is TestPilot core: the deterministic verdict kernel, the
plugin SDK (testpilot.api), the CLI host, transports, schema, and reporting.
It ships the plugin scaffold under plugins/_template/ but no concrete test
plugins. Real plugins (for example wifi_llapi and brcm_fw_upgrade) live in
their own repositories and register themselves through the
testpilot.plugins entry-point group.
TestPilot is a plugin-based test automation framework for prplOS / OpenWrt embedded devices. The architecture splits into two planes:
- Deterministic verdict kernel — test execution, evidence collection, pass/fail verdicts, and report projection.
- Copilot SDK control plane — per-case session foundation, lifecycle hooks, advisory audit, tiered environment recovery, and extension surfaces such as custom agents / skills / selective MCP.
Core principle: the Copilot SDK handles the control plane; it does NOT decide the final verdict.
Current landed control-plane subset today:
- per-case runner selection with
selection_trace - best-effort per-case Copilot session foundation
- lifecycle hook dispatch (
pre_case,post_case,pre_step,post_step,on_failure,on_retry) - advisory collection plus tier-1 deterministic remediation and opt-in tier-2 one-shot planning between retry attempts
Tier-2 is reached only after the configured consecutive tier-1 failures. Core
builds a bounded, tool-denied one-shot prompt from the plugin's advertised
environment capabilities; the plugin executes the validated plan, then core
forces deterministic verify_env. The agent never receives verdict authority
or permission to rewrite case semantics. Every intervention is marked
agent_recovered and retained in bounded/redacted case and run audit artifacts;
the marker means the agent intervened, not that the verification gate passed.
See Tier-2 Environment Recovery Design.
Custom agents / skills / MCP remain extension surfaces in the current codebase rather than default hot-path runtime wiring.
- Python 3.11+
- git
- uv — Python package manager, preferred by the installer
- serialwrap — UART serial multiplexer for DUT / STA communication; managed installs install/update it automatically
Developer checkouts that manage serialwrap manually can still set the binary path via environment variable:
export SERIALWRAP_BIN=/path/to/serialwrapOr add to configs/testbed.yaml:
testbed:
serialwrap_binary: /path/to/serialwrapResolution order:
SERIALWRAP_BINenv var →testbed.yamlconfig → error exit.
Online one-click install (requires a fine-grained read-only PAT with contents: read
for hamanpaul/testpilot-core and all plugin repos):
TESTPILOT_INSTALL_TOKEN=<fine-grained read-only PAT> bash scripts/install.sh
testpilot --verify-install
testpilot list-pluginsInstall only a subset of plugins with --plugins:
TESTPILOT_INSTALL_TOKEN=<PAT> bash scripts/install.sh --plugins wifi_llapiOnce installed, list and run plugins:
testpilot list-plugins
testpilot list-cases <plugin>
testpilot run <plugin>When a plugin context is resolved, the CLI auto-stages
plugins/<plugin>/testbed.yaml.exampleintoconfigs/testbed.yaml, so each run starts from that plugin's shipped testbed shape with no leakage between plugins. Editconfigs/testbed.yamlto match your lab — note that switching to a different plugin will overwrite it.
The supported QC/TEST install uses a managed venv (no git checkout). Online
install resolves the newest API-compatible core and plugin releases;
serialwrap stays pinned in install-manifest.yaml. An offline bundle is an
exact, SHA256-verified snapshot. Pin any component explicitly with
--plugins <name>@<version> (or by adding a version: back to the manifest):
~/.local/share/testpilot/.venv # managed runtime virtualenv
~/.local/bin/testpilot # wrapper, no activation required
~/.agents/skills/testpilot-normal-testOnline install — downloads wheels via gh release download using a fine-grained PAT:
TESTPILOT_INSTALL_TOKEN=<fine-grained read-only PAT> bash scripts/install.shOffline install — build a bundle on a networked Linux box with scripts/build-bundle.sh,
then transfer and install on the air-gapped machine:
# Build on a networked machine:
bash scripts/build-bundle.sh
# Install on the offline machine (verifies SHA256SUMS, installs with --no-index):
bash scripts/install.sh --offline testpilot-bundle-<ver>-linux-<arch>-cp<XY>.tar.gzUpdate and verify:
testpilot --update # reinstalls newest-compatible core/plugins (serialwrap pinned), reconciles plugins
testpilot --verify-install # report install health--update snapshots the environment first; if the post-update verify fails it
rolls back offline only (pip install --no-index --find-links against the
local wheel cache the installer preserves under
${TESTPILOT_HOME:-~/.local/share/testpilot}/.wheel-cache) — it never reaches a
public index. If the cached wheels are insufficient, rollback fails loudly and
asks you to reinstall from a known-good bundle via install.sh --offline.
Use the installed testpilot command for normal operation. Developer checkouts can
still use python -m testpilot.cli when debugging the repository.
Plugin-owned CLI commands are registered from installed plugin packages when
testpilot.cli is imported. --root <path> selects the runtime project root for
cases/configs/reports; it does not dynamically replace the registered plugin CLI
surface with commands from <path>/plugins.
The core host commands are:
testpilot --version
testpilot list-plugins
testpilot list-cases <plugin>
testpilot run <plugin>Usage: testpilot [OPTIONS] COMMAND [ARGS]...
TestPilot — plugin-based test automation for embedded devices.
Options: --version Show version and exit. -v, --verbose Enable debug logging. --root DIRECTORY Project root directory. --update REF Reinstall and reconcile the managed wheel install from its pinned manifest, then exit. REF is accepted but cross- version update is not yet implemented; the currently- pinned set is reinstalled. --verify-install Report managed install health and exit. --help Show this message and exit.
Commands: install-doctor Check manifest plugin API-compat against installed core... list-cases List test cases for a plugin. list-plugins List available test plugins. run Run tests for a plugin.
Usage: testpilot [OPTIONS] COMMAND [ARGS]...
TestPilot — plugin-based test automation for embedded devices.
Options: --version Show version and exit. -v, --verbose Enable debug logging. --root DIRECTORY Project root directory. --update REF Reinstall and reconcile the managed wheel install from its pinned manifest, then exit. REF is accepted but cross- version update is not yet implemented; the currently- pinned set is reinstalled. --verify-install Report managed install health and exit. --help Show this message and exit.
Commands: install-doctor Check manifest plugin API-compat against installed core... list-cases List test cases for a plugin. list-plugins List available test plugins. run Run tests for a plugin.
Repository skills for agent-assisted workflows live under skills/.
TestPilot core automatically uses Azure when all required values are present.
Without an API key it runs in deterministic/no-agent mode; a key without an
endpoint or deployment produces a non-blocking misconfiguration notice. If
COPILOT_PROVIDER_TYPE is set to a non-azure value, core also treats the
runtime as misconfigured and emits the same redacted notice.
export COPILOT_PROVIDER_BASE_URL=https://your-resource.openai.azure.com
export COPILOT_PROVIDER_API_KEY='<set in shell profile or secret store>'
export COPILOT_MODEL=your-deployment-name
export COPILOT_PROVIDER_AZURE_API_VERSION=2024-10-21
testpilot run <plugin_name>COPILOT_PROVIDER_TYPE is not an enable switch; core constructs only the Azure
provider and rejects non-azure values as misconfiguration. Azure deployment
selection is independent of plugin runner labels. Per-case planning is
advisory, tier-2 recovery requires plugin opt-in, and deterministic remediation
remains plugin-owned. Core-owned usage and observational benefit metrics are
written under artifact_dir/agent_usage; shared run-analysis tokens are not
allocated to cases. Custom/skeleton runners report unsupported_execution_path
and make no core model calls.
Copy the SDK scaffold and register it from your plugin package:
cp -r plugins/_template plugins/my_plugin[project.entry-points."testpilot.plugins"]
my_plugin = "plugins.my_plugin.plugin:Plugin"uv pip install -e .
testpilot list-pluginsImplement the PluginBase contract (declare api_version, name,
discover_cases(), execute_step(), evaluate(); override optional hooks such
as setup_env(), verify_env(), teardown(), create_reporter(),
create_runner(), and register_cli() as needed). Plugins import the public
SDK surface from testpilot.api; they must not reach into testpilot.core,
testpilot.schema, testpilot.reporting, testpilot.transport, or
testpilot.runtime internals. See plugins/_template/README.md for the full
contract and optional hook table.
For a complete, runnable reference (installable + discoverable via
testpilot.plugins), see examples/sample_echo/ and its README.
src/testpilot/
api/ # public plugin SDK surface (testpilot.api)
core/ # orchestrator, plugin_base, plugin_loader, testbed_config
reporting/ # reporter and report helpers
transport/ # transport abstractions
schema/ # YAML case schema validation
runtime/ # run backend
plugins/
_template/ # plugin SDK scaffold (not discoverable on its own)
configs/ # operator-local effective testbed.yaml (auto-staged by CLI; git-ignored)
docs/ # plan, todos, phase docs
scripts/ # utility scripts (install, policy_cli_help, gen_cases)
skills/ # repository agent skills
tests/ # core test suite
The canonical project version lives in VERSION; pyproject.toml and
src/testpilot/__init__.py mirror it and must stay identical. Release tags use
Semantic Versioning vX.Y.Z. User-facing pull requests should update
CHANGELOG.md under Unreleased, or explicitly record why no changelog entry
is needed. testpilot --version prints this core version followed by every
installed plugin distribution version and declared SDK API version; a broken
plugin metadata record is shown as unknown without hiding the remaining
inventory.
針對嵌入式裝置驗證（prplOS / OpenWrt）的 plugin 化測試自動化框架。
本 repo 是 TestPilot core：deterministic verdict kernel、plugin SDK
（testpilot.api）、CLI host、transport、schema 與 reporting。它內含
plugins/_template/ plugin 骨架，但不含任何具體測試 plugin。真正的 plugin
（例如 wifi_llapi、brcm_fw_upgrade）各自存在於獨立 repo，透過
testpilot.plugins entry-point group 自我註冊。
TestPilot 是針對 prplOS / OpenWrt 嵌入式裝置的 plugin 化測試自動化框架，架構分為兩個平面：
- Deterministic verdict kernel — 測試執行、證據蒐集、pass/fail 判定與報表投影。
- Copilot SDK control plane — per-case session foundation、lifecycle hooks、advisory audit、分層環境修復，以及 custom agents / skills / 選擇性 MCP 等擴充面。
核心原則：Copilot SDK 負責 control plane；它不決定最終 verdict。
tier-1 先執行 plugin 的 deterministic allowlist 修復；只有達到設定的連續失敗門檻，
才會在 retry 間隙 opt-in 升級 tier-2。core 以 plugin 宣告的 environment capability
catalog 建立 bounded、tool-denied one-shot prompt，plugin 執行通過 schema/budget 的
plan 後，core 一律強制重跑 deterministic verify_env。agent 不具 verdict 權限，也
不能改 case semantics。任何介入都以 agent_recovered 標記並寫入已去敏、有上限的
case/run audit；該標記只代表 agent 曾介入，不代表 verify gate 已通過。詳見
Tier-2 Environment Recovery Design。
- Python 3.11+
- git
- uv — 安裝器優先採用的 Python 套件管理器
- serialwrap — DUT / STA 通訊用的 UART serial multiplexer；managed install 會自動安裝/更新
手動管理 serialwrap 的開發者 checkout 可用環境變數指定路徑：
export SERIALWRAP_BIN=/path/to/serialwrap解析順序：
SERIALWRAP_BIN環境變數 →testbed.yaml設定 → 失敗結束。
線上一鍵安裝（需要 fine-grained read-only PAT，具備 hamanpaul/testpilot-core
及所有 plugin repo 的 contents: read 權限）：
TESTPILOT_INSTALL_TOKEN=<fine-grained read-only PAT> bash scripts/install.sh
testpilot --verify-install
testpilot list-plugins透過 --plugins 只安裝部分 plugin：
TESTPILOT_INSTALL_TOKEN=<PAT> bash scripts/install.sh --plugins wifi_llapi安裝後，列出並執行 plugin：
testpilot list-plugins
testpilot list-cases <plugin>
testpilot run <plugin>支援的 QC/TEST 安裝採用 managed venv 加 pinned wheels（不再使用 git checkout）：
~/.local/share/testpilot/.venv # managed runtime virtualenv
~/.local/bin/testpilot # wrapper，免 activate
~/.agents/skills/testpilot-normal-test線上安裝：
TESTPILOT_INSTALL_TOKEN=<fine-grained read-only PAT> bash scripts/install.sh離線安裝（先在有網路的 Linux 機器以 scripts/build-bundle.sh 建置 bundle，
再傳至目標機器）：
# 有網路機器上建置：
bash scripts/build-bundle.sh
# 離線機器上安裝（驗證 SHA256SUMS，以 --no-index 安裝）：
bash scripts/install.sh --offline testpilot-bundle-<ver>-linux-<arch>-cp<XY>.tar.gz更新與驗證：
testpilot --update # 重解析 manifest、重裝 pinned wheels、同步 plugin
testpilot --verify-install # 回報安裝狀態--update 會先快照環境；若更新後 verify 失敗，rollback 只走離線
（以 pip install --no-index --find-links 對安裝器保存在
${TESTPILOT_HOME:-~/.local/share/testpilot}/.wheel-cache 的本地 wheel cache 重裝），
絕不連到 public index。若 cache 內 wheel 不足，rollback 會明確失敗並提示改用
install.sh --offline 從已知良好的 bundle 重裝。
正常操作使用已安裝的 testpilot 指令；開發者 checkout 在除錯 repo 時仍可使用
python -m testpilot.cli。Plugin 專屬的 CLI 指令是在 import testpilot.cli
時由已安裝的 plugin 套件註冊。--root <path> 只選擇 runtime project root
（cases/configs/reports），不會用 <path>/plugins 的指令動態替換已註冊的
plugin CLI 介面。
核心 host 指令：
testpilot --version
testpilot list-plugins
testpilot list-cases <plugin>
testpilot run <plugin>當 Azure endpoint、API key 與 deployment 都存在時，TestPilot core 自動啟用
Azure；沒有 key 時使用 deterministic/no-agent mode，缺少其他欄位只會產生
非阻斷的 misconfigured notice；若 COPILOT_PROVIDER_TYPE 被設成非 azure
值，也會被視為 misconfigured。COPILOT_PROVIDER_TYPE 不作為啟用開關，core
只建立 Azure provider。
export COPILOT_PROVIDER_BASE_URL=https://your-resource.openai.azure.com
export COPILOT_PROVIDER_API_KEY='<set in shell profile or secret store>'
export COPILOT_MODEL=your-deployment-name
export COPILOT_PROVIDER_AZURE_API_VERSION=2024-10-21
testpilot run <plugin_name>每案 planning 僅供 advisory；tier-2 需 plugin opt-in，deterministic
remediation 仍由 plugin 負責。core 報表位於 artifact_dir/agent_usage，共享
run-end analysis token 不分攤到個案；custom/skeleton path 回報
unsupported_execution_path 且不呼叫 core model。效益指標為 observational，
不宣稱因果 uplift/regression。
複製 SDK 骨架並從你的 plugin 套件註冊：
cp -r plugins/_template plugins/my_plugin[project.entry-points."testpilot.plugins"]
my_plugin = "plugins.my_plugin.plugin:Plugin"uv pip install -e .
testpilot list-plugins實作 PluginBase 契約（宣告 api_version、name、discover_cases()、
execute_step()、evaluate()；視需要覆寫 setup_env()、verify_env()、
teardown()、create_reporter()、create_runner()、register_cli() 等選用
hook）。Plugin 從 testpilot.api 匯入公開 SDK 介面，不得直接 reach 進
testpilot.core / schema / reporting / transport / runtime 內部。完整
契約與選用 hook 對照表見 plugins/_template/README.md。
canonical 版本位於 VERSION；pyproject.toml 與
src/testpilot/__init__.py 為鏡像，必須一致。release tag 採 Semantic
Versioning vX.Y.Z。對外 PR 應更新 CHANGELOG.md 的 Unreleased，或明確記錄
為何不需 changelog entry。testpilot --version 會在 core 版本後列出所有已安裝
plugin 的 distribution version 與宣告的 SDK API version；單一 plugin metadata
損壞時以 unknown 顯示，不會阻斷其餘 inventory。