-
Notifications
You must be signed in to change notification settings - Fork 8
Add agent scope observation and advice tools #4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Epslion404
wants to merge
2
commits into
Scaxlibur:master
Choose a base branch
from
Epslion404:split/agent-scope-observe-advise
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # WaveBench AI-Agent Experience TODO | ||
|
|
||
| 目标:让 AI agent 能以低风险、结构化、可审计的方式观察实验台、理解已有产物、提出建议,并只通过显式 run plan 执行会改变仪器状态的动作。 | ||
|
|
||
| ## P0: 安全只读上下文 | ||
|
|
||
| - [x] 增加 MCP 工具 `scope.observe`:读取配置中的示波器,返回 IDN、状态快照、通道高阻判断、当前波形摘要;不保存文件、不改仪器状态、不暴露 raw SCPI。 | ||
| - [x] `scope.observe` 支持 CH1-CH4 多通道观察;显式 `fetch_waveform=true` 且获取到两个以上通道时返回 pairwise relationships。 | ||
| - [x] 增加 MCP 工具 `doctor.config`:把现有 `doctor` 结果结构化返回给 agent,支持配置可达性和型号匹配判断。 | ||
| - [ ] 增加 MCP 工具 `net.discover`:结构化返回只读 LAN 仪器发现结果,默认限制网段大小和端口集合。 | ||
| - [ ] 所有 MCP 工具都标注 `read_only`、`mutates_instrument=false`、`raw_scpi=false` 等安全元数据。 | ||
|
|
||
| ## P1: 离线产物理解 | ||
|
|
||
| - [ ] 增强 `capture.inspect`:可选返回 FFT 摘要、质量告警、推荐下一步采集参数,但不读取大数组进响应。 | ||
| - [ ] 增加 `capture.list`:列出 `data/raw` 下最近采集包,供 agent 找上下文。 | ||
| - [ ] 增加 `run.list` / `run.inspect`:列出和解释 `data/runs` 下的实验记录。 | ||
| - [ ] 给采集包生成 agent 友好的 `summary.md` 或结构化 `analysis.json`。 | ||
|
|
||
| ## P2: Agent 建议层 | ||
|
|
||
| - [x] 增加 `scope.advise`:基于 `scope.observe` 的结果给出时基、垂直档位、触发、点数建议;只返回建议,不应用。 | ||
| - [x] 根据实测或期望频率推荐每通道 focus time-range,默认约 10 个周期。 | ||
| - [x] 根据实测 Vpp 推荐垂直档位,默认约占 5 格。 | ||
| - [x] 识别 CH1/CH2 这类大频率跨度,建议分通道/分 profile 观察,避免单时基误判形状。 | ||
| - [x] 增加 `scope.observe.expectations`:把已知闭环信号作为结构化依据,例如 CH1 1 kHz/1 Vpp/50% 方波、CH2 50 kHz/1 Vpp/500 mVdc/30% 对称三角波,并返回逐项 pass/warn/fail。 | ||
| - [x] 支持 frequency/Vpp/mean/duty/symmetry_percent 的基础断言。 | ||
| - [x] 期望断言必须显式 `fetch_waveform=true`,避免 agent 在不知情时触发波形传输状态变化。 | ||
| - [x] 返回总体 expectation status,并把 fail/warn 加入 agent hints。 | ||
| - [x] 增加多时基建议:当多个通道的频率跨度较大时,提醒 agent 不要用单个显示时基同时判断所有通道形状,应分通道或分 profile 采集。 | ||
| - [x] 多通道关系分析增加交点:对每对通道返回交点数量、采样返回点、交点时间/电压和相对斜率方向;交点过多时截断并给 warning。 | ||
| - [ ] 让 `scope.expect` 也参与多时基建议:即使当前频率估计低置信,也能利用用户给定的期望频率识别 CH1/CH2 这种 50x 频率跨度。 | ||
| - [ ] 增加 `plan.propose`:从自然语言目标或结构化目标生成保守 run plan 草案,默认写到 `plans/`,不执行。 | ||
| - [ ] 增加 `plan.explain`:解释某个 run plan 会读写哪些仪器、哪些步骤会改变输出状态、有哪些保护。 | ||
|
|
||
| ## P3: 可控执行边界 | ||
|
|
||
| - [ ] MCP 继续保持默认只读;会改仪器状态的能力只通过显式 `run plan` 文件和人工确认入口暴露。 | ||
| - [x] 增加显式示波器显示控制 CLI:通道显示 on/off、focus 单通道、显式 autoscale;用于人类和 agent 做可审计调参。 | ||
| - [x] focus 动作只调整示波器显示/采集窗口,不改变信号源、电源或被测对象;执行后输出 mutation manifest。 | ||
| - [ ] 对所有会改状态的 plan 步骤生成 mutation manifest,便于 agent 在执行前向人类说明。 | ||
| - [ ] 增加跨进程仪器锁,避免 Windows/WSL/多个 agent 同时打开同一台仪器导致响应串线。 | ||
|
|
||
| ## P4: 人类与 Agent 共用体验 | ||
|
|
||
| - [x] 增加 `scripts/wsl-run.ps1` 作为 Windows 到 WSL 的标准执行入口,并在 README 中记录。 | ||
| - [ ] 为 WSL/Windows 推荐环境增加 `doctor environment` 或 `env doctor`,检查 Python、PyVISA、WSL、网络可达性。 | ||
| - [ ] 增加最小可视化:采集包自动生成 waveform/FFT PNG,报告中可直接查看。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,282 @@ | ||
| from __future__ import annotations | ||
|
|
||
| from typing import Any | ||
|
|
||
| import numpy as np | ||
|
|
||
| from wavebench.instruments.models import WaveformData | ||
|
|
||
|
|
||
| def evaluate_waveform_expectation( | ||
| waveform: WaveformData, | ||
| expectation: dict[str, Any], | ||
| ) -> dict[str, Any]: | ||
| summary = waveform.summary( | ||
| expected_frequency_hz=_optional_positive_float(expectation, "frequency_hz"), | ||
| frequency_tolerance_ratio=float(expectation.get("frequency_tolerance_ratio", 0.05)), | ||
| ) | ||
| checks: list[dict[str, Any]] = [] | ||
| _check_frequency(summary, expectation, checks) | ||
| _check_vpp(summary, expectation, checks) | ||
| _check_mean(summary, expectation, checks) | ||
| _check_duty(summary, expectation, checks) | ||
| _check_symmetry(waveform, expectation, checks) | ||
| statuses = {check["status"] for check in checks} | ||
| if "fail" in statuses: | ||
| status = "fail" | ||
| elif "warn" in statuses: | ||
| status = "warn" | ||
| else: | ||
| status = "pass" | ||
| return { | ||
| "status": status, | ||
| "channel": waveform.channel, | ||
| "label": expectation.get("label"), | ||
| "shape": expectation.get("shape"), | ||
| "checks": checks, | ||
| } | ||
|
|
||
|
|
||
| def expectation_summary(results: dict[int, dict[str, Any]]) -> dict[str, Any]: | ||
| statuses = {result["status"] for result in results.values()} | ||
| if "fail" in statuses: | ||
| status = "fail" | ||
| elif "warn" in statuses: | ||
| status = "warn" | ||
| else: | ||
| status = "pass" if results else "skipped" | ||
| return { | ||
| "status": status, | ||
| "channels": {str(channel): result["status"] for channel, result in sorted(results.items())}, | ||
| } | ||
|
|
||
|
|
||
| def estimate_triangle_symmetry_percent(waveform: WaveformData) -> float | None: | ||
| times = waveform.times_s | ||
| values = np.asarray(waveform.voltages_v, dtype=np.float64) | ||
| if times.size != values.size or values.size < 8: | ||
| return None | ||
| span = float(np.max(values) - np.min(values)) | ||
| if span <= 1e-12: | ||
| return None | ||
| centered = values - float(np.mean(values)) | ||
| diffs = np.diff(centered) | ||
| if diffs.size < 3: | ||
| return None | ||
| signs = np.sign(diffs) | ||
|
Owner
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P2] 这里直接根据相邻样本差分的符号变化寻找局部极值,对真实示波器噪声和量化误差非常敏感。验证中,30% symmetry、1 Vpp 的三角波加入 1 mV 噪声后被估计为约 5.56%,加入 5 mV 后变成约 50%。目前的理想无噪声测试无法暴露这个问题。 建议先进行受控平滑或滞回处理,并利用期望频率限制相邻周期/极值间距;也可以改为按完整周期做鲁棒斜率或峰谷拟合。请增加包含噪声、量化台阶和轻微过冲的测试样本,避免 expectation 在真实波形上随机 pass/fail。 |
||
| for index in range(1, signs.size): | ||
| if signs[index] == 0: | ||
| signs[index] = signs[index - 1] | ||
| maxima = [ | ||
| index | ||
| for index in range(1, values.size - 1) | ||
| if signs[index - 1] > 0 and signs[index] < 0 | ||
| ] | ||
| minima = [ | ||
| index | ||
| for index in range(1, values.size - 1) | ||
| if signs[index - 1] < 0 and signs[index] > 0 | ||
| ] | ||
| fractions: list[float] = [] | ||
| for left_min, right_min in zip(minima, minima[1:]): | ||
| if right_min <= left_min: | ||
| continue | ||
| peaks = [index for index in maxima if left_min < index < right_min] | ||
| if not peaks: | ||
| continue | ||
| peak = max(peaks, key=lambda index: values[index]) | ||
| period = float(times[right_min] - times[left_min]) | ||
| if period <= 0: | ||
| continue | ||
| fractions.append(float((times[peak] - times[left_min]) / period * 100.0)) | ||
| if not fractions: | ||
| return None | ||
| return float(np.median(np.asarray(fractions, dtype=np.float64))) | ||
|
|
||
|
|
||
| def _check_frequency( | ||
| summary: dict[str, Any], | ||
| expectation: dict[str, Any], | ||
| checks: list[dict[str, Any]], | ||
| ) -> None: | ||
| expected = _optional_positive_float(expectation, "frequency_hz") | ||
| if expected is None: | ||
| return | ||
| actual = summary.get("frequency_estimate_hz") | ||
| tolerance = float(expectation.get("frequency_tolerance_ratio", 0.05)) | ||
| low_confidence = any( | ||
| str(item).startswith("low_cycle_count") | ||
| for item in summary.get("quality_warnings", []) | ||
| ) | ||
| if not isinstance(actual, (int, float)) or actual <= 0: | ||
| checks.append(_check("frequency_hz", "warn", expected, actual, "frequency unavailable")) | ||
| return | ||
| error_ratio = abs(float(actual) - expected) / expected | ||
| if low_confidence: | ||
| checks.append( | ||
| _check( | ||
| "frequency_hz", | ||
| "warn", | ||
| expected, | ||
| float(actual), | ||
| "frequency low confidence because waveform contains too few cycles", | ||
| error_ratio=error_ratio, | ||
| tolerance_ratio=tolerance, | ||
| ) | ||
| ) | ||
| return | ||
| checks.append( | ||
| _check( | ||
| "frequency_hz", | ||
| "pass" if error_ratio <= tolerance else "fail", | ||
| expected, | ||
| float(actual), | ||
| "ok" if error_ratio <= tolerance else "frequency out of tolerance", | ||
| error_ratio=error_ratio, | ||
| tolerance_ratio=tolerance, | ||
| ) | ||
| ) | ||
|
|
||
|
|
||
| def _check_vpp( | ||
| summary: dict[str, Any], | ||
| expectation: dict[str, Any], | ||
| checks: list[dict[str, Any]], | ||
| ) -> None: | ||
| expected = _optional_positive_float(expectation, "vpp_v") | ||
| if expected is None: | ||
| return | ||
| actual = summary.get("voltage_vpp_v") | ||
| tolerance = float(expectation.get("vpp_tolerance_ratio", 0.10)) | ||
| if not isinstance(actual, (int, float)): | ||
| checks.append(_check("vpp_v", "warn", expected, actual, "Vpp unavailable")) | ||
| return | ||
| error_ratio = abs(float(actual) - expected) / expected | ||
| checks.append( | ||
| _check( | ||
| "vpp_v", | ||
| "pass" if error_ratio <= tolerance else "fail", | ||
| expected, | ||
| float(actual), | ||
| "ok" if error_ratio <= tolerance else "Vpp out of tolerance", | ||
| error_ratio=error_ratio, | ||
| tolerance_ratio=tolerance, | ||
| ) | ||
| ) | ||
|
|
||
|
|
||
| def _check_mean( | ||
| summary: dict[str, Any], | ||
| expectation: dict[str, Any], | ||
| checks: list[dict[str, Any]], | ||
| ) -> None: | ||
| expected = _optional_float(expectation, "mean_v") | ||
| if expected is None: | ||
| expected = _optional_float(expectation, "offset_v") | ||
| if expected is None: | ||
| return | ||
| actual = summary.get("voltage_mean_v") | ||
| tolerance = float(expectation.get("mean_tolerance_v", 0.05)) | ||
| if not isinstance(actual, (int, float)): | ||
| checks.append(_check("mean_v", "warn", expected, actual, "mean unavailable")) | ||
| return | ||
| error = abs(float(actual) - expected) | ||
| checks.append( | ||
| _check( | ||
| "mean_v", | ||
| "pass" if error <= tolerance else "fail", | ||
| expected, | ||
| float(actual), | ||
| "ok" if error <= tolerance else "mean out of tolerance", | ||
| error_abs=error, | ||
| tolerance_abs=tolerance, | ||
| ) | ||
| ) | ||
|
|
||
|
|
||
| def _check_duty( | ||
| summary: dict[str, Any], | ||
| expectation: dict[str, Any], | ||
| checks: list[dict[str, Any]], | ||
| ) -> None: | ||
| expected = _optional_float(expectation, "duty_cycle") | ||
| if expected is None and "duty_percent" in expectation: | ||
| expected = _optional_float(expectation, "duty_percent") | ||
| if expected is not None: | ||
| expected /= 100.0 | ||
| if expected is None: | ||
| return | ||
| actual = summary.get("duty_cycle") | ||
| tolerance = float(expectation.get("duty_tolerance", 0.05)) | ||
| if not isinstance(actual, (int, float)): | ||
| checks.append(_check("duty_cycle", "warn", expected, actual, "duty unavailable")) | ||
| return | ||
| error = abs(float(actual) - expected) | ||
| checks.append( | ||
| _check( | ||
| "duty_cycle", | ||
| "pass" if error <= tolerance else "fail", | ||
| expected, | ||
| float(actual), | ||
| "ok" if error <= tolerance else "duty out of tolerance", | ||
| error_abs=error, | ||
| tolerance_abs=tolerance, | ||
| ) | ||
| ) | ||
|
|
||
|
|
||
| def _check_symmetry( | ||
| waveform: WaveformData, | ||
| expectation: dict[str, Any], | ||
| checks: list[dict[str, Any]], | ||
| ) -> None: | ||
| expected = _optional_float(expectation, "symmetry_percent") | ||
| if expected is None: | ||
| return | ||
| actual = estimate_triangle_symmetry_percent(waveform) | ||
| tolerance = float(expectation.get("symmetry_tolerance_percent", 5.0)) | ||
| if actual is None: | ||
| checks.append(_check("symmetry_percent", "warn", expected, actual, "symmetry unavailable")) | ||
| return | ||
| error = abs(actual - expected) | ||
| checks.append( | ||
| _check( | ||
| "symmetry_percent", | ||
| "pass" if error <= tolerance else "fail", | ||
| expected, | ||
| actual, | ||
| "ok" if error <= tolerance else "symmetry out of tolerance", | ||
| error_abs=error, | ||
| tolerance_abs=tolerance, | ||
| ) | ||
| ) | ||
|
|
||
|
|
||
| def _check(name: str, status: str, expected: Any, actual: Any, message: str, **extra: Any) -> dict[str, Any]: | ||
| return { | ||
| "metric": name, | ||
| "status": status, | ||
| "expected": expected, | ||
| "actual": actual, | ||
| "message": message, | ||
| **extra, | ||
| } | ||
|
|
||
|
|
||
| def _optional_float(data: dict[str, Any], name: str) -> float | None: | ||
| if name not in data or data[name] is None: | ||
| return None | ||
| value = data[name] | ||
| if isinstance(value, bool): | ||
| return None | ||
| try: | ||
| return float(value) | ||
| except (TypeError, ValueError): | ||
| return None | ||
|
|
||
|
|
||
| def _optional_positive_float(data: dict[str, Any], name: str) -> float | None: | ||
| value = _optional_float(data, name) | ||
| if value is None or value <= 0: | ||
| return None | ||
| return value | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P1] 当 expectation 字段无效、拼写错误或越界时,当前解析逻辑会静默忽略该字段;如果最终没有生成任何 check,这里仍会返回
status="pass"。例如{"frequency_hz": "typo"}、{"vpp_v": -1}或拼错的{"frequncy_hz": 1000}都会得到pass, checks=[]。MCP 发布的 JSON Schema 并未在服务端真正执行,不能依赖客户端自动遵守。建议在运行时完整校验字段名、类型、有限性和取值范围,遇到无效输入直接返回
ConfigError;如果没有任何可执行检查,状态至少应为skipped或error,不能是pass。