Replies: 1 comment
|
Thorough as always — and the most useful thing I can tell you is that the tool-layer failures this audit is built around were fixed within days of your benchmark, and shipped today in v0.7.0:
So the audit's core thesis — "the tool layer can silently approve broken boards even when the guidance is followed" — was true of v0.6.1 and is the specific thing v0.7.0 exists to end. If you re-run the benchmark on v0.7.0 I'd expect the TRANSFER and REVIEW gates to behave very differently, and I'd genuinely value that re-run as confirmation. On the guidance-layer P0s that remain open:
The skills themselves also now carry a guard that validates their tool examples against the live schemas (#183/#217 work), so guidance drift of the "this example can never succeed" kind is caught at CI rather than by your benchmarks. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Claude workflow guidance (SKills) gaps in Konnect v0.6.1 — proposed roadmap
Important
This is a Codex-assisted evaluation of the Claude-facing guidance published with
Konnect v0.6.1. It is based on the exact tagged source, upstream issues and
discussions, and an actual end-to-end Codex Konnect/KiCad benchmark. It is
not a direct Claude benchmark and not a maintainer-authored compatibility claim.
This report audits instructions, orchestration, and safety gates rather than
proposing a second implementation stack. Every recommendation keeps Konnect's
Rust-only direction, uses Konnect MCP tools and supported KiCad IPC for design
mutation, and avoids Python and direct editing of KiCad source files.
Freerouting is treated as Konnect's official whole-board routing direction,
not as an optional competing workflow.
Pinned scope
506abe094204c6d4acd77415892e9e0e8fdb35fb.PreToolUsehook embedded in the v0.6.1 manifest.
and the maintainer's source-level confirmation.
#240,
#249,
#250, and
#253.
The audit is intentionally limited to assets shipped in v0.6.1. At the time of
review, upstream
mainwas 21 commits ahead of the tag; those post-releasefixes and PRs, the standalone
konnect-codexcompanion, and the retiredKiCAD-MCP-Server are not treated as released Konnect capabilities.
Executive assessment
The published Claude material is useful and materially better than raw MCP tool
descriptions. It teaches protected-file discipline, on-demand toolset loading,
revision-checked schematic-to-PCB transfer, basic layout order, ERC/DRC, and a
manufacturing checklist.
The remaining gap is not primarily more electrical-design prose. Claude needs a
deterministic operating procedure with observable gates. Today it can select a
valid tool but still choose the wrong sequence, trust contradictory evidence,
route a whole board with obstacle-blind L-bends, continue after losing the live
board, or approve a result whose artifacts disagree with the tool response.
The recommended target is five cooperating roles backed by shared skills:
Run these roles sequentially for a single project. In particular, do not split
Freerouting into a concurrent routing agent: one PCB builder should own the live
KiCad document and its IPC session from placement checkpoint through route
acceptance.
What v0.6.1 already does well
modifications and refuse ad hoc KiCad source edits. It also teaches dynamic
tool discovery rather than hard-coding a tool count. See the
konnectskill.returned plan revision, and one transactional IPC apply for
update_pcb_from_schematic. See thelayout sequence.
and the manufacturing skill requires DRC before export.
responsibilities. Those are good foundations for deterministic delegation.
every representable KiCad 10 footprint layer and refuses an unrepresentable
one before IPC submission rather than hoping KiCad accepts it. The
release notes
document the reproduced crash and the fail-closed repair.
Prioritized workflow gaps
route_pad_to_padthe primary router even though it is an obstacle-blind L-bend and Freerouting is the advertised direction.LOOKS GOOD/READYwhile direct DRC still had 25 errors and one unrouted item. Contradictory results need a defined authority order.P0-1: make Freerouting the explicit whole-board route path
Evidence
Konnect advertises Freerouting in the
README
and registers
autorouteas DSN export, Freerouting, and SES import in theintegrationtoolset.However, the v0.6.1 handler
always returns unavailable.
Meanwhile, the PCB skill describes
route_pad_to_padas the primary routing tool.The benchmark proved that the installed KiCad Freerouting ActionPlugin could
complete an autonomous route, while also proving that imported widths and
clearances still needed explicit acceptance checks. Issue
#253 records the capability
and discovery mismatch.
Guidance change
Add a
kicad-pcb-builderagent and a progressively disclosed Freeroutingreference under the PCB skill. The default whole-board sequence should be:
unfilled/intentional zone state;
separate capabilities;
accepting the route or filling zones.
Keep
route_trace,route_pad_to_pad, andadd_viafor deliberate short linksand understood local repairs. Do not present repeated L-bends as a substitute
for a whole-board router with obstacle avoidance and rip-up/retry.
Rust/IPC implementation boundary
Konnect's Rust code may discover a standalone JAR or the engine bundled in the
KiCad plugin and may launch the Java process. Board identity, DSN export, SES
import, refresh, and result verification must use a supported KiCad IPC or
in-editor contract. If the installed KiCad IPC version does not expose that
exchange, return a structured
bridge_unavailablecapability and hand off thesmallest explicit step to the official KiCad ActionPlugin. Do not add a Python
bridge and do not parse or rewrite
.kicad_pcbas a routing workaround.Acceptance criteria
KiCad bridge ready.”
are unchanged, required connections are routed, widths/clearances satisfy the
loaded rules, and direct DRC has no unwaived errors.
and gives the ActionPlugin handoff instead of improvising another writer.
P0-2: enforce one live board and fail closed after IPC loss
Evidence
The released hook only prints an instruction before a subset of PCB tools; its
matcher
does not cover the complete mutation surface, including schematic-to-PCB update
and several component/zone operations. It also does not make an IPC or board-path
decision. Issue #240 documents
the released fallback editing a stale saved board immediately after KiCad had
held and then lost that board.
Guidance and Rust change
that its canonical path equals the requested board.
PreToolUsepreflight that consumesthe hook input and returns a structured deny when identity or liveness cannot
be proven. Claude Code officially supports a
PreToolUsedeny decision.experience but must not be the only protection against stale IPC state.
mutation phase. Reopen, re-identify, re-inventory, and resume from the last
saved gate.
Acceptance criteria
from the tool input.
file fallback.
in its denial.
saved and re-inventoried.
P0-3: define evidence authority and make incomplete coverage visible
Evidence
The benchmark's direct KiCad DRC reported 25 errors and one unrouted connection,
while
run_design_reviewreturnedLOOKS GOOD,validate_for_manufacturingreturned
READY, and review coverage reported zero pads on a populated board.The maintainer
confirmed the fail-open paths and their causes.
The same run found
find_orphan_itemsdeclaring every valid pin-mounted labelfloating; an attempted “repair” created a real short. Issue
#249 documents the divergent
connectivity model.
The packaged prose compounds that uncertainty. The review skill describes its
shortcut as a comprehensive substitute, although the tagged shortcut did not
consume direct ERC/DRC evidence. The schematic builder's result template reports
an ERC outcome even though its workflow neither loads the export toolset that
owns ERC nor runs ERC. An agent must never be prompted to fill a success field
for evidence its prescribed sequence did not collect.
Guidance change
Teach an explicit authority order:
A weaker result may add a question; it must not override stronger contradictory
evidence or trigger automatic geometry changes. Any required check that fails to
run, returns structurally impossible coverage, or disagrees with direct evidence
makes the verdict INCOMPLETE, not clean.
Also make the review agent's universal “quality bars” conditional engineering
defaults. The current agent makes ESD on every external interface and bulk
capacitance on every rail unconditionally critical. Requirements and datasheets
should decide whether a default applies; the reviewer should record the evidence
rather than turn a useful heuristic into a universal rule.
Acceptance criteria
ready verdict.
produces
INCOMPLETEwith the missing coverage named.detection, and net connectivity disagree with it.
P0-4: require a physical custom-part pin map
Evidence
The library skill includes useful mechanics, but its generic
pin-numbering table
states fixed BJT and MOSFET number orders that are not universal manufacturer
pinouts. Its custom IC workflow says to create from the datasheet but does not
require proof that each physical lead, symbol pin, and footprint pad is the same
electrical node. This is especially risky for bottom views, circular packages,
tubes/displays, connectors, mirrored parts, and replaceable sockets.
Guidance change
Add a
kicad-library-builderagent and a custom-part acceptance reference. Itmust record one row per physical lead:
Require an explicit top/bottom/component/pin-side declaration, a walk from the
manufacturer's key in the documented direction, exact part/package suffix, and
a query-back plus visible disposable placement before the part enters a real
schematic.
Acceptance criteria
documented.
physical viewing direction are verified.
blocks use and receives an independent review.
P1 workflow branches
Visual placement acceptance
The released skill says to check courtyard overlaps and verify moves, but it has
no artifact or checkpoint. Before routing, require a saved board inventory, a 2-D
render showing outline, pads, holes, courtyards and references, direct overlap and
edge checks, and an exception list. A 3-D render is additional evidence, not a
substitute for copper/courtyard inspection. Any placement change or IPC ownership
change invalidates the checkpoint.
Accept when: no blocking pad/hole/courtyard/edge/access conflict remains and
the rendered placement is explicitly approved for routing.
Incremental/ECO preservation
Extend the strong revision-checked sync procedure with a baseline of references,
positions, pads, traces by net/layer, vias, zones, and DRC. Review the dry-run
delta; apply only the exact plan revision; prove unaffected placement and routing
did not change; identify affected nets; and reroute only those nets when possible.
Use whole-board Freerouting only when the change invalidates the global route.
Accept when: the before/after report accounts for every change and preserves
all unaffected approved board state.
Power, thermal, and noisy-load layout
Trigger a separate reference for batteries, motors, heaters, solenoids,
converters, and other high-current or high-
di/dtloads. Establish min/nominal/transient/reverse voltage, continuous/peak current, copper weight, temperature
rise, voltage-drop allowance, fault energy, connector/fuse ratings, and return
paths. Size copper from those inputs and separate noisy returns from MCU, IMU,
analog, and oscillator references.
Accept when: the report states assumptions and calculated drop/loss at
continuous and peak current, verifies protection placement and component ratings,
and leaves no unbounded current or shared sensitive/noisy return path.
Repeatable review evidence
Create a timestamped evidence directory outside protected KiCad source files.
Include raw ERC/DRC, connectivity/short/orphan output, component/pad/trace/via/net
inventories, placement render, route provenance and DSN/SES identity, manufacturing
artifact list with sizes/hashes, custom-part maps, waivers, and the final verdict.
Record failed or unavailable checks instead of omitting them.
Accept when: another reviewer can reproduce the verdict from the saved
evidence without trusting a prose summary or an in-memory tool response.
P2 handoff branches
Firmware and bring-up planner
Add a read-only
kicad-bringup-planneragent after independent review. It shouldproduce a GPIO table, reset/boot/programming behavior, safe startup states, test
point expectations, current-limited first-power sequence, rail checks,
proof-of-life behavior, stop conditions, and interface/load enable order. It must
not modify the board or energize hardware.
Legacy and manual assembly
Add a conditional manufacturing reference for through-hole, socketed, surplus,
or hand-assembled parts: exact suffix, lifecycle/source/date, alternates,
lead/drill/pitch/height/tolerance, polarity/key, socket dimensions, rework access,
stock attrition, and counterfeit/oxidation risk. A distributor listing alone is
not proof of active production.
Make the agents actually share and use the skills
The six installed skills do not name either installed agent, so delegation is
left to the model. The package also installs nine reference files, but none of
the six parent
SKILL.mdfiles tells Claude when to read any of them. The agentfiles duplicate condensed workflow rules rather than preloading the corresponding
skill. Claude's own documentation says a
subagent starts with an isolated context and supports a
skillsfield to preload skill content.Recommended orchestration:
kicad-library-builderkonnect,kicad-librarykonnect,kicad-schematickicad-pcb-builderkonnect,kicad-pcbkonnect,kicad-review,kicad-manufacturekicad-bringup-plannerAdd this routing table to the top-level
konnectskill using clear triggerconditions. Keep detailed procedure in the branch skills/references so there is
one source of truth rather than duplicated agent prose. Claude skills are loaded
on demand, so branch-specific material can remain behind precise descriptions;
see Anthropic's skills documentation.
Proposed implementation order
authority, add custom-part and placement gates, bind existing agents to their
skills, and make agent delegation deterministic.
bring-up planner. Keep one live PCB owner and one independent reviewer.
structured denial on stale or lost IPC state.
KiCad bridge readiness; use supported IPC when available and fail closed to
the ActionPlugin checkpoint otherwise.
ECO update, a power/noisy-load board, Freerouting import, independent review,
manufacturing export, and bring-up handoff. Neuter each gate once to prove the
test catches its absence.
Boundaries
.kicad_pro,.kicad_sch,.kicad_pcb,.kicad_sym,.kicad_mod, or KiCad library tables outside Konnect's supportedRust/MCP operations.
Freerouting ActionPlugin step is an explicit handoff when IPC lacks the needed
exchange, not a hidden fallback.
aggregate summary. The saved result and direct KiCad evidence are authoritative.
session.
Bottom line
Konnect already exposes an unusually broad Rust/KiCad tool surface. That does not
remove the need for workflow guidance; it makes the guidance more important. An
AI needs to know which evidence is authoritative, when a phase is complete, when
to stop, and which specialist owns the next phase.
For v0.6.1, the highest-value change is not another generic design checklist. It
is a deterministic Claude workflow: validate custom parts, build and verify the
schematic, approve visible placement, route the complete board through
Freerouting, independently prove the result, package the evidence, and only then
prepare manufacturing and bring-up. That keeps Konnect's Rust/IPC architecture
clean while giving Claude enough direction to use it safely.
All reactions