Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
401e276
Unity MCP: stable framing handshake + non-blocking script writes; rem…
dsarno Aug 15, 2025
7eeac65
Bridge framing hardening: 64MiB cap, zero-length reject, timeout Read…
dsarno Aug 15, 2025
eafe309
ManageScript: improve method span parsing and validation behavior for…
dsarno Aug 15, 2025
73d212f
Unity MCP: prefer micro-edits & resources; add script_apply_edits pri…
dsarno Aug 16, 2025
a12dcab
test: add initial script and asset edit tests
dsarno Aug 16, 2025
92a6825
Merge pull request #3 from dsarno/codex/refactor-mcp-editing-tools-wi…
dsarno Aug 16, 2025
96c4b80
Merge branch 'CoplayDev:main' into protocol-framing
dsarno Aug 16, 2025
de4a6bc
Maintain manage_script compatibility and add safety checks
dsarno Aug 16, 2025
560213f
Merge pull request #6 from dsarno/codex/update-managescript-for-backw…
dsarno Aug 16, 2025
c13a2da
Support explicit validation levels
dsarno Aug 16, 2025
031f5d7
Merge pull request #7 from dsarno/codex/update-managescript-for-backw…
dsarno Aug 16, 2025
49a3355
Fix script tool returns and handshake edge cases
dsarno Aug 17, 2025
19b1b4f
Merge pull request #8 from dsarno/codex/streamline-protocol-framing-i…
dsarno Aug 17, 2025
c735117
Prevent overflow when counting edit bytes
dsarno Aug 17, 2025
fedf444
Merge pull request #10 from dsarno/codex/fix-potential-integer-overfl…
dsarno Aug 17, 2025
79ba5ec
Fix refresh debounce scheduling
dsarno Aug 17, 2025
7ee2191
Merge pull request #12 from dsarno/codex/fix-roslyn-error-handling-lo…
dsarno Aug 17, 2025
01fc4f1
Clean up temp backups and guard symlinked edits
dsarno Aug 17, 2025
a53204a
Merge pull request #13 from dsarno/codex/fix-roslyn-error-handling-lo…
dsarno Aug 17, 2025
fd79127
Fix debouncing race condition
dsarno Aug 17, 2025
fb0f464
Merge pull request #14 from dsarno/codex/fix-debouncing-logic-in-refr…
dsarno Aug 17, 2025
200483e
Add initial transport handshake tests with plan placeholders
dsarno Aug 17, 2025
a3c81d6
Fix dummy server startup and cleanup in transport tests
dsarno Aug 17, 2025
b01978c
test: enforce no prints and handshake preamble
dsarno Aug 17, 2025
555d965
feat: add defensive server path resolution in tests
dsarno Aug 17, 2025
e4544f6
Refine server source path lookup
dsarno Aug 17, 2025
9dbb4ff
Refine handshake tests and stdout hygiene
dsarno Aug 18, 2025
fde29d0
Convert skipped tests to xfail and improve framing robustness
dsarno Aug 18, 2025
5386b23
clarify stdout test failure messaging
dsarno Aug 18, 2025
63b070b
Add handshake fallback and logging checks
dsarno Aug 18, 2025
6cedb80
Claude Desktop: write BOM-free config to macOS path; dual-path fallba…
dsarno Aug 17, 2025
3bc9cd0
MCP: natural-language edit defaults; header guard + precondition for …
dsarno Aug 18, 2025
3cc1acd
MCP: add anchor_delete/anchor_replace structured ops; normalize NL/te…
dsarno Aug 18, 2025
22f55ce
Convert skipped tests to xfail and improve framing robustness
dsarno Aug 18, 2025
f3e94db
clarify stdout test failure messaging
dsarno Aug 18, 2025
d1362ac
Add handshake fallback and logging checks
dsarno Aug 18, 2025
e4e89e4
Claude Desktop: write BOM-free config to macOS path; dual-path fallba…
dsarno Aug 17, 2025
f1d773b
MCP: resolve merge conflicts; unify NL parsing and text-edit guards; …
dsarno Aug 18, 2025
c26ee13
MCP: add spec resource (script-edits); tighten script_apply_edits des…
dsarno Aug 18, 2025
3962cad
MCP: add script-edits spec resource; route all-structured edits via '…
dsarno Aug 18, 2025
c28bb38
CI: gate Unity compile steps behind secrets.UNITY_LICENSE to avoid 'C…
dsarno Aug 19, 2025
3943abd
CI: add Claude NL/T prompt at .claude/prompts/nl-unity-suite.md
dsarno Aug 19, 2025
77c841b
Tests: switch NL suite to ClaudeTests/longUnityScript-claudeTest.cs; …
dsarno Aug 19, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 103 additions & 0 deletions .claude/prompts/nl-unity-suite.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# CLAUDE TASK: Run NL/T editing tests for Unity MCP repo and emit JUnit

You are running in CI at the repository root. Use only the tools that are allowed by the workflow:
- View, GlobTool, GrepTool for reading.
- Bash for local shell (git is allowed).
- BatchTool for grouping.
- MCP tools from server "unity" (exposed as mcp__unity__*).

## Test target
- Primary file: `ClaudeTests/longUnityScript-claudeTest.cs`
- For each operation, prefer structured edit tools (`replace_method`, `insert_method`, `delete_method`, `anchor_insert`, `apply_text_edits`, `regex_replace`) via the MCP server.
- Include `precondition_sha256` for any text path write.

## Output requirements
- Create a JUnit XML at `reports/claude-nl-tests.xml`.
- Each test = one `<testcase>` with `classname="UnityMCP.NL"` or `UnityMCP.T`.
- On failure, include a `<failure>` node with a concise message and the last evidence snippet (10–20 lines).
- Also write a human summary at `reports/claude-nl-tests.md` with checkboxes and the windowed reads.

## Safety & hygiene
- Make edits in-place, then revert them at the end (`git stash -u`/`git reset --hard` or balanced counter-edits) so the workspace is clean for subsequent steps.
- Never push commits from CI.
- If a write fails midway, ensure the file is restored before proceeding.

## NL-0. Sanity Reads (windowed)
- Tail 120 lines of `ClaudeTests/longUnityScript-claudeTest.cs`.
- Show 40 lines around method `Update`.
- **Pass** if both windows render with expected anchors present.

## NL-1. Method replace/insert/delete (natural-language)
- Replace `HasTarget` with block-bodied version returning `currentTarget != null`.
- Insert `PrintSeries()` after `GetCurrentTarget` logging `1,2,3`.
- Verify by reading 20 lines around the anchor.
- Delete `PrintSeries()` and verify removal.
- **Pass** if diffs match and verification windows show expected content.

## NL-2. Anchor comment insertion
- Add a comment `Build marker OK` immediately above the `Update` method.
- **Pass** if the comment appears directly above the `public void Update()` line.

## NL-3. End-of-class insertion
- Insert a 3-line comment `Tail test A/B/C` before the last method or immediately before the final class brace (preview, then apply).
- **Pass** if windowed read shows the three lines at the intended location.

## NL-4. Compile trigger
- After any NL edit, ensure no stale compiler errors:
- Write a short marker edit, then **revert** after validating.
- The CI job will run Unity compile separately; record your local check (e.g., file parity and syntax sanity) as INFO, but do not attempt to invoke Unity here.

## T-A. Anchor insert (text path)
- Insert after `GetCurrentTarget`: `private int __TempHelper(int a, int b) => a + b;`
- Verify via read; then delete with a `regex_replace` targeting only that helper block.
- **Pass** if round-trip leaves the file exactly as before.

## T-B. Replace method body with minimal range
- Identify `HasTarget` body lines; single `replace_range` to change only inside braces; then revert.
- **Pass** on exact-range change + revert.

## T-C. Header/region preservation
- For `ApplyBlend`, change only interior lines via `replace_range`; the method signature and surrounding `#region`/`#endregion` markers must remain untouched.
- **Pass** if signature and region markers unchanged.

## T-D. End-of-class insertion (anchor)
- Find final class brace; `position: before` to append a temporary helper; then remove.
- **Pass** if insert/remove verified.

## T-E. Temporary method lifecycle
- Insert helper (T-A), update helper implementation via `apply_text_edits`, then delete with `regex_replace`.
- **Pass** if lifecycle completes and file returns to original checksum.

## T-F. Multi-edit atomic batch
- In one call, perform two `replace_range` tweaks and one comment insert at the class end; verify all-or-nothing behavior.
- **Pass** if either all 3 apply or none.

## T-G. Path normalization
- Run the same edit once with `unity://path/ClaudeTests/longUnityScript-claudeTest.cs` and once with `ClaudeTests/longUnityScript-claudeTest.cs` (if supported).
- **Pass** if both target the same file and no path duplication.

## T-H. Validation levels
- After edits, run `validate` with `level: "standard"`, then `"basic"` for temporarily unbalanced text ops; final state must be valid.
- **Pass** if validation OK and final file compiles in CI step.

## T-I. Failure surfaces (expected)
- Too large payload: `apply_text_edits` with >15 KB aggregate → expect `{status:"too_large"}`.
- Stale file: change externally, then resend with old `precondition_sha256` → expect `{status:"stale_file"}` with hashes.
- Overlap: two overlapping ranges → expect rejection.
- Unbalanced braces: remove a closing `}` → expect validation failure and **no write**.
- Header guard: attempt insert before the first `using` → expect `{status:"header_guard"}`.
- Anchor aliasing: `insert`/`content` alias → expect success (aliased to `text`).
- Auto-upgrade: try a text edit overwriting a method header → prefer structured `replace_method` or return a clear error.
- **Pass** when each negative case returns the expected failure without persisting changes.

## T-J. Idempotency & no-op
- Re-run the same `replace_range` with identical content → expect success with no change.
- Re-run a delete of an already-removed helper via `regex_replace` → clean no-op.
- **Pass** if both behave idempotently.

### Implementation notes
- Always capture pre- and post‑windows (±20–40 lines) as evidence in the JUnit `<failure>` or as `<system-out>`.
- For any file write, include `precondition_sha256` and verify the post‑hash in your log.
- At the end, restore the repository to its original state (`git status` must be clean).

# Emit the JUnit file to reports/claude-nl-tests.xml and a summary markdown to reports/claude-nl-tests.md.
110 changes: 110 additions & 0 deletions .github/workflows/claude-nl-suite.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
name: Claude NL suite + (optional) Unity compile

on: { workflow_dispatch: {} }



permissions:
contents: write # allow Claude to write test artifacts
pull-requests: write # allow annotations / comments
issues: write

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
nl-suite:
if: github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4
with: { fetch-depth: 0 }

# If your MCP server needs Python deps (adjust to your repo layout)
- name: Install Python + uv
uses: astral-sh/setup-uv@v4
with:
python-version: '3.11'

- name: Prepare Unity MCP server deps (adjust path or remove if N/A)
run: |
if [ -f UnityMcpServer/requirements.txt ]; then
uv pip install -r UnityMcpServer/requirements.txt
fi

- name: Run Claude NL/T test suite
id: claude
uses: anthropics/claude-code-base-action@beta
with:
# All the test instructions live here (see next file)
prompt_file: .claude/prompts/nl-unity-suite.md

# Keep tools tight: read, grep, glob, run shell, orchestrate batches,
# and call your MCP server tools. (Adjust the mcp__ prefix to match.)
allowed_tools: "Bash(git:*),View,GlobTool,GrepTool,BatchTool,mcp__unity__*"

# Inline MCP config (or put this JSON in .claude/mcp.json)
mcp_config: |
{
"mcpServers": {
"unity": {
"command": "python",
"args": ["UnityMcpServer/src/server.py"]
}
}
}
Comment on lines +49 to +57
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Verify MCP server path; consider resolving dynamically if location differs.

The configured path is UnityMcpServer/src/server.py, while other parts of the repo may use UnityMcpBridge/UnityMcpServer~/src/server.py. Confirm the correct entrypoint.

Run:

If needed, update args accordingly. For maximal robustness, you can wrap the MCP config in a tiny launcher script that resolves the path at runtime.


🏁 Script executed:

#!/bin/bash
set -euo pipefail
echo "Searching for 'server.py' candidates used by MCP:"
fd -a --glob '*Unity*Mcp*/*/server.py' || true
sed -n '1,120p' .claude/prompts/nl-unity-suite.md | rg -n 'UnityMcpServer|UnityMcpBridge' -n -C2 || true

Length of output: 279


🏁 Script executed:

#!/bin/bash
set -euo pipefail
echo "Searching for all server.py files in the repo:"
fd --glob 'server.py' -t f || true

echo ""
echo "Listing directories matching UnityMcpServer*:"
fd --glob 'UnityMcpServer*' -t d || true

echo ""
echo "Listing directories matching UnityMcpBridge*:"
fd --glob 'UnityMcpBridge*' -t d || true

echo ""
echo "Searching for any references to 'server.py' in workflow files:"
rg -n --glob '*.yml' 'server.py' -C2 || true

Length of output: 881


Fix MCP server entrypoint path in workflow

The workflow is pointing to a non-existent path. The only server.py in the repo lives under UnityMcpBridge/UnityMcpServer~/src/server.py, not UnityMcpServer/src/server.py. Update the args in .github/workflows/claude-nl-suite.yml accordingly.

Locations to update:

  • .github/workflows/claude-nl-suite.yml (around lines 49–57)

Suggested diff:

           mcp_config: |
             {
               "mcpServers": {
                 "unity": {
                   "command": "python",
-                  "args": ["UnityMcpServer/src/server.py"]
+                  "args": ["UnityMcpBridge/UnityMcpServer~/src/server.py"]
                 }
               }
             }

Committable suggestion skipped: line range outside the PR's diff.

🤖 Prompt for AI Agents
In .github/workflows/claude-nl-suite.yml around lines 49 to 57, the MCP server
entrypoint path in the mcp_config args points to a non-existent
UnityMcpServer/src/server.py; update the args value to the correct repository
location UnityMcpBridge/UnityMcpServer~/src/server.py so the workflow runs the
actual server.py file (replace the existing args entry with the corrected path).


# Model + guardrails
model: "claude-3-7-sonnet-20250219"
max_turns: "10"
timeout_minutes: "20"
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}

- name: Upload JUnit (Claude NL/T)
if: always()
uses: actions/upload-artifact@v4
with:
name: claude-nl-tests
path: reports/claude-nl-tests.xml

- name: Annotate PR with test results (Claude NL/T)
if: always()
uses: dorny/test-reporter@v1
with:
name: Claude NL/T
path: reports/claude-nl-tests.xml
reporter: java-junit

# --- Optional: Unity compile after Claude’s edits (satisfies NL-4) ---
# If your repo is a *Unity project*:
- name: Unity compile (Project)
if: ${{ always() && hashFiles('ProjectSettings/ProjectVersion.txt') != '' && secrets.UNITY_LICENSE != '' }}
uses: game-ci/unity-test-runner@v4
env:
UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }} # OR UNITY_* for Pro
with:
projectPath: .
githubToken: ${{ secrets.GITHUB_TOKEN }}
# Even with no tests, this compiles; add EditMode/PlayMode tests later.
testMode: EditMode
Comment on lines +83 to +91
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix “secrets” usage in if: expressions (actionlint error; GitHub disallows secrets. in conditions).*

Current “if” references secrets.UNITY_LICENSE, which is not available in expressions and will cause validation/runtime errors.

Apply this diff to route the secret through env and check env.UNITY_LICENSE instead:

 jobs:
   nl-suite:
     if: github.event_name == 'workflow_dispatch'
     runs-on: ubuntu-latest
+    env:
+      UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
@@
-      - name: Unity compile (Project)
-        if: ${{ always() && hashFiles('ProjectSettings/ProjectVersion.txt') != '' && secrets.UNITY_LICENSE != '' }}
+      - name: Unity compile (Project)
+        if: ${{ always() && hashFiles('ProjectSettings/ProjectVersion.txt') != '' && env.UNITY_LICENSE != '' }}
         uses: game-ci/unity-test-runner@v4
-        env:
-          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}  # OR UNITY_* for Pro
+        env:
+          UNITY_LICENSE: ${{ env.UNITY_LICENSE }}  # OR UNITY_* for Pro
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if: ${{ always() && hashFiles('ProjectSettings/ProjectVersion.txt') != '' && secrets.UNITY_LICENSE != '' }}
uses: game-ci/unity-test-runner@v4
env:
UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }} # OR UNITY_* for Pro
with:
projectPath: .
githubToken: ${{ secrets.GITHUB_TOKEN }}
# Even with no tests, this compiles; add EditMode/PlayMode tests later.
testMode: EditMode
jobs:
nl-suite:
if: github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
env:
UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
steps:
- name: Unity compile (Project)
if: ${{ always() && hashFiles('ProjectSettings/ProjectVersion.txt') != '' && env.UNITY_LICENSE != '' }}
uses: game-ci/unity-test-runner@v4
env:
UNITY_LICENSE: ${{ env.UNITY_LICENSE }} # OR UNITY_* for Pro
with:
projectPath: .
githubToken: ${{ secrets.GITHUB_TOKEN }}
# Even with no tests, this compiles; add EditMode/PlayMode tests later.
testMode: EditMode
🧰 Tools
🪛 actionlint (1.7.7)

83-83: context "secrets" is not allowed here. available contexts are "env", "github", "inputs", "job", "matrix", "needs", "runner", "steps", "strategy", "vars". see https://docs.github.com/en/actions/learn-github-actions/contexts#context-availability for more details

(expression)

🤖 Prompt for AI Agents
.github/workflows/claude-nl-suite.yml around lines 83-91: the step if-condition
currently uses secrets.UNITY_LICENSE which is disallowed in expressions; replace
that reference with env.UNITY_LICENSE and ensure the secret is routed into an
environment variable that's visible to expressions (move UNITY_LICENSE: ${{
secrets.UNITY_LICENSE }} into the job- or workflow-level env or otherwise expose
it before the if runs) so the if uses env.UNITY_LICENSE instead of
secrets.UNITY_LICENSE.


# If your repo is primarily a *Unity package*, prefer packageMode:
- name: Unity compile (Package)
if: ${{ always() && hashFiles('Packages/manifest.json') != '' && hashFiles('ProjectSettings/ProjectVersion.txt') == '' && secrets.UNITY_LICENSE != '' }}
uses: game-ci/unity-test-runner@v4
env:
UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
with:
packageMode: true
unityVersion: 2022.3.45f1 # <-- set explicitly for packages
projectPath: . # or a small sample project path
githubToken: ${{ secrets.GITHUB_TOKEN }}
Comment on lines +95 to +103
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix “secrets” usage in if: expressions for Package mode as well.

Mirror the env-based gating for the package compilation block.

-      - name: Unity compile (Package)
-        if: ${{ always() && hashFiles('Packages/manifest.json') != '' && hashFiles('ProjectSettings/ProjectVersion.txt') == '' && secrets.UNITY_LICENSE != '' }}
+      - name: Unity compile (Package)
+        if: ${{ always() && hashFiles('Packages/manifest.json') != '' && hashFiles('ProjectSettings/ProjectVersion.txt') == '' && env.UNITY_LICENSE != '' }}
         uses: game-ci/unity-test-runner@v4
-        env:
-          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
+        env:
+          UNITY_LICENSE: ${{ env.UNITY_LICENSE }}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if: ${{ always() && hashFiles('Packages/manifest.json') != '' && hashFiles('ProjectSettings/ProjectVersion.txt') == '' && secrets.UNITY_LICENSE != '' }}
uses: game-ci/unity-test-runner@v4
env:
UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
with:
packageMode: true
unityVersion: 2022.3.45f1 # <-- set explicitly for packages
projectPath: . # or a small sample project path
githubToken: ${{ secrets.GITHUB_TOKEN }}
- name: Unity compile (Package)
if: ${{ always() && hashFiles('Packages/manifest.json') != '' && hashFiles('ProjectSettings/ProjectVersion.txt') == '' && env.UNITY_LICENSE != '' }}
uses: game-ci/unity-test-runner@v4
env:
UNITY_LICENSE: ${{ env.UNITY_LICENSE }}
with:
packageMode: true
unityVersion: 2022.3.45f1 # <-- set explicitly for packages
projectPath: . # or a small sample project path
githubToken: ${{ secrets.GITHUB_TOKEN }}
🧰 Tools
🪛 actionlint (1.7.7)

95-95: context "secrets" is not allowed here. available contexts are "env", "github", "inputs", "job", "matrix", "needs", "runner", "steps", "strategy", "vars". see https://docs.github.com/en/actions/learn-github-actions/contexts#context-availability for more details

(expression)

🤖 Prompt for AI Agents
In .github/workflows/claude-nl-suite.yml around lines 95-103, the package-mode
step's if expression uses secrets.UNITY_LICENSE directly while the other package
compilation block gates on the environment variable; change the condition to
check env.UNITY_LICENSE instead (e.g. env.UNITY_LICENSE != '') and ensure
UNITY_LICENSE is exported into the step/job env from secrets (UNITY_LICENSE: ${{
secrets.UNITY_LICENSE }}) before this if is evaluated so the gating mirrors the
env-based check used elsewhere; keep the existing hashFiles checks and other
conditions unchanged.


- name: Clean working tree (discard temp edits)
if: always()
run: |
git restore -SW :/
git clean -fd

Loading