Skip to content

test: add deterministic Moonshine integration and enforce session resume identity - #3

Open
Charlie-Wang-03 wants to merge 2 commits into
DeepMathLLM:mainfrom
Charlie-Wang-03:test/runner-moonshine-integration-session-identity
Open

test: add deterministic Moonshine integration and enforce session resume identity#3
Charlie-Wang-03 wants to merge 2 commits into
DeepMathLLM:mainfrom
Charlie-Wang-03:test/runner-moonshine-integration-session-identity

Conversation

@Charlie-Wang-03

Copy link
Copy Markdown
Contributor

Problem

The archive runner had deterministic offline coverage, but it still lacked a regression layer against the real Moonshine runtime.

In particular, resume state only passed session_id back into Moonshine and then checked project_slug afterward. A same-project session created with the wrong Moonshine mode or agent could therefore be accepted by the archive runner.

This matters because the persisted session_id is part of the runner's recovery contract: resuming the wrong runtime identity can silently continue work under incompatible agent semantics.

Contract

A resumable archive session must be bound to the complete runtime identity expected by the workflow:

  • mode="chat"
  • the target object's project_slug
  • agent_slug="moonshine-core"

The runner must fail closed before continuing when a persisted session violates any of those constraints.

The integration harness should exercise that contract through the real Moonshine runtime while keeping model behavior deterministic and offline.

Implementation

  • Add a dedicated Moonshine integration workflow on Python 3.11.
  • Pin Moonshine to commit 72a64f8984e0e7e1b0a50f6e53e4d00e4fffbb7d.
  • Keep Moonshine integration coverage outside the offline test_*.py discovery layer so the existing Python 3.9 / 3.11 suite remains dependency-free.
  • Exercise real MoonshineApp, session persistence, skill/tool infrastructure, material staging, verification-event storage, recovery, and archive publication.
  • Fake only main-provider and verification-provider responses.
  • Resume sessions by passing the complete expected identity into MoonshineApp.start_shell_state(...).
  • Translate Moonshine identity mismatches into runner-level RunnerError.

Deterministic integration coverage

The integration suite covers:

  1. A verified run through the real Moonshine runtime path.
  2. Recovery from persisted verification on a fresh MoonshineApp without another model turn.
  3. Wrong-project session rejection.
  4. Same-project wrong-mode session rejection.
  5. Same-project wrong-agent session rejection.

Regression evidence

Candidate A was originally developed TDD-style: workflow run 33901898148 reproduced the wrong-mode / wrong-agent acceptance bug. After the fix, run 33902182551 passed all five integration tests.

After upstream PRs #1 and #2 merged, the work was rebuilt directly on current main (5f7fbb72) and revalidated:

  • Moonshine integration run 33951560203: 5/5 passed on Python 3.11.
  • Offline regression run 33951560213: 17/17 passed on both Python 3.9 and Python 3.11.

No real LLM/API credentials or Tavily/MCP network calls are required.

Scope / limitations

This PR verifies the Creative-Intelligence ↔ Moonshine runtime boundary. It does not claim end-to-end model quality, live-provider compatibility, or web/MCP behavior.

The Moonshine commit is intentionally pinned so future runtime changes cannot silently alter this regression baseline.

Replays the completed deterministic Moonshine runtime integration harness and the session resume identity fix onto the latest upstream-synced main.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant