Add real-platform personas and a personas × models field-test matrix#60
Merged
Conversation
The field-test harness gains three personas built from the actual production system prompts of the platforms we onboard on — bolt.diy (from its repo), Lovable and Replit Assistant (from the public system-prompt collection) — with policy/environment sections kept verbatim and response-format machinery replaced by a harness bridge. Provenance, adaptation rule, and caveats are documented in field-test/README.md. matrix.mjs runs the personas × agent-CLI cross-product (claude, codex, gemini, plus a scripted stub for free plumbing self-tests), delegating each cell to run.mjs and aggregating scorecards into one table under field-test/results/matrix-<timestamp>/. Validated: stub self-test green across all three personas; live runs claude × bolt-diy and codex × replit both 8/8. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Adds a cohesive personas×models matrix runner and matching documentation. 🎯 Quality: 91% Elite · 📦 Size: Large — consider splitting if possible 📈 This month: Your 221st PR — above team average · Averaging Excellent |
Contributor
Author
|
/review |
patchstackdave
approved these changes
Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
bolt-diy(fromstackblitz-labs/bolt.diy's repo),lovableandreplit(from the publicx1xhlol/system-prompts-and-models-of-ai-toolscollection). Policy, behavior, and environment sections are kept verbatim; response-format machinery (<boltArtifact>,lov-*,<proposed_*>) is replaced with a harness bridge so the agent acts through its own CLI tools. Provenance (source commits, retrieval date), the adaptation rule, and caveats are documented infield-test/README.md.field-test/matrix.mjs: runs the personas × agent-CLIs cross-product (named agents:claude,codex,gemini, and a scriptedstubfor free plumbing self-tests), delegating each cell torun.mjsand aggregating scorecards intofield-test/results/matrix-<timestamp>/matrix.md+matrix.json. Refusal behavior is a function of (system prompt × model); this measures both axes.run.mjsusage comment now reflects arbitrary persona names.Why
Real refusals differ per platform and per model. Hand-written personas guessed at platform policy; these carry the policy the real products run, and the matrix covers the model-choice variable that Bolt.diy exposes — without ToS-breaking browser automation of hosted platforms.
Validation
claude × bolt-diy: 8/8 checks passed.codex × replit: 8/8 checks passed — first cross-model-family run of the harness.🤖 Generated with Claude Code