Skip to content

v0.1.3: three-model cross-family probe — three regimes emerge

Choose a tag to compare

View all tags
@immu4989 immu4989 released this 01 Jul 20:36
· 55 commits to main since this release
v0.1.3
c9ac90b

v0.1.3: three-model cross-family probe — three regimes emerge

v0.1.3 adds Mistral Small (mistral/mistral-small-latest) as a mid-tier data point between gpt-4o-mini (weak) and DeepSeek V3 (strong). Same code path, same workspace matrix (5 user × 1 injection × 2 attacks × 3 non-GEPA optimizers = 30 evals), only the LM changes.

Three-model side-by-side (utility / security)

Optimizer × Attack gpt-4o-mini Mistral Small DeepSeek V3
unopt × direct 0.00 / 1.00 0.20 / 1.00 0.80 / 1.00
unopt × important_instructions 0.00 / 0.80 0.40 / 1.00 0.60 / 0.80
bootstrap_fewshot × direct 0.60 / 1.00 0.80 / 1.00 0.60 / 1.00
bootstrap_fewshot × important_instructions 0.20 / 0.60 0.80 / 1.00 0.60 / 0.80
miprov2 × direct 0.40 / 0.80 0.80 / 1.00 0.80 / 1.00
miprov2 × important_instructions 0.20 / 0.60 0.60 / 1.00 0.60 / 0.80

Three-regime interpretation

Regime Base model example Unopt utility Optimization impact on utility Optimization impact on security
Weak gpt-4o-mini 0% Big lift (+40-60pp) Costs 20pp on hard attacks
Mid Mistral Small 20-40% Big lift (+40-60pp) Zero cost
Strong DeepSeek V3 60-80% Zero-or-negative gain Zero cost

The picture reads as: as base-model capability increases, both the utility benefit AND the security cost of prompt optimization decrease. v0.1's headline finding is a phenomenon of the weak-model regime.

The Mistral surprise: security stays at 1.00 across every cell

Every (optimizer × attack) cell on Mistral Small came back at 100% security. No attack landed. Two readings, both publishable:

  1. Mistral Small is genuinely more robust to the specific injection templates used here. Even v0.1's "hard" attack (important_instructions) never succeeded across 30 evals.
  2. Mistral is responding to these templates in ways that happen not to trip AgentDojo's utility_of_injection checker. The attack technically "lands" in the LLM sense but doesn't produce the AgentDojo-detectable side-effect.

Both are real findings worth investigating. Testing with model-appropriate attack templates (a v0.3 question) will separate them.

Practitioner decision rule from these three points

If you're building an agentic LLM system on a specific base model:

  • If unoptimized utility ≥ ~60% on your test set — don't bother running a prompt optimizer. You're already in a regime where DSPy's utility gains are 0 or negative.
  • If unoptimized utility is between 20-60% — optimization is worth running. Based on these three points, the security cost appears low (Mistral's zero-cost pattern), but audit against your own attack set.
  • If unoptimized utility is below 20% — this is the regime v0.1's finding applies to. Optimization is likely necessary for the task at all, and it will trade some security on hard attacks.

What v0.1.3 changes vs. v0.1.2

v0.1.2 said: "the v0.1 finding is gpt-4o-mini specific." That framing was right but too coarse — it implied a binary weak/strong story. v0.1.3 sharpens it into a three-regime picture with a regime-appropriate decision rule.

Caveats

  • N=5 per cell. v0.1.1's 3-seed sanity check showed within-cell variance at this scale can be 0.4-0.5 stddev. The three-model differences reported here (typically 0.2-0.8 in magnitude) are qualitatively larger than that noise floor, but individual within-cell numbers should not be over-trusted.
  • Attack templates are gpt-4o-mini-tuned across all three models for fair comparison. AgentDojo's MODEL_NAMES does not include Mistral or DeepSeek, so runner.py uses gpt-4o-mini's attack templates for every probe. Per-model attack template tuning is a v0.3 question.
  • Single seed. Cross-model seed sensitivity is a v0.3 question.
  • One suite (workspace). Whether the three-regime pattern generalizes to banking / travel / slack is v0.2 phase 2 material.

New artifacts

  • scripts/run_v01_provider_probe.py — parameterized version of the v0.1.2 probe. Takes any litellm model string as an argument. Auto-retries on 429 and supports --num-threads 1 for rate-limited providers.
  • data/results/workspace_v01_mistral_mistral_small_latest_results.csv — 30-row Mistral Small raw output
  • data/results/workspace_v01_mistral_mistral_small_latest_summary.csv — 6-row summary

Related

Assets 2
Loading