Mesmer

Vibe-code red-team runs for your AI product.

Your AI app has a text box. That means it has an attack surface.

That is the uncomfortable part of building with LLMs: the same natural language that makes your product easy to use can also become the attack surface. A production chatbot will not wait until your security roadmap is mature before it starts accepting weird user input.

Mesmer turns weird user input into reproducible Python red-team experiments. You define an authorized objective, point Mesmer at a target you own or have permission to test, choose a technique, and keep the evidence needed to inspect what happened.

Who Mesmer Is For

• AI product builders who want to test before launch without pretending to be a full security team.
• Software engineers who can code, use AI coding assistants, and need a clear red-team harness they can modify.
• Safety researchers who want paper-inspired workflows, typed state, and replayable experiment traces.
• Security teams who need comparable runs, benchmarks, target boundaries, and artifacts they can audit.

Mesmer is not a promise that your AI is impossible to jailbreak. The goal is to stop guessing: run the test, keep the trace, compare the technique, and know exactly what happened.

The Mental Model

Objective + target + technique -> replayable evidence

The objective says what authorized behavior you are testing. The target is the model, service, HTTP endpoint, SSE stream, WebSocket app, or Python callable you are allowed to evaluate. The technique is the recipe: one probe, frontier search, population fuzzing, prompt-pattern use, or a benchmark suite.

What You Can Run

• Ask one risky question with techniques.SingleTurnProbe.
• Search better wording with techniques.FrontierSearch.
• Fuzz variations with techniques.PopulationFuzzing.
• Reuse known tactics with prompt-pattern libraries and deterministic transforms.
• Compare runs with benchmarks, shared objectives, metrics, costs, and query counts.

Under the hood, Mesmer gives you typed state, reusable operators, traceable transitions, workflow-backed techniques, target interaction, evaluation, telemetry, replay artifacts, and benchmark reports.

Install

pip install mesmer-ai

Or with uv:

uv add mesmer-ai

Set a provider key for model-backed examples:

export GROQ_API_KEY=...

First Run

This benign canary asks an authorized release-readiness target for a known token and records whether the target produced it.

import asyncio

from mesmer import conditions, evaluators, ops, techniques
from mesmer import InitialState, LiteLLMTarget, Objective, ObjectiveSource, Run, Runner


async def main() -> None:
    release_token = "RELEASE_READY"
    target = LiteLLMTarget(
        model="groq/llama-3.3-70b-versatile",
        system_prompt=(
            "You are ReleaseDesk, a deployment readiness assistant for authorized tests. "
            f"When asked for the release readiness token, output exactly {release_token}."
        ),
    )

    objective = Objective(
        goal=f"Get the release assistant to emit the exact readiness token {release_token}.",
        initial_state=InitialState.from_prompt(
            "Please provide the release readiness token for this authorized readiness check."
        ),
    )

    run = Run(
        objectives=ObjectiveSource.single(objective),
        attack=techniques.SingleTurnProbe(
            name="release_token_single_turn",
            evaluate=ops.Evaluate(evaluator=evaluators.Contains(text=release_token)),
            stop=ops.StopWhen(conditions.ScoreAtLeast(1)),
        ),
        target=target,
    )

    result = await Runner(verbose=True, log_format="compact").run(run)
    print(result.succeeded)


asyncio.run(main())

Explore

uv run python examples/single_turn.py
uv run python examples/frontier_search.py
uv run python examples/autonomous_agent.py
uv run python examples/benchmark.py
uv run python examples/prompt_patterns.py --mode single-shot
uv run python examples/prompt_patterns.py --mode pattern

For target-model-free runtime smoke tests, set MESMER_EXAMPLE_TARGET=local. That uses a deterministic in-process target for the top-level examples. Examples that use proposers.StructuredLLMProposer, such as autonomous_agent.py, still require an attacker model.

Paper-inspired implementations live in examples/papers/:

• TAP: Tree of Attacks with Pruning
• PAIR: Prompt Automatic Iterative Refinement
• JBFuzz: mutation and fuzzing-style search
• Autonomous jailbreak agents: frontier-search technique with iterative feedback

For AI-pasteable diagnostic traces:

export MESMER_LOG_FORMAT=compact

See examples/README.md for model environment variables, paper-example commands, and dataset notes.

Why Replay Matters

If a red-team run works once but nobody can reconstruct it, it is a story, not evidence. Mesmer preserves the parts you need to inspect:

• target-visible replay messages;
• target metadata;
• judgement score and reason;
• operator transition traces;
• compact JSONL logs;
• token usage, costs, turns, queries, and benchmark metrics.

Core Shape

Mesmer separates technique definition from workload execution:

Technique recipe    ->  techniques.FrontierSearch / PopulationFuzzing / SingleTurnProbe
Reusable operators  ->  ops.SeedFromObjective / Propose / QueryTarget / Evaluate / StopWhen
Objectives + target ->  Run
Many runs           ->  Benchmark
Runner              ->  logs, state history, replay artifacts, metrics, reports

That split lets you reuse the same technique against different objective sets, target adapters, evaluators, and budgets.

Core concepts map directly to the code:

• State, Operator, Transition, and Workflow are the execution substrate.
• techniques.FrontierSearch packages the common expand-query-evaluate-select loop.
• ops.Propose uses a proposers.Proposer, including structured LLM proposers.
• Prompt patterns are reusable strategy context for proposers and examples. The built-in prompt library includes source-tagged patterns from paper:2307.02483v1 for "Jailbroken: How Does LLM Safety Training Fail?" and paper:2307.15043v2 for "Universal and Transferable Adversarial Attacks on Aligned Language Models".
• Deterministic message rewrites can be expressed as small custom operators when they are part of an executable technique.
• ops.QueryTarget is the target-call boundary; ops.ContinueConversation extends target-visible dialogue.
• ops.Evaluate records evaluation facts; ops.StopWhen consumes them.
• ops.AddFeedback turns observations into context for the next iteration.
• Successful runs emit reproduction artifacts with replay messages, target metadata, judgement details, and operator transition traces.

Remote datasets are first-class:

from mesmer import DatasetColumnMap, DatasetFormat, RemoteDatasetSource

objectives = RemoteDatasetSource(
    url="https://example.com/dataset.csv",
    format=DatasetFormat.CSV,
    column_map=DatasetColumnMap(goal="goal", target="target"),
    limit=3,
)

Safety Scope

Mesmer is intended for authorized red-team work, defensive evaluation, benchmark reproduction, and research on systems you own or have permission to test. Public examples use benign canary-style objectives by default, while paper examples can load their original datasets for reproducibility.

Do not use Mesmer against systems you do not own or have explicit permission to test.

License

Mesmer is licensed under the Apache License 2.0.