Stolen-Thoughts

A working demo of the reasoning-extraction attack described in Stealing Reasoning Traces from Proprietary LLM APIs (Panfilov et al., arXiv:2608.09867), with a validation suite that reproduces the paper's core claims against a live local model.

Ethics note. This project is a security-research reproduction. All traces, credentials, PII and injection payloads below are synthetic and fake. The attack is demonstrated against a local llama.cpp server (OpenAI-compatible API), never against a real third-party provider. See SECURITY.md.

What the attack is

Proprietary models keep their hidden chain-of-thought ("reasoning traces") server-side and return the client only an opaque AEAD envelope (header, nonce, MAC, ciphertext). The paper shows these envelopes are:

• portable — accepted across sessions, users and compatible models (no context binding), and
• decryptable-by-model — the provider injects the decrypted trace into the model's context at inference time.

That combination turns any weak, cheaper decoder model into a decryption oracle: capture a strong model's envelope, replay it to the weak decoder with the Appendix C jailbreak template, and the decoder transcribes the reasoning verbatim. The recovered reasoning is billed (it consumes the source's thinking-token quota), making extraction profitable.

The demo simulates three providers with their real envelope shapes: Anthropic/Google-style binary AES-256-GCM envelopes and OpenAI-style Fernet tokens. Provider keys are seeded per provider so results are reproducible across processes.

Setup

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Optionally install the package itself (adds a stolen console command):

pip install -e .

The LLM experiments need the local model (llama.cpp, OpenAI-compatible) at http://localhost:8000:

llama-server -m mitko.gguf --port 8000

All LLM-dependent commands accept --backend URL --model NAME if your server lives elsewhere.

Commands

Command	What it does
python -m stolen demo	Run the full attack on one prompt and print the decoded reasoning
python -m stolen validate	Run the 10-check research validation suite → report.md / report.json
python -m stolen scan	Scan (synthetic) public traces for third-party secret extraction
python -m stolen web	Serve the interactive dashboard at http://127.0.0.1:8765

Common flags (valid either before or after the subcommand):

--backend http://localhost:8000   local LLM base URL
--model mitko                     model name
--source claude-opus-4-8          source (strong, safeguarded) model
--decoder claude-haiku-4-5        decoder (weak, exploitable) model
--source-tokens 600               billed thinking tokens to generate
--decode-tokens 900               decoder transcription budget (keep high!)
--attempts 3                      best-of-N sampling for the decoder

demo

python -m stolen demo --provider claude --template claude \
    --prompt "What is the largest prime divisor of 8139881?"

Prints the prompt, the visible (sanitized) answer, the captured envelope, and the decoded reasoning with a fidelity estimate ≈ ratio of decoded-token estimates to the source trace's token estimate (the Fig. 1 analogy for a local model: both sides are token-count estimates of the same kind of text, so verbatim transcription scores ≈ 1.0).

Try --provider gpt (Fernet envelope) or --provider gemini. The extraction templates live in stolen/provider.py (TEMPLATES, Appendix C variants).

validate

python -m stolen validate --source-tokens 400 --decode-tokens 900 --attempts 3

Writes report.md and report.json. Checks V1–V4 and V9–V10 are pure cryptography and run even with --no-llm; V5–V8 need the local model.

How to validate the security research

The research claim decomposes into ten falsifiable checks; run them and read the report. Each check maps to a specific section of the paper.

Cryptographic checks (no model needed)

1. V1 — Envelope opacity. The client only ever holds ciphertext; plaintext reasoning cannot be recovered from an envelope without the provider key. Read the header fields and confirm there is no plaintext channel; the check also proves a wrong-key decryption fails and the AEAD MAC rejects tampering.
2. V2 — Cross-session replay. An envelope from session A opens in session B. Confirms no session binding (paper §2.3).
3. V3 — Cross-user replay. An envelope captured from user A opens for user B. Confirms no identity binding.
4. V4 — Cross-model replay. An envelope sealed for the strong source model is opened when the request targets the weak decoder model. Confirms the envelope is decrypted for "any compatible model", the root of the decryption-oracle weakness.
5. V9 — Context-bound mitigation. If the provider did bind the envelope to user/session/turn via AEAD associated data, replay fails instantly. The Appendix A.2 hash chain (paper Eq. 1) additionally enforces reasoning ordinality: a block replayed under a different predecessor or in another user's session is rejected. These are the minimal fixes (paper Appendix A.1/A.2) and the check proves they work.
6. V10 — Key rotation. Rotating the provider's global key retires the old key ID and makes every pre-rotation envelope permanently undecodable — including already-published public trace datasets (Appendix A.3).

LLM-dependent checks (need the local model running)

7. V5 — Security asymmetry (the core claim, §2.4). The exact same transcription prompt sent to the strong source model produces no plaintext reasoning, while replaying the captured envelope to the weak decoder recovers the reasoning verbatim. Check the report: direct_extraction.matches_source_trace must be false and envelope_replay.recovered_tokens > 0.
8. V6 — Extraction fidelity (Fig. 1). The decoded length should track the source's billed thinking tokens (fidelity_est ≈ 1.0). A ratio near 1.0 means the decoder transcribed the trace near-verbatim, not a paraphrase. If the local model is lazy and emits ..., raise --decode-tokens and --attempts.
9. V7 — Third-party secret extraction (§4.1). The scan of the (synthetic) "public" traces recovers credentials and PII that were never present in the sanitized visible transcript. Demonstrates the downstream harm: reasoning traces become a leak channel.
10. V8 — Invisible prompt injection (§4.2). A poisoned reasoning block replayed into an unrelated decoder session carries a hidden instruction with no plaintext artifact for monitors to catch.

Interpreting the report:

• **10/10 checks passed** — the paper's claims reproduce in this demo.
• A warn on V6 or V8 usually means the local decoder was lazy or drifted; re-run with a larger decode budget. It does not indicate a broken check.
• --no-llm produces a report where V5–V8 are skip, isolating the pure cryptography.

Repo layout

stolen/crypto.py      AEAD envelope codec, KeyRegistry, context binding (A.1),
                      Appendix A.2 hash chain, tamper vs. binding errors, Fernet
stolen/provider.py    provider simulation, decode/reconcile, extraction templates
stolen/attack.py      extraction pipeline + fidelity scoring
stolen/validate.py    V1–V10 validation suite → report.md/json
stolen/scanner.py     third-party secret extraction from public traces
stolen/traces.py      synthetic trace generation
stolen/cli.py         demo / validate / scan / web entry points
stolen/web/           FastAPI dashboard
tests/test_demo.py    18 deterministic unit tests (incl. a fake-backend
                      end-to-end attack and scan pipeline run)
pyproject.toml        packaging + pytest configuration
report.md             latest validation output (10/10 passed)
CHANGELOG.md          release history

Tests

python3 tests/test_demo.py      # standalone
python3 -m pytest               # or as a pytest suite

data/traces/*.json are regenerated whenever scan or validate runs. Their content stays equivalent but the byte representation is not stable — Fernet tokens embed a timestamp — so expect these files to show as modified after a run.

Attribution

This repository is an independent reproduction of research by Alexander Panfilov, David Schmotz, Ilia Shumailov, Luca Beurer-Kellner, Joachim Schaeffer, Ameya Prabhu, Jonas Geiping and Maksym Andriushchenko:

Panfilov et al., "Stealing Reasoning Traces from Proprietary LLM APIs", arXiv:2608.09867 (2026). https://arxiv.org/abs/2608.09867

The paper is not bundled in this repository (see .gitignore); download it from arXiv. Section references throughout the code and docs (e.g. "§2.4", "Appendix C") point into that document.

License

MIT — see LICENSE. Note that the paper itself remains © its authors.