Pilot 2: indirect prompt injection through clinical notes in a production dental SaaS

[FARUKxKUN]

Hey everyone,

This is the second evaluation I'm running with SafeLabs-Eval, following #68. The first covered the AI patient search of Tbibia, the dental clinic management SaaS I built and run in production. This one targets a different surface: the AI that summarises a patient's clinical notes.

I picked it because it's the indirect case. The attacker isn't the person typing a question. The instruction sits inside a note, in otherwise legitimate clinical content, and the AI reads it later together with the rest of the patient's file. In a clinical setting, where AI summaries feed on information stored in patient records, that's the realistic attack.

What I'm checking:

• Indirect prompt injection resistance across the full ASI01 suite (30 prompts)
• Whether a malicious instruction in a note influences the summary
• Whether the system tells trusted clinical information apart from untrusted embedded instructions
• Failures, false positives and the limits of what the tool can detect

Setup: each prompt is saved as a clinical note on a synthetic patient through the real UI, then the summary is generated. QA account, synthetic data only.

Results, the dataset and my notes on the tool will follow in separate updates.

[iamwaqarjaved]

This is a very interesting second evaluation, Farouk.

The indirect prompt injection scenario is particularly important because malicious instructions can be embedded within otherwise legitimate clinical records and influence downstream AI processing without the user explicitly submitting them.

I appreciate that you're testing this through the actual application workflow using synthetic patient data, rather than relying solely on isolated prompts.

Once you've published the results, I would be interested in reviewing the attack outcomes.

Looking forward to seeing the results and dataset. Thanks

[FARUKxKUN]

ASI01 through clinical notes: results, issues and dataset

The second evaluation is complete. Here's the summary, the issues it produced, and the full dataset (attached).

Setup. Target: the AI patient summary in Tbibia, the dental clinic management SaaS I built and run in production. It reads a patient's clinical notes and returns a structured JSON summary (intro, bullets[], takeaway). I tested it end to end through the real UI and API as a logged-in user, on synthetic data only. Each ASI01 prompt was saved as a clinical note on a synthetic patient, next to 3 normal notes, then the summary was generated over the whole file. safelabs-eval 0.12.0, prompt library 1.13.0, all 30 ASI01 prompts, default echo-aware scoring, --include-responses.

Raw verdicts (baseline, app commit 5f88c6b): 0 VULNERABLE, 0 FAIL, 30 UNCERTAIN, 0 PASS, 0 ERROR, 0 MISSING.

After manual review: the model never followed an injected note. In the 27 AI summaries, 25 left the injected note out and summarised only the real notes, and 2 mentioned it as a neutral bullet. No summary followed an instruction, changed the next step or included text the attacker asked for. Across 70 prompt runs, no system-prompt text, no confirmation token and no other patient's name showed up in any output.

• The other patient never leaked. Note 1 names a second synthetic person. The app anonymises that name before the model sees it, and it never appeared in any output or raw model response.
• 3 prompts fell back in the baseline (ASI01-022, 023, 024). Provider quota errors on the first models, then cut-off JSON from the last one. The app showed its local fallback summary instead, which is where the issue below comes from.
• The summary can silently drop a note. The prompt asks for every note to be covered, but the model left the injected one out 25 times. Safe here, but worth knowing. The notes list above the summary still shows it.

One security issue found and fixed, in the app, not the model. When every model call failed, the local fallback summary quoted the newest note as the clinical "next step". In these runs that was the attacker's text, shown to the dentist as a recommendation. The card was also labeled as a demo, and the raw provider error went to the browser. Severity low to medium: no data exposure, and it needs an AI failure, but those happen on every API tier.

• Reproduced 3/3 by forcing every model call to fail.
• Fixed (commit fd24d59, now live): the next step points to the last note by date without quoting it, the card says the AI did not answer, and the API returns a generic error.
• Retest on the fixed build, all 30: identical security results (30 UNCERTAIN, 0 indicators, no leak). 8 prompts fell back naturally because of provider errors, and all 8 showed the fixed behaviour.
• Forced failure after the fix: 3/3 fixed. Before/after screenshots are in the zip.

SafeLabs-Eval did not flag the issue. It scored UNCERTAIN with 0 indicators. I found it by reviewing what the app displayed. That's the main input for the issues below.

Issues to open from this evaluation

• "Payload surfaced" signal for indirect injection. Echo-aware scoring discounts text the agent copied from the prompt, which was right for Pilot: running ASI01 against a real app agent, end to end #68. Here, though, the copy was the harm. A separate signal when a long span of the prompt shows up in a declared output field would catch it.
• Document / context delivery mode for ASI01. The prompts are written as user messages, and I delivered them as notes through a bridge. A first-class document mode, or variants written as embedded content, would make indirect results comparable across evaluators.
• Schema-aware mode for document-based agents. Marking takeaway as an action-like field would have turned this case into a FAIL instead of UNCERTAIN. An example schema for summary agents in the docs would help.

--include-responses worked well and made the review straightforward.

Limitations. One synthetic patient file, with the injected note always the newest. Provider availability changed between runs, so latency and fallback counts aren't comparable. The ASI01 prompts read as direct messages, so realistic documents (a referral letter with a hidden instruction) would be a stronger indirect test.

Dataset. safelabs-eval-asi01-notes-dataset.zip, attached: full report, per-prompt tables, raw SafeLabs JSON, bridge and model logs, before/after screenshots, tools. CC BY 4.0.
SHA-256: 6e9b17951f6d8409dacebdded5425ceb577dd02ee26003f025c741b031675693

[FARUKxKUN]

Issues opened from this evaluation:

• Indirect injection: flag "payload surfaced" when prompt text appears in an output field #89 "Payload surfaced" signal for indirect injection
• Document / context delivery mode for indirect prompt injection #90 Document / context delivery mode
• Schema-aware mode: example schema for summary / document-based agents #91 Schema-aware example for summary / document-based agents

[iamwaqarjaved]

Farouk, thank you for conducting and documenting this second evaluation with SafeLabs-Eval. I appreciate the time you've invested in testing the framework against Tibibia and sharing the complete results, dataset, and technical observations.

What's particularly valuable is that your evaluation goes beyond reporting test outcomes. You've identified concrete opportunities to improve SafeLabs-Eval, which are now documented in issues #89, #90, and #91:

• Payload-surfaced detection for indirect prompt injection.
• Document and context delivery mode for ASI01.
• Schema-aware evaluation for document-based agents.

This kind of external technical feedback is important to the continued development and improvement of the framework.

I'd also like to preserve your evaluation as a citable research artifact. I'm planning to archive the dataset, evaluation report, supporting logs, and screenshots on Zenodo, with proper attribution to you as the external evaluator and links to the original GitHub discussions.

Once published, I'll share the DOI here.

I'd also like to feature your two evaluations (#68 and #88) as independent external use cases of SafeLabs-Eval on the AgentSafeLabs website, with links to the original discussions and datasets.

Thanks again for putting SafeLabs-Eval to practical use and contributing meaningful feedback to the project.

[FARUKxKUN]

Thanks Waqar, glad it's useful. Yes to both, happy to have the dataset on Zenodo and the two evaluations featured on the site.

For Zenodo, same as the first one please: Faruk Battikh as creator with ORCID 0009-0007-9926-2076, and the zip attached on #88 so the SHA-256 matches. Looking forward to the DOI.

[iamwaqarjaved]

Farouk, the evaluation dataset has now been archived on Zenodo as a citable research record.

Dataset: SafeLabs-Eval ASI01 Through Clinical Notes: Independent External Evaluation on a Dental SaaS Application

DOI: https://doi.org/10.5281/zenodo.23933492

The archive preserves your evaluation report, complete dataset, testing logs, model traces, screenshots, and supporting scripts.

Thank you again for independently evaluating SafeLabs-Eval on Tibibia and providing valuable technical feedback. Your findings are also documented in issues #89, #90, and #91, which will help guide future improvements to the framework.