promptdiff — static analysis for LLM prompts (lint, diff, score, A/B test)

[HadiFrt20]

If you are routing prompts across providers with LiteLLM, this might help catch issues before they hit any model.

promptdiff is a CLI that treats .prompt files as structured documents and applies static analysis:

• 10 lint rules: conflicting constraints, vague language, role confusion, missing injection guards, example quality
• Semantic diff: rates changes by behavioral impact ("Output consistency may decrease" when examples removed)
• Quality score: 0-100 across structure, specificity, examples, safety, completeness
• A/B compare: test same prompt against different models and score outputs

npm install -g promptdiff
promptdiff lint my-agent.prompt
promptdiff compare v1.prompt v2.prompt --input "test" --model gpt4o

Also supports prompt composition (extends/includes), custom rules via .promptdiffrc, and MLflow integration.

3 deps, 217 tests, everything local.

GitHub: https://github.com/HadiFrt20/promptdiff