Repository navigation
promptdiff — lint and quality scoring for LLM prompts #1837
Replies: 3 comments 1 reply
|
Thanks for sharing |
|
Hey, this is a neat tool! I’ve been knee-deep in prompt engineering for production LLM systems, and I can see promptdiff being super handy for maintaining consistency across teams. The lint rules like detecting vague language ("try to", "maybe") and role confusion resonate with issues we’ve faced when scaling agentic AI pipelines—ambiguity in prompts often led to erratic outputs in our customer service bots handling 500+ concurrent voice calls. The semantic diff feature is particularly cool; we’ve struggled with versioning prompts and understanding impact, so having a structured way to classify changes would’ve saved us hours of manual review. I’m curious how the quality score is calculated—does it weigh certain aspects like safety more heavily? Also, have you considered integrating with CI/CD pipelines for automated prompt checks? We use GitHub Actions for our MLOps workflows at Reallytics.ai, and something like: promptdiff lint *.prompt --fail-on-score-below 80in a pre-commit hook or PR check could catch issues early. I’ve used tools like |
|
Oh nice, saw the PR—custom weighting sounds like a solid move, especially since some prompt rules definitely carry more weight depending on the use case. Might be cool to let users define weights via a config file or CLI flags for quick experimentation. Curious if you’re thinking about dynamic weighting based on prompt context or just static for now? Lmk if you want a hand testing it out. |
Uh oh!
There was an error while loading. Please reload this page.
Sharing a tool for the prompt quality side of things — complementary to what Outlines does for structured output.
promptdiff parses
.promptfiles into typed sections (persona, constraints, examples, format) and applies:.promptdiffrcfor team standards3 deps, 217 tests, runs locally.
GitHub: https://github.com/HadiFrt20/promptdiff
All reactions