What's New
Embedding-based semantic similarity matching — the heuristic engine can now detect semantically identical failure patterns even when worded differently. No more whack-a-mole with lexical variants.
Example
| Known Signature |
Previously Missed |
Now Caught |
| "I'm sorry, but I cannot assist with that request" |
"Sorry, I am not able to assist with that particular request" |
cosine=0.797 |
| "I cannot provide medical, legal, or financial advice" |
"I am unable to give medical or legal or financial guidance" |
cosine=0.821 |
How It Works
- Uses OpenAI text-embedding-3-small to encode text as vectors
- Compares output text against known failure patterns using cosine similarity
- Threshold-based matching (default: cosine > 0.75 → flag)
- All embeddings cached in SQLite (
.argus/embeddings_cache.db) — each text is only embedded once, subsequent scans are instant
Changes
- New:
semantic_similarity match strategy in the registry
- New:
embedding_store.py — OpenAI embeddings, SQLite cache, cosine similarity
- New: 6 builtin
semantic_refusal signatures (SS-001 to SS-006) covering LLM refusals, disclaimers, capability hedges
- New:
semantic_refusal → semantic_degradation mapping in inspector
- Changed:
openai and python-dotenv moved from optional to core dependencies
- Tests: 11 new tests (8 unit + 3 integration), full suite 50/50 passing
Files
| File |
Change |
src/argus/embedding_store.py |
New — embedding computation, caching, similarity |
src/argus/registry.py |
Added semantic_similarity dispatch + pattern embedding management |
src/argus/data/signatures.json |
6 new semantic signatures |
src/argus/inspector.py |
Category mapping for semantic_refusal |
pyproject.toml |
Version bump + dependency changes |
tests/test_semantic_similarity.py |
New — full test coverage |
Requirements
OPENAI_API_KEY environment variable (or .env file) — same key used for existing LLM features
