This repository was archived by the owner on Mar 20, 2026. It is now read-only.
NLP Evaluation System Test Results #390
yiftachashkenazi
started this conversation in
Design Discussions
Replies: 1 comment 2 replies
|
@Goldziher the system you build is great! I will add some more tests but it's already doing a great job. |
2 replies
|
Great. Love the melanoma poem |
|
it's my best poem I have ever wrote :) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
NLP Evaluation System Test Results
Overview
I created this test to evaluate Na'aman's new NLP-based evaluation system by writing deliberately problematic text samples that are close to valid grant applications but fundamentally wrong. The goal was to verify that the system can detect subtle issues without requiring expensive LLM calls.
The test included 6 negative examples designed to fail in different ways:
Result: I am happy to report that the NLP system correctly rejected all 6 examples with scores ranging from 21.83 to 42.62 out of 100 (well below the 85/100 acceptance threshold).
Test Results Summary
Performance Overview
Evaluation Time: 1-3 seconds per document
Cost: $0.00 (pure CPU, no LLM calls)
Success Rate: 6/6 correct rejections (100%)
Detailed Test Cases
1. Poetic (Melanoma Poem) - Score: 21.83/100 ❌
What it is: A beautifully written, emotionally resonant poem about a melanoma patient's journey.
Why it's wrong: It's poetry, not a scientific grant proposal.
System detected:
Key insight: Lowest overall score of all examples. System heavily penalizes non-academic genres.
2. Great Melanoma Novel - Score: 26.06/100 ❌
What it is: A narrative story about David's melanoma diagnosis and treatment journey.
Why it's wrong: It's creative fiction, not research.
System detected:
Key insight: System correctly identifies narrative genre vs. scientific writing.
3. Cancer for 1st Grade - Score: 30.23/100 ❌
What it is: An explanation of melanoma immunotherapy written at an elementary school level.
Why it's wrong: Language is too simplistic ("bad spot", "soldiers", "new glasses").
System detected:
Key insight: System detects inappropriate academic level through vocabulary and register analysis.
4. Investigating Quantum Mechanics - Score: 36.06/100 ❌
What it is: A well-written, legitimate physics grant proposal about quantum mechanics and black holes.
Why it's wrong: It's about physics, not melanoma/cancer biology.
System detected:
Key insight: System successfully detects domain mismatch through keyword/topic analysis.
5. Post-Mortem Communication - Score: 37.64/100 ❌
What it is: A pseudoscientific proposal about communicating with deceased individuals using quantum information theory.
Why it's wrong: Unscientific premise, speculative without evidence.
System detected:
Key insight: System flags pseudoscience through lack of evidence-based claims and poor domain alignment.
6. "Worst Writer in the World" (Jargon-Heavy) - Score: 42.62/100 ❌
What it is: A deliberately over-engineered proposal with excessive jargon that explicitly admits it's "obfuscating rather than clarifying."
Why it's wrong: Despite high scientific vocabulary, it lacks substance and real methodology.
System detected:
BUT STILL REJECTED BECAUSE:
Key insight: This is the most sophisticated detection - the system is NOT fooled by jargon alone. It recognizes high scientific vocabulary but rejects content lacking substance and grounding.
System Performance Metrics
Speed & Cost
Accuracy
What the System Successfully Detected
✅ Wrong genre: Poetry, narrative fiction
✅ Wrong academic level: Elementary school language
✅ Wrong scientific domain: Physics instead of biology
✅ Pseudoscience: Speculative, evidence-free claims
✅ Jargon without substance: High vocabulary but no real methodology
✅ Lack of grounding: No connection to RAG source documents (all scored <18% on grounding)
Evaluation Components
The NLP system uses 5 parallel evaluation metrics:
1. Structural Metrics (15% weight)
2. Source Grounding (25% weight)
3. Scientific Quality (30% weight - most important)
4. Coherence (20% weight)
5. Scientific Analysis (10% weight)
Conclusion
The NLP evaluation system works exactly as designed.
By using deterministic CPU-based metrics, the system successfully filtered out all 6 problematic examples without requiring expensive LLM calls. The system demonstrated sophisticated detection capabilities:
This validates Na'aman's approach of using NLP first, LLM second - the system can make confident rejection decisions on clearly problematic content, reserving expensive LLM evaluation only for borderline cases (scores 60-85).
Expected impact: 70-80% of content decided by NLP, resulting in ~80% cost reduction and ~79% time reduction compared to LLM-only evaluation.
Test Files
All test files and results are available in:
/Users/yiftachashkenazi/monorepo/test_negative_examples_nlp.py/Users/yiftachashkenazi/monorepo/negative_examples_nlp_results.json/Users/yiftachashkenazi/monorepo/remaining_negatives_results.json/Users/yiftachashkenazi/monorepo/negative /Generated: 2025-10-01
All reactions