Design-doc - #1
Conversation
…ownership table , known risks(2 known risks) , Defination of done, Public API design sections in design doc
Expanded the technical approach section with detailed methods, results, and recommendations for near-duplicate detection and language contamination detection. Updated evaluation plan and addressed package name conflict and team size reduction issues.
|
Added my Technical Feasibility section (near-duplicate threshold analysis on 20 real RUEmoCorp pairs, langdetect reliability testing on 12 real samples, and scoring formulas for all 5 checks). Also added a third Known Risk based on what I found — langdetect can't reliably detect Roman Urdu at any text length tested (0% correct across 44 detections), which affects check_language_contamination design. Included a proposed mitigation in the risk entry. Reviewed the Problem Statement — looks solid and concrete. Ready for your review on the |
|
|
||
| 2. **Reduced team size.** Unlike Projects A and B, this project has two members rather than three ,there is no dedicated Integration/Evaluation Engineer. Fixture generation, test-suite ownership, and performance/coverage verification are split between the Lead and Research/Implementation Engineer. **Mitigation:** evaluation responsibilities are explicitly assigned per-file in the Module Ownership Table above, with the Lead performing an independent final verification pass rather than relying solely on self-review. | ||
|
|
||
| 3. **langdetect cannot reliably detect Roman Urdu.** Testing on 12 real RUEmoCorp messages at 3/5/10/20-token lengths (44 total detections) found `langdetect` never correctly identified Roman Urdu at any length tested — guesses scattered across 10+ unrelated languages (Somali, Indonesian, Estonian, Tagalog, etc.) with no improvement at longer lengths. A sanity check confirmed the tool works correctly on languages it does support (100% accuracy on real English/French/Spanish text), ruling out a tool malfunction — the issue is that Roman Urdu simply has no language profile in `langdetect`, a structural limitation rather than a short-text weakness. **Mitigation:** `check_language_contamination` will not use "not detected as Urdu" as its contamination signal, since that would flag the entire dataset as 100% contaminated. Instead, the check will only flag samples confidently (>0.9 probability) identified as a real, unrelated language (e.g. genuine English text mixed into the corpus). `DetectorFactory.seed = 0` will be set at module init to ensure reproducible results, since testing also confirmed detection is non-deterministic without it (2/5 test texts returned different results across repeated runs on identical input before the seed was set). |
There was a problem hiding this comment.
That's a good catch, flagging "not detected as Urdu" as contamination would have made every Roman Urdu dataset score near-zero on this check, which is a false signal. Reframing it to, only flag confident detection of a genuinely different language, is a good fix. But what happens when language="auto" is passed for a Roman Urdu dataset? since langdetect can't recognize Roman Urdu at all, "auto" mode has no reliable way to even establish what the "expected" language is in the first place for this specific case. That might need its own documented limitation.
khadijja1
left a comment
khadijja1
left a comment
There was a problem hiding this comment.
Strong Technical Feasibility section,the langdetect testing was thorough . The Roman Urdu mitigation approach is sound. One thing worth clarifying before merge: what expected_language behavior looks like when language="auto" is passed for a Roman Urdu dataset, since langdetect has no profile for it. Overall nice work.
Khubaib8281
left a comment
Khubaib8281
left a comment
There was a problem hiding this comment.
I appreciate your project clarification. make sure to perform as described in the design doc. best of luck
3 participants
What does this PR do?
This PR adds the Design doc with these sections: project summary , problem statement , evaluation plan , module ownership table , known risks(2 known risks) , Defination of done, Public API design sections filled in.
Which module(s) does it touch?
Docs/Design doc
How did you test it?
Manual review
Edge cases handled:
N/A
Checklist: