Skip to content

fix(parser): retain single-letter language skills #855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
qtjg wants to merge 1 commit into from
Closed

fix(parser): retain single-letter language skills #855

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/lib/heuristics/corpus.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ const TRUTH_ANNOTATED_FIELD_FLOOR = 150;
* `npm run check:baselines` on every run, and bounded here — undescribed debt may
* not GROW. File the issue and flip the entry to `open`; then lower this.
*/
const UNFILED_TRUTH_CEILING = 9;
const UNFILED_TRUTH_CEILING = 8;

@s-annam s-annam Aug 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking (finding 1). The measured unfiled count on this head is 7, not 8 — npm run check:baselines lists exactly seven ⚠ … status "unfiled" entries after this PR resolves the two C-drop ones. main's 9 was tight against 9 real entries, so resolving two lands on 7.

Leaving it at 8 keeps the assertion green but opens one free slot, which contradicts this constant's own docblock three lines above ("undescribed debt may not GROW") — the next PR could add a brand-new unfiled disagreement and this gate would not notice.

#832 step 4 asks for a lowering by 2 and says to take the LOWER number when a sibling issue has already moved it; a sibling moved 10 → 9 before this branch, which is why the literal 8 from the issue text is now one too high.

I'm leaving this as prose rather than a one-click suggestion because it changes what a gate asserts — that's outside the bound on reviewer-applied fixes, even at one character.


/** Generator category = the fixture root's immediate subdirectory. */
function categoryOf(repoRelPdfPath: string): string {
Expand Down
6 changes: 6 additions & 0 deletions src/lib/heuristics/extract/skills.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,12 @@ describe("tokenizeSkillLine", () => {
expect(tokenizeSkillLine(",,,;;;")).toEqual([]);
});

it("keeps the defensible single-letter languages and rejects stray glyphs (#832)", () => {
const result = tokenizeSkillLine("C, R, D, X");
expect(result).toEqual(expect.arrayContaining(["C", "R", "D"]));
expect(result).not.toContain("X");
});
Comment on lines +82 to +86

@s-annam s-annam Aug 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Secondary (finding 2). The behaviour is right — I ran #832's whole AC token set against this head and every row passes. The gap is that only the X rejection is asserted.

J is the one worth pinning: #832 explicitly says "Do NOT add J, K, Q, or F", so a test holding J as rejected is what stops SINGLE_LETTER_SKILLS being widened later without an argument. And the mixed line is a positive control — without it, a regression that made this return [] for any line containing a stray glyph would still pass.

Verified outputs, so the suggestion below is asserting real behaviour rather than a guess:

"Languages: C, R, D"      -> ["C","R","D"]
"Skills: x, J, •, ("      -> []
"Skills: Python, x, J, C" -> ["Python","C"]
Suggested change
it("keeps the defensible single-letter languages and rejects stray glyphs (#832)", () => {
const result = tokenizeSkillLine("C, R, D, X");
expect(result).toEqual(expect.arrayContaining(["C", "R", "D"]));
expect(result).not.toContain("X");
});
it("keeps the defensible single-letter languages (#832)", () => {
expect(tokenizeSkillLine("Languages: C, R, D")).toEqual(
expect.arrayContaining(["C", "R", "D"]),
);
});
it("still rejects other single-character noise tokens (#832)", () => {
// `J` is a real language name but also a plausible stray glyph; #832
// deliberately keeps it out of the allowlist. Pin that so a later widening
// of SINGLE_LETTER_SKILLS has to argue for itself.
expect(tokenizeSkillLine("Skills: x, J, •, (")).toEqual([]);
});
it("keeps an allowlisted letter alongside real skills on one line (#832)", () => {
// Positive control: distinguishes rejecting noise from rejecting everything.
expect(tokenizeSkillLine("Skills: Python, x, J, C")).toEqual(["Python", "C"]);
});


it("drops the whole cell when a URL is present in a comma-separated list", () => {
// tokenizeCell's looksLikeContactLink check fires on the ENTIRE cleaned
// cell before the split. "github.com/janesmith" matches the path-slash
Expand Down
7 changes: 7 additions & 0 deletions src/lib/heuristics/extract/skills.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,12 @@ const PROFILE_HOST_RE =
* "Socket.io", "ASP.NET") that has no slash. */
const URLISH_RE = /(https?:\/\/|www\.|\b[a-z0-9-]+\.[a-z]{2,}\/\S)/i;

/** One-character tokens that are real, commonly-listed languages. The length
* floor in `isSkillToken` is a noise guard against stray glyphs left by column
* splitting; these are the only single characters that are not noise, so they
* are allowlisted rather than lowering the floor. */
const SINGLE_LETTER_SKILLS = new Set(["c", "r", "d"]);

/** True when a candidate skill token is really a professional-profile link
* (GitHub / LinkedIn / portfolio, etc.) or its bare heading word. Such links
* belong only in the contact/profile section, never in Skills. */
Expand All @@ -139,6 +145,7 @@ function looksLikeContactLink(tok: string): boolean {
}

function isSkillToken(tok: string): boolean {
if (tok.length === 1 && SINGLE_LETTER_SKILLS.has(tok.toLowerCase())) return true;

@s-annam s-annam Aug 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirming this line rather than flagging it: the early return true is exactly what #832 asked for, and it earns its keep — an allowlisted letter skips the numeric / URL / date-range / word-count checks below, none of which a bare letter can trip, so the early return states the intent instead of relying on them to be no-ops. toLowerCase() gives the case-insensitivity the issue's step 2 wanted, and I verified "Languages: c, r, d" survives as ["c","r","d"]. No change requested here.

if (tok.length < 2 || tok.length > 40) return false;
if (/^\d+$/.test(tok)) return false;
// A professional-profile link (or its bare "GitHub" / "LinkedIn" heading) is
Expand Down
6 changes: 3 additions & 3 deletions tests/fixtures/pdfs/google-docs/google-docs-skia-proxy-role-first-experience.expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
"phoneIsValid",
"skills"
],
"skillsCount": 11,
"skillsCount": 12,
"experienceCount": 2,
"educationCount": 1,
"projectsCount": 0,
Expand Down Expand Up @@ -74,7 +74,7 @@
"sectionSource": "regex",
"pageCount": 1,
"rawCharCount": 1428,
"extractedCharCount": 1146,
"extractedCharCount": 1147,
"sections": [
{
"name": "profile",
Expand All @@ -101,7 +101,7 @@
"hasSummary": false,
"experienceCount": 2,
"educationCount": 1,
"skillsCount": 11
"skillsCount": 12
},
"linkAnnotationCount": 0,
"disagreements": []
Expand Down
6 changes: 3 additions & 3 deletions tests/fixtures/pdfs/google-docs/google-docs-skia-proxy-role-first-experience.truth.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,9 @@
"note": "Role 2's employer line reads “Multicultural Engineering Program – State Polytechnic University”; `company` comes back as just the university — the program half is not lost, the parser puts it on `team`, but `experience.company` scores the `company` field alone. The identical shape is measured on unknown/single-column-title-below-anchor. Possibly a defensible org/team split rather than a defect — recorded rather than assumed, because ground truth's job is to state what the page says and let a human adjudicate."
},
"skills": {
"issue": null,
"status": "unfiled",
"note": "Two independent disagreements on one field: the single-letter token “C” is DROPPED from the Programming Languages row (the identical drop is measured on latex/multi-degree-coursework, so it is not fixture-specific), and “Fluent in Spanish” is admitted as a skill from the “Language:” row."
"issue": 833,
"status": "open",
"note": "The independent disagreement — “Fluent in Spanish” is admitted as a skill from the “Language:” row — is tracked by #833. The separate single-letter “C” drop was fixed by #832."
}
}
}
6 changes: 3 additions & 3 deletions tests/fixtures/pdfs/latex/deedy-resume-macfonts.expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
"skills",
"website_url"
],
"skillsCount": 22,
"skillsCount": 23,
"experienceCount": 6,
"educationCount": 3,
"projectsCount": 0,
Expand Down Expand Up @@ -83,7 +83,7 @@
"sectionSource": "markdown",
"pageCount": 1,
"rawCharCount": 3205,
"extractedCharCount": 2156,
"extractedCharCount": 2157,
"sections": [
{
"name": "profile",
Expand Down Expand Up @@ -114,7 +114,7 @@
"hasSummary": false,
"experienceCount": 6,
"educationCount": 3,
"skillsCount": 22
"skillsCount": 23
},
"linkAnnotationCount": 9,
"disagreements": []
Expand Down
6 changes: 3 additions & 3 deletions tests/fixtures/pdfs/latex/deedy-resume-openfonts.expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
"skills",
"website_url"
],
"skillsCount": 22,
"skillsCount": 23,
"experienceCount": 6,
"educationCount": 3,
"projectsCount": 0,
Expand Down Expand Up @@ -83,7 +83,7 @@
"sectionSource": "markdown",
"pageCount": 1,
"rawCharCount": 3207,
"extractedCharCount": 2158,
"extractedCharCount": 2159,
"sections": [
{
"name": "profile",
Expand Down Expand Up @@ -114,7 +114,7 @@
"hasSummary": false,
"experienceCount": 6,
"educationCount": 3,
"skillsCount": 22
"skillsCount": 23
},
"linkAnnotationCount": 9,
"disagreements": []
Expand Down
6 changes: 3 additions & 3 deletions tests/fixtures/pdfs/latex/multi-degree-coursework.expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
"website_url",
"work_authorization"
],
"skillsCount": 17,
"skillsCount": 18,
"experienceCount": 4,
"educationCount": 2,
"projectsCount": 3,
Expand Down Expand Up @@ -77,7 +77,7 @@
"sectionSource": "markdown",
"pageCount": 1,
"rawCharCount": 3428,
"extractedCharCount": 2625,
"extractedCharCount": 2626,
"sections": [
{
"name": "profile",
Expand Down Expand Up @@ -108,7 +108,7 @@
"hasSummary": false,
"experienceCount": 4,
"educationCount": 2,
"skillsCount": 17
"skillsCount": 18
},
"linkAnnotationCount": 6,
"disagreements": []
Expand Down
8 changes: 1 addition & 7 deletions tests/fixtures/pdfs/latex/multi-degree-coursework.truth.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,5 @@
"Raspberry Pi",
"iOS"
],
"knownWrong": {
"skills": {
"issue": null,
"status": "unfiled",
"note": "The single-letter token “C” is DROPPED from the Languages row while “C++” survives. Second independent measurement of the same drop (see google-docs/google-docs-skia-proxy-role-first-experience)."
}
}
"knownWrong": {}
Comment on lines 60 to +61

@s-annam s-annam Aug 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (finding 5), non-blocking. With the entry resolved, this leaves the only empty knownWrong in the corpus — 10 of the 16 truth sidecars omit the key entirely and none carries {}. Dropping the key matches the neighbours.

Suggested change
],
"knownWrong": {
"skills": {
"issue": null,
"status": "unfiled",
"note": "The single-letter token “C” is DROPPED from the Languages row while “C++” survives. Second independent measurement of the same drop (see google-docs/google-docs-skia-proxy-role-first-experience)."
}
}
"knownWrong": {}
]

}
6 changes: 3 additions & 3 deletions tests/fixtures/pdfs/unknown/pdflib-leading-glyph-skills-header.expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
"phoneIsValid",
"skills"
],
"skillsCount": 8,
"skillsCount": 9,
"experienceCount": 1,
"educationCount": 1,
"projectsCount": 0,
Expand Down Expand Up @@ -74,7 +74,7 @@
"sectionSource": "markdown",
"pageCount": 1,
"rawCharCount": 462,
"extractedCharCount": 289,
"extractedCharCount": 290,
"sections": [
{
"name": "profile",
Expand All @@ -101,7 +101,7 @@
"hasSummary": false,
"experienceCount": 1,
"educationCount": 1,
"skillsCount": 8
"skillsCount": 9
},
"linkAnnotationCount": 0,
"disagreements": []
Expand Down