Idea: verify_pdf.py --contains can't see LaTeX's typographic substitutions

[9scorp4]

Idea: verify_pdf.py --contains can't see LaTeX's typographic substitutions

TL;DR: normalize_text() folds whitespace only, so --contains compares what a user
types against what LaTeX renders. --contains "Master's degree" fails on a CV that
plainly contains it, and --contains "2016-2024" fails on the en-dash the template itself
produces — the exact failure mode 05-cv-templates.md already documents as a confirmed
Workday import failure. Reproduces on lualatex, stock template, current master.

Asking before building, per CONTRIBUTING's "an Idea thread costs nothing".

Reproduction

cv/main_example.tex at 79cd383, placeholders filled with ordinary CV content:

Master's degree in Statistics and a Bachelor's degree in Physics.
Led the company's analytics team and owned the client's reporting stack.
Six Sigma Green Belt, 2016--2024.

Compiled with the documented command (lualatex, 2 pages, clean), then:

$ python3 tools/verify_pdf.py cv/atskw.pdf --contains "Master's degree"     FAIL
$ python3 tools/verify_pdf.py cv/atskw.pdf --contains "Bachelor's degree"   FAIL
$ python3 tools/verify_pdf.py cv/atskw.pdf --contains "company's"           FAIL
$ python3 tools/verify_pdf.py cv/atskw.pdf --contains "2016-2024"           FAIL
$ python3 tools/verify_pdf.py cv/atskw.pdf --contains "Six Sigma"           PASS
$ python3 tools/verify_pdf.py cv/atskw.pdf --contains "Statistics"          PASS

The text layer is correct; the comparison isn't:

'Master’s degree'   U+004D ... U+0072 U+2019 U+0073 ...   (' became U+2019)
'2016–2024'         U+0032 U+0030 U+0031 U+0036 U+2013    (-- became U+2013)

Both substitutions are LaTeX behaving correctly. normalize_text() is
" ".join(text.split()), so nothing folds them back before the in test.

Why this is worth more than it looks

1. It produces a false negative on exactly the check the docs steer users toward.
The ATS Parseability section says to verify "posting keywords covered or honestly absent"
and warns never to stuff. A user checking Master's degree — one of the most common ATS
keywords there is — gets FAIL on a CV that contains it, and the documented remedy for a
missing keyword is to add it. The tool nudges toward the one thing the docs forbid.

2. The tool can't detect the failure the docs already confirmed.
05-cv-templates.md (§ "Date fields must be ASCII ranges") documents 2016--2024
reaching the text layer as 2016<U+2013>2024, a real Workday import that dropped an end
date and every education entry, and instructs: "after extracting the text layer, confirm
every experience entry shows a start and an end separated by an ASCII hyphen."

There is no tooling for that. The natural invocation, --contains "2016-2024", returns
FAIL — but it returns FAIL identically when the dates are simply absent. The result can't
distinguish "present but en-dashed" (the documented bug) from "missing". The check the
docs call for is currently un-automatable with the tool that exists to automate it.

3. CI passes today only by accident of vocabulary.
.github/workflows/ci.yml uses --contains for the LaTeX smoke test with
'Professional Experience', 'Achievement', 'Dear [Hiring Manager / Team]' — no
apostrophes, no dashes. Add one possessive to a placeholder and the smoke test breaks for
a reason no one would guess from the diff.

The design tension, which is the actual question

Folding is not uniformly correct, which is why I'm asking rather than sending a patch:

• For keyword coverage, folding is right. Real ATS keyword matching is fuzzy about
  curly quotes, and the question being asked is "does this term appear".
• For the date-format rule, folding is exactly wrong. If --contains "2016-2024"
  starts passing against an en-dash, the tool now hides the Workday failure instead of
  merely failing to detect it. That is worse than today.

So one flag can't serve both, and the useful version is probably two behaviours.

What I'd build, if wanted

1. Fold typographic punctuation in normalize_text() before comparison — U+2018/U+2019
   to ', U+201C/U+201D to ", U+2013/U+2014 to -, U+00A0 to space, plus
   unicodedata.normalize("NFC", ...). Applies to --contains, the coverage use case.
2. Add a distinct check for the date rule that deliberately does not fold — e.g.
   --ascii-dates, asserting every \d{4}[^\d]\d{4} run in the text layer is separated by
   U+002D. This is the one that makes the documented Workday guidance executable.
3. Tests covering both, plus a CI case with a possessive in it so the vocabulary accident
   above can't recur.

Roughly one function and one new flag. Happy to split into two PRs, since (1) is a fix
and (2) is a feature and CONTRIBUTING is clear about one concern per PR.

Alternatives I'd equally accept: fold in normalize_text() and do nothing else,
leaving the date rule manual; or document the gotcha in the ATS section and change no code.

Smaller, separate observation

Unrelated mechanism, mentioning it only so it isn't a second thread: under pdflatex
without \usepackage[T1]{fontenc}, the text layer stores accents decomposed (NFD), so
--contains "Genève" fails there too. Measured: 0/5 accented keywords matched without T1,
5/5 with it; both extractor backends agree. This does not affect lualatex either way,
so it only concerns users whose lualatex is broken and who fall back — which is how I found
all of this. A one-line fontenc addition would cover it, but it's off the documented path
and I'd drop it if you'd rather not carry defensive lines for unsupported engines.

Environment: TeX Live 2023 (Debian), LuaHBTeX 1.17.0, Poppler pdftotext 24.02.0,
pypdf 6.16.2, Linux, upstream 79cd383.

[MadsLorentzen]

Confirmed - normalize_text() is whitespace-only (tools/verify_pdf.py:50), and LaTeX's curly-apostrophe (U+2019) and en-dash substitutions are real; your repro table matches what the tool does today. The fix direction I'd take: fold both sides in normalize_text() - NFC-normalize (which also resolves #384's pdflatex case), then map U+2019 to ', en/em-dash to -, and any other substitutions the moderncv template actually produces, with the mapping table commented as such. Keep it a comparison-time fold; never rewrite the extracted dump itself, since the ATS parser sees the raw layer and the dump should stay honest. Happy to see this and #384 arrive as one scoped PR - invitation is yours.