Beyond CRUD Flows

Beyond-CRUD Flows

PULSE is more than a CRUD app (create / read / update / delete forms). Analysts need the system to recognize when two messy records are really the same organization or person, even when the spelling, language or formatting differs. That recognition work is Entity Resolution (#4).

The first building block is identifier normalization (#26): before we match anything, we clean names, websites, LinkedIn links, emails and phone numbers into one standard form. The original text from the source is kept; the cleaned value sits beside it for comparison.

Why this matters (business)

Quebec ecosystem data mixes French and English. The same company might appear as:

• Groupe Exemple ltée
• GROUPE EXEMPLE
• Groupe Exemple Inc.
• website https://www.boutique.exemple.qc.ca/a-propos

Without cleaning, the dashboard would show three “different” organizations and undercount activity. After normalization, those name variants all become the comparison key groupe exemple, and the website becomes exemple.qc.ca. Downstream matching (#61) can then link them with confidence.

This is also our main beyond-CRUD evidence for the course code rubric: real domain rules and libraries, not just storing and retrieving rows.

What we built (technical, in plain terms)

Piece	Role
normalization.py	One clean-up function per identifier type (name, domain, LinkedIn, email, phone)
legal_suffixes.toml	Editable list of company endings to strip (inc, ltée, s.e.n.c., …) so we do not hard-code them in Python
test_normalization.py	Unit tests for normal cases, empty input, accents and spelling variants

The module is pure logic: it takes strings in and returns cleaned keys (or “unusable”). It does not talk to the database or the internet. Other pipeline steps (for example mapping Omerlo payloads in #57) call it after they pull fields out of a source record.

Cleaning rules (what each field becomes)

Field	Business intent	What the code does
Organization name	Treat French/English legal endings and punctuation as noise	Lowercase, remove accents for comparison, collapse & / et / and, strip trailing legal forms from the config file
Person name	Keep hyphenated Quebec names consistent	Lowercase, remove accents; treat hyphens like spaces so Marie-Ève and Marie Eve match
Website / domain	Compare the real company site, not a deep link	Drop https, www, path and query; keep the registered domain (e.g. exemple.qc.ca)
LinkedIn	One key per company or person profile	Reduce to company/<slug> or in/<slug>, ignore tracking parameters
Email	Stable address + optional domain signal	Lowercase and trim; also expose the domain part for weaker matching
Phone	One international form for Canadian numbers	Convert to E.164 (default region Canada). Bad numbers return “empty / unparseable / impossible” warnings instead of crashing

Libraries we are allowed to use

Chosen on Infrastructure and Tools (#15):

Library	Why (business)	How (technical)
phonenumbers	Phone formats vary a lot; we need one standard	Formats to E.164 with Canada as the default region
tldextract	.qc.ca and similar suffixes are easy to get wrong by hand	Uses the public suffix list; offline only (no download at runtime)
unicodedata (built into Python)	Accents should not block a match	Folds Québec → quebec without an extra dependency

How this fits the pipeline

Source data (Omerlo, etc.)
    → extract fields
    → normalize identifiers  ← this page / #26
    → match same entities    ← #61 (later)
    → scores, changes, dashboard

Normalization alone does not decide “these two rows are the same company.” It only makes that decision possible and reliable.