Flaw in Job Scraper Software

[Madacool01]

Hello! Been making a job scraper and job match score calculator using Cosine Similarity recently for Fathers Day.
I hit a major flaw with my code. My skill extractor that is used for calculating the similarity between skills in a Resume and Job description uses a fixed keywords list for skill extraction.
What if a job description uses a synonym for a skill they are looking for, and the script ignores it entirely? Furthermore, lets say someone adds a new skill to their resume, and the keyword is still not in the list, so it gets ignored entirely. After some research, I found out that the problem I am facing is a Out-Of-Vocabulary (OOV) Problem and the Synonymity Problem.
This is my current code:

@Language.factory("custom_entity_ruler")
def create_custom_ruler(nlp, name):
    technical_keywords = [
        "Forscher", " Hochgeschwindigkeitskameras", "PIV-Systemen", "Maschinenbau", "Maschinenbauingenieur",
        "Wasserkraftwerke", "Analysieren", "Mechanische", "Konstruktion", "Wasserturbine", "Generator",
        "Ventil", "Reparatur", "PFD", "P&ID", "Software", "Softwareentwicklung", "Fluiddynamik", "Strukturdesign",
        "Hydraulikk"
    ]

    ruler = EntityRuler(nlp)
    patterns = [{"label": "TECH_SKILL", "pattern": [{"LOWER": s.lower()}]} for s in technical_keywords]
    ruler.add_patterns(patterns)
    return ruler

def extract_technical_skills(resume_text):
    nlp = spacy.load("en_core_web_sm")

    nlp.add_pipe("custom_entity_ruler", before="ner")

    clean_text = re.sub(r'\b(Projects|Experience|Education|Summary|Skills|Awards):', '', resume_text, flags=re.IGNORECASE)

    doc= nlp(clean_text)
    return sorted(list(set(ent.text for ent in doc.ents if ent.label_ == "TECH_SKILL")))

.

I am still actively searching for a solution.

[Madacool01]

Alright update! I think I have managed to implement a tool to fix the issue above. Firstly, instead of a small language processing model, I switched to a Medium one. The main difference between them is a concept called Word Embeddings. This is where the numerical (mathematical) representation of words with similar meaning are mapped to neighbouring points in $\mathbb{R}^{n}$ space. This makes it so that "Programming" and "Software Development" are mapped close to each other in the math world.
Secondly, since I am in Austria, the German language makes our work a bit easier, as Nouns and Proper Nouns are usually important and technical words. Lastly, I also used the ".lemma"_ function to turn every plural word into its root.
Here is the implementation:

import spacy
import re
from matcher import extract_text_from_pdf
import os

#german medium model that constains Word Vectors for synonyms!!
nlp = spacy.load("de_core_news_md")

def extract_dynamic_skills(text):
    clean_text = re.sub(r'\b(Projects|Experience|Education|Summary|Skills|Awards|Erfahrung|Ausbildung|Fähigkeiten):', '', text,
                        flags=re.IGNORECASE)
    
    doc = nlp(clean_text)
    extracted_features = set() #set for ignoring duplicates

    #we want only Nouns / Proper nouns
    # we ignore common words (und, order, der, ...)
    # we ignore tiny words since they are most likely not skills
    for token in doc:
        if token.pos_ in ["NOUN", "PROPN"] and not token.is_stop and len(token.text) > 2:
            extracted_features.add(token.lemma_.lower()) # .lemma_ e.g: "Wasserkraftwerke" becomes "Wasserkraftwerk" this prevents singular/plural mismatches!

    
    return list(extracted_features)

[Madacool01]

Well... . After getting some feedback and some experimentation, I found out that the job score matcher is essentially useless in the practical world. The resumes and descriptions are too different and unpredictable, and the perfect job-resume match can you give you 30% score?! Therefore, I will have to "Kill my Darlings", as they call it in SD, and remove this feature which I thought was a game-changer.
Regardless, I still learned something new about Word Embeddings, Cosine Similarity, and NLP! I'd call that a victory 🧐