Words after hyphens are removed by CDBMaker

[seowyian]

pls see repost below

[seowyian]

Sorry - reposting with escaped ~
New to MedCAT, please pardon ignorance. I was trying out the MedCAT2 basic tutorial on building a MedCAT Model, and following the code. I notice that CBDMaker removes hyphens and words immediately after hyphens, without spacing. Examples:
"Wagner-Unverricht syndrome" becomes "wagner~syndrome"
"Wernicke-Korsakoff syndrome" becomes "wernicke~syndrome"
but words that follow a hyphen after a space are retained eg "DM - Diabetes" becomes "dm~diabetes'

Is this intentional or a bug?

This is my script:

vocab = Vocab()
cnf = Config()
cdb = CDB(cnf)
maker = CDBMaker(cnf, cdb)

cui = '73211009'
namelist = ["diabetes",
"diabetes mellitus (disorder)",
"DM - Diabetes mellitus", # word separated from hyphen by space is retained.
"diabetes mellitus",
"dm"]
cui_df = pd.DataFrame({'cui': cui, 'name': namelist})
maker.prepare_csvs([cui_df])

cui2 = '396230008'
namelist2 = ['Dermatomyositis (disorder)',
'Dermatomyositis',
'DM - Dermatomyositis',
'Polymyositis with skin involvement',
'Wagner Unverricht syndrome'] # no hyphen
cui_df2 = pd.DataFrame({'cui': cui2, 'name': namelist2})
maker.prepare_csvs([cui_df2])

cui3 = '69482004'
namelist3 = ['Korsakoff's psychosis',
'Wernicke-Korsakoff syndrome',
'Korsakov syndrome - alcoholic']
cui_df3 = pd.DataFrame({'cui': cui3, 'name': namelist3})
maker.prepare_csvs([cui_df3])

print('CUI2Info: ', cdb.cui2info.keys())
print('Name2Info: ', cdb.name2info.keys())

Output:
CUI2Info: dict_keys(['73211009', '396230008', '69482004'])
Name2Info: dict_keys(['diabetes', 'diabetes~mellitus', 'dm~diabetes~mellitus', 'dm', 'dermatomyositis', 'dmdermatomyositis', 'polymyositis~with~skin~involvement', 'wagner~unverricht~syndrome', 'korsakoff~psychosis', 'wernicke~syndrome', 'korsakov~syndrome~alcoholic'])

[mart-r]

Hi.

Thank you for bringing this to our attention. This seems to be an issue with the default regex based tokenizer.
Assuming you've got spacy (or medcat[spacy]) installed, you should be able to get teh desired behaviour by changing the base tokenizer:

vocab = Vocab()
cnf = Config()
cnf.general.nlp.provider = 'spacy'  # NEW LINE OF CODE to change the base tokenizer
cdb = CDB(cnf)
maker = CDBMaker(cnf, cdb)
# your existing code

With that said, we will - of course - investigate and fix the issue with the regex tokenizer as well.

EDIT:
If you don't have spacy installed you can install an appropriate version with:

python -m pip install "medcat[spacy]>=2.0.0"

[seowyian]

Tks for looking at it. I'll close this ticket. Rgds.