cui2icd10 question + some setup feedback

[rahul-evenup-ai]

Hi! Thanks for building this great tool, had some issues with the setup, but here's what I have in terms of a rqs.txt (incase this is helpful).

blis==0.7.4 catalogue==1.0.0 certifi==2021.5.30 chardet==4.0.0 click==7.1.2 cymem==2.0.5 datasets==1.6.0 dill==0.3.3 elasticsearch==7.10.0 filelock==3.0.12 Flask==1.1.0 fsspec==2021.6.0 gensim==3.8.0 huggingface-hub==0.0.10 idna==2.10 importlib-metadata==4.5.0 itsdangerous==2.0.1 Jinja2==3.0.1 joblib==1.0.1 MarkupSafe==2.0.1 multiprocess==0.70.11.1 murmurhash==1.0.5 numpy==1.20.0 packaging==20.9 pandas==1.2.4 pathy==0.5.2 plac==1.1.3 preshed==3.0.5 pyarrow==4.0.1 pydantic==1.7.4 pyparsing==2.4.7 python-dateutil==2.8.1 pytz==2021.1 regex==2021.4.4 requests==2.25.1 sacremoses==0.0.45 scikit-learn==0.24.0 scipy==1.6.3 six==1.16.0 smart-open==3.0.0 spacy==2.3.5 spacy-legacy==3.0.5 srsly==1.0.5 thinc==7.4.5 threadpoolctl==2.1.0 tokenizers==0.10.3 torch==1.8.1 tqdm==4.49.0 transformers==4.5.1 typer==0.3.2 typing-extensions==3.10.0.0 urllib3==1.26.5 wasabi==0.8.2 Werkzeug==2.0.1 xxhash==2.0.2 zipp==3.4.1

Basically I went through your setup.py manually, removing the the ~=, and making it ==. also for sklearn, I installed scikit-learn==0.24.0, not sure what sklearn~=0.0 does. Also the version of spacy is different, was seeing some errors in loading the spacy model, and followed this ticket to resolve them; I'm using spacy==2.3.5. More generally I'm using python3.7 as that's what your medium post was using and am on a mac, OS version 11.4 (Big Sur).

Also the config for your medmen trained CDB class uses the en_core_sci_lg model, not the en_core_sci_md model.

Anyways, the question I have is, in order for the CDB class to have the cui2icd10 key in addl_info filled out, do I need the UMLS license? Seems like that mapping is blank in the provided medmen trained model, and wanted to see if the model trained on NLM would have the icd codes filled out. I tried applying for the license, but am getting 500s on the sign up page right now, will check again later.

[rahul-evenup-ai]

oh also when using the get_entities() function, cui2snomed causes a KeyError; not that the other keys return anything right now, but cui2snomed isn't found here

[w-is-h]

Hi @rahul-evenup-ai, thank you for a detailed report. The issue with get_entites is resolved in the latest version. I've just created a new python env and run pip install --upgrade medcat downloaded the models and all went through without a problem. Do you mind testing that out and reporting the problems you encounter - I'll try to address them asap.

Re en_core_sci_lg you are right, the medmen cdb uses that model, I'll change it to en_core_sci_md, thanks.

[w-is-h]

For the cui2icd10 once you have a UMLS licence I'll share the big models that contain those fields also. We are not allowed to make that public, because of UMLS rules.

[rahul-evenup-ai]

yeah so the issue was actually the pip version, you need pip 21.1.1 or higher. Apart from the spacy_model loading, everything else worked! And got it, thanks for the quick reply! Also it works with python 3.8.10 as well.

[w-is-h]

Interesting (re pip), I'll make the clear in the README and close the issue - if that is fine with you.

[rahul-evenup-ai]

yup sure, thanks for the info