How to use SNOMED trained model

[aylee2008]

Hi, thanks for your wonderful work. I am trying to use your trained model for entity linking on SNOMED-CT ontology.
However, I got some errors when I tried to run get_entities from a simple text.

This is my code:

cat = CAT.load_model_pack("mc_modelpack_snomed_int_16_mar_2022_25be3857ba34bdd5.zip")

#text = "My simple document with Huntington Chorea"
text = "My simple document with date of birth"
entities = cat.get_entities(text)
print(entities)

And this is the error message:

Traceback (most recent call last):
  File "/home/myname/repo/scribble.py", line 8, in <module>
    entities = cat.get_entities(text)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/myname/miniconda3/envs/myenv/lib/python3.11/site-packages/medcat/cat.py", line 1008, in get_entities
    doc = self(text)
          ^^^^^^^^^^
  File "/home/myname/miniconda3/envs/myenv/lib/python3.11/site-packages/medcat/cat.py", line 433, in __call__
    return self.pipe(text)  # type: ignore
           ^^^^^^^^^^^^^^^
  File "/home/myname/miniconda3/envs/myenv/lib/python3.11/site-packages/medcat/pipe.py", line 267, in __call__
    return self._nlp(text) if len(text) > 0 else None  # type: ignore
           ^^^^^^^^^^^^^^^
  File "/home/myname/miniconda3/envs/myenv/lib/python3.11/site-packages/spacy/language.py", line 1054, in __call__
    error_handler(name, proc, [doc], e)
  File "/home/myname/miniconda3/envs/myenv/lib/python3.11/site-packages/spacy/util.py", line 1722, in raise_error
    raise e
  File "/home/myname/miniconda3/envs/myenv/lib/python3.11/site-packages/spacy/language.py", line 1049, in __call__
    doc = proc(doc, **component_cfg.get(name, {}))  # type: ignore[call-arg]
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/myname/miniconda3/envs/myenv/lib/python3.11/site-packages/medcat/linking/context_based_linker.py", line 109, in __call__
    cui, context_similarity = self.context_model.disambiguate(cuis, entity, name, doc)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/myname/miniconda3/envs/myenv/lib/python3.11/site-packages/medcat/linking/vector_context_model.py", line 136, in disambiguate
    vectors = self.get_context_vectors(entity, doc)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/myname/miniconda3/envs/myenv/lib/python3.11/site-packages/medcat/linking/vector_context_model.py", line 71, in get_context_vectors
    values.extend([self.config.linking['weighted_average_function'](step) * self.vocab.vec(tkn.lower_)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/myname/miniconda3/envs/myenv/lib/python3.11/site-packages/medcat/linking/vector_context_model.py", line 71, in <listcomp>
    values.extend([self.config.linking['weighted_average_function'](step) * self.vocab.vec(tkn.lower_)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'builtin_function_or_method' object is not subscriptable

It ran fine when I used the same text in the tutorial. ("My simple document with Huntington Chorea")
I am currently using MedCat 1.10.2.
What could possibly be the problem?

[mart-r]

Thank you for letting us know of this issue! We've not encountered this before.
And after having looked at this in some detail, I can confirm that it is indeed a general issue not just something to do with your specific installation.

This has something to do with how the weighted average function is pickled within the model pack (or more specifically, the config).

The easiest workaround would be:

cat.config.linking.filters.weighted_average_function = lambda step: max(0.1, 1 - (step ** 2 * 0.0004))

This will ensure there's a valid method there to be called.

PS:
I'll be looking into the specific issue and a general resolution.
PPS:
Seems to work fine on python 3.10. So there's some issue with the differences between python 3.10 and python 3.11.

[aylee2008]

Works gracefully after downgrading to Python 3.10.
BTW I'll leave the issue open until a general resolution is available.

Thanks!

[mart-r]

Works gracefully after downgrading to Python 3.10. BTW I'll leave the issue open until a general resolution is available.

Thanks!

I'm glad downgrading to Python 3.10 worked for you (I expected it would).
However, overall this still remains an issue for us with python 3.11. Downgrading may not always be an option. And even if it is, it may not be desirable (e.g the speed improvements in 3.11). So we do not wish to have that as a general recommendation.

With that said, I've prepared a PR (#411) that should fix this on the fly. Though I'm still not fully sure what the underlying issue is. I've managed to identify that there is some issue with dumping functools.partial methods using dill with older python versions (3.10 and below) and subsequently loading them in later ones (3.11 and onwards). And the fix I've proposed really only addresses the default value. I don't expect many models to have vastly different weighted average functions set, but it would be better if we were able to solve it in a more general manner.

[mart-r]

Closing as #411 was merged in. Internal discussion suggests we're unlikely to come to a more robust solution with a reasonable time investment.