medcat/v2.10.1

🩺 MedCAT v2.10.1 Release Notes

This release focuses on improved model evaluation and diagnostics, component development tools, and performance estimation, alongside fixes to training, tokenisation, and statistics.

🚀 New Features & Enhancements

• Detailed Model Statistics – Expanded get_stats with additional evaluation metrics, including character-level Intersection over Union (IoU), golden character IoU, and Cohen's kappa. Statistics can now also be gathered more easily for NER-only or linker-only evaluation through simplified arguments.
• Failure Mode Analysis – Added diagnostics to help explain why expected concepts were missed during evaluation. Failure modes include missing CUIs, missing names, missing CUI-name associations, and disambiguation errors involving siblings, parents, children, or other concepts. New summary methods make it easier to understand common error patterns and identify areas for improvement.
• Component Contracts – Introduced contracts defining the expected inputs and outputs of different component types. These help developers validate custom component implementations in tests without adding runtime overhead to inference.
• Model Performance Estimation – Introduced a separate [medcat-performance-estimator](https://pypi.org/project/medcat-performance-estimator/) package for estimating relative concept-linking difficulty from a model and a list of concepts. Concepts are grouped into difficulty categories to provide a rough indication of which may need additional training. These estimates are indicative rather than reliable predictions of individual concept performance.
• LLM Components – Added new components for integrating large language model (LLM) functionality into MedCAT workflows as a separate plugin / package (see on PyPI and GitHub).

🐛 Bug Fixes

• NER Link Candidate Evaluation – Fixed an issue with NER link candidates that caused incomplete results for linker-only stats.
• Fine-tuning Reset – Fixed an issue with the optional training reset during fine-tuning when the dataset contains concepts absent from the model's CDB.
• Tokenisation and Annotation Text – Fixed an edge case where tokenisation could include extra punctuation, such as a trailing colon, in an entity span. Also corrected the text associated with PredicatedAnnotations.
• Statistics Accuracy – Fixed issues affecting NER-only statistics and the printing of evaluation results.
• Relation Extraction – Improved RelCAT handling in the MedCAT service.

🧰 Other Improvements

• Repository Structure – Renamed the medcat-v2 directory to medcat. Users relying on hard-coded repository paths may need to update them.
• Python Version Support – Dropped Python 3.10 support, as it has reached end of life.
• Build and Release Workflow – Fixed a typo in the release workflow.

---

This release improves evaluation depth and interpretability, provides better tools for validating custom components, and introduces an initial approach to estimating relative concept-linking performance to help guide model improvement.

What's Changed

• docs(medcat): Move away from v2 prefix by @mart-r in #607
• feat(medcat): Add component contracting by @mart-r in #582
• feat(medcat-llm-components): CPDT-108 Add llm components by @mart-r in #609
• feat(medcat): Get Stats: harder, better, (probably not) faster, stronger by @adam-sutton-1992 in #606
• Mcat service rel cat improvement by @vladd-bit in #619
• minor stats fixes to printing and NER only measurement by @adam-sutton-1992 in #620
• Examples fix by @adam-sutton-1992 in #624
• fix(medcat):CPDT-219 Fix cheating NER link candidates / names by @mart-r in #622
• fix(medcat) - use ner_ents for NER only statistics by @adam-sutton-1992 in #628
• feat(medcat): Simplify new stats methods / objects import by @mart-r in #630
• fix(medcat): Training reset during fine tune by @mart-r in #629
• feat(medcat-performance-estimation): Add MedCAT Performance estimation by @mart-r in #623
• feat(medcat): CPDT-223 Add failure modes to get stats by @mart-r in #627
  #642
• fix(medcat): fix tokenisation - and correct PredicatedAnnotations text by @adam-sutton-1992 in #639
• build(medcat):CPDT-291 Drop python 3.10 support (EoL) by @mart-r in #649
• medcat(build): Fix typo in release workflow by @mart-r in #654

Full Changelog: medcat/v2.9.0...medcat/v2.10.1