oncept Annotation Tool Trainer
+
+[](https://github.com/CogStack/cogstack-nlp/actions/workflows/medcat-trainer_qa.yml?query=branch%3Amain)
+[](https://github.com/CogStack/cogstack-nlp/actions/workflows/medcat-trainer_release.yml)
+[](https://readthedocs.org/projects/cogstack-nlp-medcat-trainer/badge/?version=latest)
+[](https://github.com/CogStack/cogstack-nlp/releases/latest)
+
+MedCATTrainer is an interface for building, improving and customising a given Named Entity Recognition
+and Linking (NER+L) model (MedCAT) for biomedical domain text.
+
+MedCATTrainer was presented at EMNLP/IJCNLP 2019 :tada:
+[here](https://www.aclweb.org/anthology/D19-3024.pdf)
+
+# Documentation and Discussion
+
+Official docs available [here](https://docs.cogstack.org/projects/medcat-trainer)
+
+If you have any questions why not reach out to the community [discourse forum here](https://discourse.cogstack.org/)
diff --git a/v1/medcat-trainer/client/README.md b/v1/medcat-trainer/client/README.md
new file mode 100644
index 00000000..d5d13132
--- /dev/null
+++ b/v1/medcat-trainer/client/README.md
@@ -0,0 +1,88 @@
+
+---
+
+# MedCATtrainer Client
+
+A Python client for interacting with a MedCATTrainer web application instance. This package allows you to manage datasets, concept databases, vocabularies, model packs, users, projects, and more via Python code or the command line.
+
+## Features
+
+- Manage datasets, concept databases, vocabularies, and model packs
+- Create and manage users and projects
+- Retrieve and upload project annotations
+- Command-line interface (CLI) for automation
+
+## Installation
+
+```sh
+pip install mctclient
+```
+
+Or, if installing from source:
+
+```sh
+cd client
+python -m build
+pip install dist/*.whl
+```
+
+## Python Usage
+
+```sh
+export MCTRAINER_USERNAME=+ username: admin + password: admin ++ +2\. We strongly recommend creating a new admin user before using the trainer in 'production' and storing sensitive +clinical documents on the trainer. To add a new user navigate to select `http://localhost:8001/admin/` and select 'Users'. + + + +3\. Select 'Add User' and complete the form with a new username / password. + + + +4\. Once created, select the new user, and tick the 'Staff Status' or 'Superuser Status' to allow the user to +access the admin app. + +5\. Remove the default admin user by navigating to step 2, select the user and the action + + diff --git a/v1/medcat-trainer/docs/advanced_usage.md b/v1/medcat-trainer/docs/advanced_usage.md new file mode 100644 index 00000000..60bb35b3 --- /dev/null +++ b/v1/medcat-trainer/docs/advanced_usage.md @@ -0,0 +1,3 @@ +# Advanced Usage + +- ReST API Usage for bulk dataset / project creation: available in: docs/API_Examples.ipynb \ No newline at end of file diff --git a/v1/medcat-trainer/docs/annotator_guide.md b/v1/medcat-trainer/docs/annotator_guide.md new file mode 100644 index 00000000..981b3e1e --- /dev/null +++ b/v1/medcat-trainer/docs/annotator_guide.md @@ -0,0 +1,72 @@ +# Annotation Interface + +The annotation interface can be split into 5 sections. + + + +## Section 1 - Document Summary List +A list of documents to be completed in this project. Currently selected documents are highlighted in blue +left border. Submitted documents are marked with a . + +## Section 2 - Clinical Text +The selected documents text, highlighted with each concept recognised by the configured MedCAT model. +Highlighted spans of text indicate status of the annotation: +- Grey: A User has *not reviewed* this span that has been recognised and linked by MedCAT to a CDB concept. +- Blue: A User has reviewed the span and marked it as ***correct*** in terms of its linked MedCAT concept. +- Red: A User has reviewed the span and marked it as **incorrect** in terms of its linked MedCAT concept. +- Dark Red: A User has reviewed the span and marked it to **terminate**, meaning the text span should never again + link to this text span, this informs MedCAT that +- Turqoise: A User has reviewed the span and marked it as an **alternative** linked concept. The user has used the + 'Concept Picker' to choose the correct concept that should be linked. + +### Additional Annotations +MedCAT may miss text spans that are acronyms, abbreviations or misspellings of concepts. Missing annotations can be +added to the text by directly highlighting the text span, right clicking, selecting 'Add Term', searching for +concept (via ID, or name), and selecting Add Term: + + ->  ->  + +Select: +- Add Term: to add this annotation to the text span and link the selected concept +- Cancel: (Shortcut esc): to cancel adding the annotation to the text. + + + +## Section 3 - Action Bar + +### Concept Navigation +Navigating between the list of concepts as they appear in the document: +- Action buttons, left and right +- Left and right arrow keys on keyboard +- Directly clicking on the concept within the text. + + +### Concept Status Buttons +A concept can be marked with only one status. Status is recorded but only sent to MedCAT for +training on **submit** of the document and if the projects configured with "Train Model On Submit" is ticked. + +### Submit Button +Submit is disabled until all concepts have been reviewed and marked with a status. Clicking submit will produce +a submission confirmation dialog with an annotation summary. Confirming submission will send all new annotations +to MedCATTrainer middle tier, and re-train the MedCAT model. The following document will be selected and annotated +by the newly trained MedCAT model + +## Section 4 - Header Toolbar +Lists the current name of the document under review and the number of remaining documents to annotate in this project +action buttons for: +- : Summary of current annotations. f A similar view is shown before confirmation of submission of the annotations +- : Help dialog, showing shortcuts for document & concept navigation, concept annotation and submission. +- : Reset document. If an annotation is incorrectly added, or incorrectly submitted resetting the document will + clear all previous annotations and their status. + +## Section 5 - Concept Summary +Lists the current selected concepts details. + +|Concept Detail| Description | +|--------------| ------------| +|Annotated Text| The text span linked to the concept| +|Name | The linked concept name from within the MedCAT CDB| +|Type ID | The higher level group of concepts that this concept sits under. This may be 'N/A' depending if your CDB has Type IDs or not.| +|Concept ID | The unique identifier for this linked concept from the MedCAT CDB.| +|Accuracy | The MedCAT found accuracy of the linked concept for this span. Text spans will have an accuracy 1.0, if they are uniquely identified by that name in the CDB| +|Description | The MedCAT associated description of the concept. SNOMED-CT does not provide descriptions of concepts, only alternative names whereas UMLS does provide descriptions| \ No newline at end of file diff --git a/v1/medcat-trainer/docs/client.md b/v1/medcat-trainer/docs/client.md new file mode 100644 index 00000000..d5d13132 --- /dev/null +++ b/v1/medcat-trainer/docs/client.md @@ -0,0 +1,88 @@ + +--- + +# MedCATtrainer Client + +A Python client for interacting with a MedCATTrainer web application instance. This package allows you to manage datasets, concept databases, vocabularies, model packs, users, projects, and more via Python code or the command line. + +## Features + +- Manage datasets, concept databases, vocabularies, and model packs +- Create and manage users and projects +- Retrieve and upload project annotations +- Command-line interface (CLI) for automation + +## Installation + +```sh +pip install mctclient +``` + +Or, if installing from source: + +```sh +cd client +python -m build +pip install dist/*.whl +``` + +## Python Usage + +```sh +export MCTRAINER_USERNAME=
oncept Annotation Tool Trainer
+
+MedCATTrainer is an interface for building, improving and customising a given Named Entity Recognition
+and Linking (NER+L) model (MedCAT) for biomedical domain text.
+
+MedCATTrainer was presented at EMNLP/IJCNLP 2019.
+
diff --git a/v1/medcat-trainer/docs/maintenance.md b/v1/medcat-trainer/docs/maintenance.md
new file mode 100644
index 00000000..b21af39b
--- /dev/null
+++ b/v1/medcat-trainer/docs/maintenance.md
@@ -0,0 +1,61 @@
+# Maintanence
+
+MedCATtrainer is actively maintained. To ensure you receive the latest
+security patches of the software and its dependencies you should regularly
+be upgrading to the latest release.
+
+The latest stable releases update the `docker-compose.yml` and `docker-compose-prod.yml` files.
+
+To update these docker compose files, either copy them directly from the [repo](https://github.com/CogStack/cogstack-nlp/tree/main/medcat-trainer)
+or update the cloned files via:
+
+```shell
+$ cd MedCATtrainer
+$ git pull
+$ docker-compose up
+# alternatively for prod releases use:
+$ docker-compose -f docker-compose-prod.yml up
+```
+
+MedCATtrainer follows [Semver](https://semver.org/), so patch and minor release should always be backwards compatible,
+whereas major releases, e.g. v1.x vs 2.x versions signify breaking changes.
+
+Neccessary Django DB migrations will automatically applied between releases, which should largely be invisible to an end admin
+or annotation user. Nevertheless, migrating ORM / DB models, then rolling back a release can cause issues if values are defaulted
+or removed from a later version.
+
+## Backup and Restore
+
+### Backup
+Before updating to a new release, a backup will be created in the `DB_BACKUP_DIR`, as configured in `envs/env`.
+A further crontab runs the same backup script at 10pm every night. This does not cause any downtime and will look like
+this in the logs:
+```shell
+medcattrainer-medcattrainer-db-backup-1 | Found backup dir location: /home/api/db-backup and DB_PATH: /home/api/db/db.sqlite3
+medcattrainer-medcattrainer-db-backup-1 | Backed up existing DB to /home/api/db-backup/db-backup-2023-09-26__23-26-01.sqlite3
+medcattrainer-medcattrainer-db-backup-1 | To restore this backup use $ ./restore.sh /home/api/db-backup/db-backup-2023-09-26__23-26-01.sqlite3
+```
+
+A backup is also automatically performed each time the service starts, and any migrations are performed, in the events of a new release
+introducing a breaking change and corrupting a DB.
+
+### Restore
+If a DB is corrupted or needs to be restored to an existing backed up db use the following commands, whilst the service is running:
+
+```shell
+$ docker ps
+CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
+a2489b0c681b cogstacksystems/medcat-trainer-nginx:v2.11.2 "/docker-entrypoint.…" 4 days ago Up 4 days 80/tcp, 0.0.0.0:8001->8000/tcp, :::8001->8000/tcp medcattrainer-nginx-1
+20fed153d798 solr:8 "docker-entrypoint.s…" 4 days ago Up 4 days 0.0.0.0:8983->8983/tcp, :::8983->8983/tcp mct_solr
+2b250a0975fe cogstacksystems/medcat-trainer:v2.11.2 "/home/run.sh" 4 days ago Up 4 days medcattrainer-medcattrainer-1
+$ docker exec -it 2b250a0975fe bash
+root@2b250a0975fe:/home/api# cd ..
+$ restore_db.sh db-backup-2023-09-25__23-21-39.sqlite3 # run the restore.sh script
+Found backup dir location: /home/api/db-backup, found db path: home/api/db/db.sqlite3
+DB file to restore: db-backup-2023-09-25__23-21-39.sqlite3
+Found db-backup-2023-09-25__23-21-39.sqlite3 - y to confirm backup: y # you'll need tp confirm this is the correct file to restore.
+Restored db-backup-2023-09-25__23-21-39.sqlite3 to /home/db/db.sqlite3
+```
+
+The `restore_db.sh` script will automatically restore the latest db file, if no file is specified.
+
diff --git a/v1/medcat-trainer/docs/make.bat b/v1/medcat-trainer/docs/make.bat
new file mode 100644
index 00000000..8084272b
--- /dev/null
+++ b/v1/medcat-trainer/docs/make.bat
@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+ set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=.
+set BUILDDIR=_build
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+ echo.
+ echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+ echo.installed, then set the SPHINXBUILD environment variable to point
+ echo.to the full path of the 'sphinx-build' executable. Alternatively you
+ echo.may add the Sphinx directory to PATH.
+ echo.
+ echo.If you don't have Sphinx installed, grab it from
+ echo.https://www.sphinx-doc.org/
+ exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd
diff --git a/v1/medcat-trainer/docs/meta_annotations.md b/v1/medcat-trainer/docs/meta_annotations.md
new file mode 100644
index 00000000..bafcd8a1
--- /dev/null
+++ b/v1/medcat-trainer/docs/meta_annotations.md
@@ -0,0 +1,42 @@
+# Meta Annotations
+
+MedCAT is also able to learn project & context specific annotations that overlay on top of the base layer of concept annotations.
+
+Example use cases of these annotations could be to train models to predict if:
+
+- all disease concepts were **experienced** by the patient, a relative, or N/A.
+- all symptom concepts are **temporally** reference present day, or are historical.
+- all drug concepts are mentions of patients consuming drugs rather than **hypothetical** mentions.
+- a complaint for a patient is **primary** or **secondary**.
+
+MedCATTrainer is configurable (via the administrator app), to allow for the collection of these meta annotations. We
+currently have not integrated the active learning components of the concept recognition.
+
+## Meta Annotation Configuration
+
+To create a new Meta Annotation Task and attach to an existing project:
+
+1\. Enter your project configuration settings via the admin page (http://localhost:8001/admin/)
+
+
+
+2\. At the bottom of the form, select the + icon to bring up the new Meta Annotation Task Form.
+
+
+
+3\. Complete the form and add additional meta task values if required for your task via the '+' icon and the 'values' input.
+Values are enumerated options for your specific task. These can be re-used across projects or be project specific.
+Ensure the default is one of the corresponding values available. Descriptions appear alongside the tasks in interface
+and in full in the help dialog.
+
+
+
+4\. Select desired Meta Annotation tasks for the project by holding down (ctrl / cmd) and clicking the meta tasks,
+then select 'Save' to save the project changes.
+
+
+
+5\. Meta Annotations now appear in the interface for that project under the concept summary. Meta-annotations
+only appear for concepts that are correct.
+
+
\ No newline at end of file
diff --git a/v1/medcat-trainer/docs/project_admin.md b/v1/medcat-trainer/docs/project_admin.md
new file mode 100644
index 00000000..19f1e30a
--- /dev/null
+++ b/v1/medcat-trainer/docs/project_admin.md
@@ -0,0 +1,205 @@
+# Annotation Project Creation
+
+## Creating an Annotation Project
+Annotation projects are used to inspect, validate and improve concepts recognised & linked by MedCAT.
+They can also be used collect annotations for defined MetaCAT models tasks, and coming soon RelCAT, or relation annotation models.
+
+Using the admin page, a configured admin or superuser can create, edit and delete annotation projects.
+
+1\. Navigate to `http://localhost:8001/admin/` or the `http://![](\"./../docs/_static/img/admin_page.png\")
\n",
+ "| \n", + " | doc_name | \n", + "anno_value | \n", + "cui | \n", + "
|---|---|---|---|
| 0 | \n", + "Psych Text 1 | \n", + "psychopathology | \n", + "C0004936 | \n", + "
| 1 | \n", + "Psych Text 1 | \n", + "constipated | \n", + "C0009806 | \n", + "
| 2 | \n", + "Psych Text 1 | \n", + "depression | \n", + "C0011570 | \n", + "
| 3 | \n", + "Psych Text 1 | \n", + "depression | \n", + "C0011570 | \n", + "
| 4 | \n", + "Psych Text 1 | \n", + "fatigued | \n", + "C0015672 | \n", + "
| Name * | ++ |
| Concept ID * | ++ |
| Description | ++ |
| Type | ++ |
| Type IDs | ++ |
| Synonyms | ++ |
| Annotated Text | +Concept ID | +Concept Name | +ICD-10 | +OPCS-4 | +{{task.name}} | +
|---|---|---|---|---|---|
| + {{leftContext(concept)}} + {{concept.value}} + {{rightContext(concept)}} + | +{{concept.cui}} | +{{concept.pretty_name}} | +
+ {{`${code.code} | ${code.desc}`}}
+ |
+ + | ++ {{taskMaps[task.id][task.value]}} + | +
Confirm clearing cached MedCAT Model Project {{clearModelModal}} (and any other Projects that use the same model).
++ This will remove any interim training done (if any). + To recover the cached model, re-open the project(s), and re-submit all documents. + If you're unsure you should not clear the model state. +
+ + + + + +False positive model predictions can be the result of:
+Clicking on these annotations will not highlight this annotation as it doesn't exist in the + dataset
+False negative model predictions can be the result of:
+True positive model predictions are annotations that were correctly predicted by the model.
+| Doc Name | +CUI | +Value | +Accuracy | +Text | +
|---|
{{info.description}}
+| Value | +Description | + + +
|---|---|
| {{value}} | +{{valueHelp}} | +
{{selectedProjectGroup.description}}
+ + +| Shortcut Key | +Key Name | +Description | +
|---|---|---|
|
+ |
+ Up Arrow | +Previous Document | +
|
+ |
+ Down Arrow | +Next Document | +
|
+ |
+ Left Arrow | +Previous Concept | +
|
+ |
+ Right Arrow or Space bar | +Next Concept | +
|
+ |
+ Enter Key | +Submit / Submit Confirm (on submit summary) | +
| + | 1 Key | +Correct | +
| + | 2 Key | +Incorrect | +
| + | 3 Key (if set on your project) | +Terminate | +
| + | 4 Key | +Alternative | +
| + | 5 Key (if set on your project) | +Irrelevant | +
{{ errors.description }}
+Full Error:
+{{ errors.stacktrace }}
+Exit this window to review annotations or return to the project selection screen
+ + + + + + +Confirm reset of document annotations
+This will clear all current annotations in this document
+