Lesion annotation network (LesaNet)
This project contains the code and labels of the CVPR 2019 oral paper: “Holistic and Comprehensive Annotation of Clinically Significant Findings on Diverse CT Images: Learning from Radiology Reports and Label Ontology” (arXiv).
LesaNet  predicts the body part, type, and attributes of a variety of lesions in CT images. It is a multi-label classification CNN. It learns to annotate lesion images by leveraging radiology reports and the relations between labels (ontology).
You can use LesaNet to:
- Given a lesion image patch, predict the lesion's body part, type, and attributes;
- Given a label (e.g., kidney), find all lesions with that label in your database;
- Given a lesion image patch, find similar lesions in your database (content-based image retrieval).
- PyTorch 0.4.1
- Python 2.7
- The DeepLesion dataset [2,3,4]
- The roi_pooling layer, which can be compiled from faster-rcnn.pytorch and put in
requirements.txtcan be used to build a virtual environment for LesaNet.
Usage on DeepLesion
- You can train your own model or download the model in  from here and put it in
./run.shfor both training and inference.
default.ymlfor configurations of training and inference.
config.ymlfor hyper-parameters of the algorithm.
- With a trained model, you can set
default.yml. A Matlab
.matfile will be saved in
results/which contains the prediction results of the test set. You can also set
generate_features_all=True, which will produce a
.matfile containing the embeddings of all lesions.
- After the
.matfiles are generated, you can run
visualize_results.pyto generate a
.htmlfile to visualize the lesion classification and retrieval results.
- This project is designed for the DeepLesion dataset. You can modify
load_ct_img.pyto adapt to your own data.
- Requirement: a trained model, a nifti CT image (e.g.,
- Because we use lesion patches as the input, please specify the coordinates of the 2D bbox of the lesion in a text file
with the format of
slice_number, left, top, right, bottom, see
demo_coords.txtfor an example.
./run.sh, input the path of the image and text files. The predictions will be printed on the console with the image patch saved to
Lesion labels and ontology (
text_mined_labels_171_and_split.json: Labels and data split. It contains several variables:
term_listis the 171 labels used in ;
train/val/test_lesion_idxsare the lesion indices of the train/val/test sets used in , where the indices are based on
DL_info.csvin DeepLesion  starting from 0;
train/val/test_relevant/uncertain_labelsare the text-mined labels  from the reports of DeepLesion.
lesion_ontology_181022.xlsx: The ontology of the labels, including id (obsolete), class (bodypart/type/attribute), label name, synonyms, number of occurrence in DeepLesion, exclusive labels, parent labels, and child labels. The ontology is an adaptation of RadLex  v3.15 under the license. The 171 labels used in  is a subset of this ontology.
hand_labeled_test_set.json: 500 random lesions in the test set manually annotated by two radiologists in a more comprehensive fashion .
labels_for_demo.xlsx: The sizes, accuracies, and thresholds of the 171 labels on the hand-labeled test set.
Framework and sample lesion ontology, see :
- Because of the complexity of the free-text radiology reports and the limitation of the text-mining algorithm, the labels in
text_mined_labels_171_and_split.jsonmay contain noises.
- Radiologists typically do not describe every label of a lesion in the report, so the labels in
text_mined_labels_171_and_split.jsonmay also be incomplete.
- The lesion ontology can be further refined by professionals.
- LesaNet was trained on lesions in DeepLesion, so it may be inaccurate on nonlesions or lesions that are rare in DeepLesion.
- K. Yan, Y. Peng, V. Sandfort, M. Bagheri, Z. Lu, and R. M. Summers, “Holistic and Comprehensive Annotation of Clinically Significant Findings on Diverse CT Images: Learning from Radiology Reports and Label Ontology,” in CVPR, 2019. (arXiv)
- The DeepLesion dataset. (download)
- K. Yan, X. Wang, L. Lu, and R. M. Summers, “DeepLesion: Automated Mining of Large-Scale Lesion Annotations and Universal Lesion Detection with Deep Learning,” J. Med. Imaging, 2018. (paper)
- K. Yan et al., “Deep Lesion Graphs in the Wild: Relationship Learning and Organization of Significant Radiology Image Findings in a Diverse Large-scale Lesion Database,” in CVPR, 2018. (arXiv)
- http://www.radlex.org/; https://bioportal.bioontology.org/ontologies/RADLEX