Text mining cancer biomarkers for the CIVIC database
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.



This is a text mining project to assist curation in the Clinical Interpretation of Variants in Cancer (CIViC) database. CIViC catalogs information on diagnostic, predictive, predisposing and prognostic variants in cancer. This project aims to text mine this data from PubMed and Pubmed Central Open Access subset

In brief: it finds sentences that describe genes, variants, cancer types and optionally drugs. We then annotated many of them to create training data. This is then used to build a classifier using the Kindred package. This is then applied to all of PubMed and PubMed Central Open Access to extract sentences and structured information. This is then collated for download and viewing through a web viewer.

System Requirements

This is a Python3 project which has been tested on Centos 6/7 but should work on other Linux operating systems and MacOS. An individual process of this can be run on a laptop or desktop computer. But in order to process all of the literature (PubMed, etc), this should really be run on a cluster or server-like machine. A cluster that uses Slurm or the SunGrid engine (SGE) are supported. Each node needs only 4 GBs on RAM.

This project relies on text mining using Kindred and resource management with PubRunner. These can be installed through pip.

Installation Guide

You can clone this repo using Git or download the ZIP file of it.

git clone https://github.com/jakelever/civicmine.git

The dependencies can be installed with the command below. Remember to install the English language model for Spacy.

pip install kindred pubrunner
python -m spacy download en

Installation should take a maximum of 15 minutes (mostly due to the Spacy and language models installation).

Instructions for Use

A full run of CIViCmine will take some time as the large corpora need to be downloaded and parsing takes a long time. A cluster is strongly recommended for this task. PubRunner uses Snakemake for execution and can therefore be used on most clusters.

A test run (which is what the Travis-CI test does) can be executed with:

pubrunner --test .

The full run can be executed with:

pubrunner .


The project uses wordlists from the BioWordlists project for cancers, genes, drugs, variants and conflicting terms. The corpora used are PubMed abstracts and full-text papers from the PubMed Central Open Access Subset. While processing the full-text articles, subsection headers (e.g. Results) are also extracted to make it easier to locate where statements are made. The list of possible headers extracted can be found here. The actual extraction is managed by PubRunner.

We have annotated 1500 sentences that comprise the civicmine_corpus which are used for training a Kindred classifier.


The three output files of CIViCmine is outlined below. You likely want civicmine_collated.tsv if you just want the list of cancer biomarkers. If you want the supporting sentences, look at civicmine_sentences.tsv. You can use the matching_id column to connect the two files. If you want to dig further and are okay with a higher false positive rate, look at civicmine_unfiltered.tsv.

civicmine_collated.tsv: This contains the cancer biomarkers with citation counts supporting them. It contains the normalized cancer and gene names along with IDs for HUGO, Entrez Gene and the Disease Ontology.

civicmine_sentences.tsv: This contains the supporting sentences for the cancer biomarker in the collated file. Each row is a single supporting sentence for one cancer biomarker. This file contains information on the source publication (e.g. journal, publication date, etc), the actual sentence and the cancer biomarker extracted.

civicmine_unfiltered.tsv: This is the raw output of the applyModelsToSentences.py script across all of PubMed, Pubmed Central Open Access and PubMed Central Author Manuscript Collection. It contains every predicted relation with a prediction score above 0.5. So this may contain many false positives. Each row contain information on the publication (e.g. journal, publication date, etc) along with the sentence and the specific cancer biomarker extracted (with HUGO, Entrez Gene and Disease Ontology IDs). This file is further processed to create the other two.

Shiny App

The code in shiny/ is the Shiny code used for the web viewer. If it is helpful, please use the code for your own projects. The list of dependencies is found at the top of the app.R file.


The code to generate all the figures and text for the paper can be found in paper/. This may be useful for generating an up-to-date version of the plots for a newer version of CIViCmine.


A paper is in the works.


If you encounter any problems, please file an issue along with a detailed description.