Merge pull request #68 from monarch-initiative/develop

Develop

.gitignore

@@ -22,7 +22,7 @@ loinc2hpo-core/src/main/java/org/monarchinitiative/loinc2hpo/util/TST.java
 
 
 
-loinc-hpoTermId4LoincTest-mapping.tsv
+loinc-hpoTerm4TestOutcome-mapping.tsv
 loinc2hpo-core/src/main/resources/hp.owl
 loinc2hpo-core/src/main/resources/data_UNC_annotated.csv
 loinc2hpo-core/src/main/resources/loinc_most_frequent.csv

CHANGELOG.md

@@ -95,7 +95,19 @@ Additional changes for this version
 
   "system" is renamed to "FHIR";
   Code for "presence" changed from "P" to "POS", code for "not presence" changed from "NP" to "NEG" to be consistent with FHIR
-
 
+* Show version in "About" message
+
+* Refactor to remove deprecated classes
+
+* Allow user to specify the path to hp OWL and Obo
+
+## v1.1.3
+
+* Add function to simulate patient data (patient and observation resources)
+
+* Add function to allow uploading simulated data to hapi-fhir server
+
+* Add function to allow downloading patient data from fhir server
 
 

docs/source/Configuration.rst

@@ -26,9 +26,12 @@ affect the operation of the app.
 
 * Set biocurator ID. From the menu bar, click **"Configuration"** - **"Set biocurator ID"**, specify your biocurator ID. If you are not assigned one, create one for yourself with the following format: organization name first followed by `:`, then your name/id.
 
-* Once you are done, click **"Edit"** - **"Show settings"** to view all your settings. The first two settings should **NOT** be null in order for the app to work correctly.
+* Once you are done, click **"Configuration"** - **"Show settings"** to view all your settings. The first two settings should **NOT** be null in order for the app to work correctly.
+
+* Use customized hpo. If you prefer to use your own versions of hpo, click **"Configuration"** - **"Change hpo.owl"** to set the path to your hpo.owl file; use the button below to set the path to the hpo.obo file. An important note: inconsistencies of hpo.owl and hpo.obo files will lead to errors, so make sure your hpo OWL and OBO files are serialized from the same HPO.
+
 
 Change Settings
 ---------------
-The settings will be saved to a local file so that you do not need to repeat the above steps every time you run the app. Should you want to change the settings, follow the above steps accordingly to overwrite the original settings.
+The settings will be saved to a local file so that you do not need to repeat the above steps every time you run the app. This applies even after you upgrade to a newer version of the app. Should you decide to change the settings, follow the above steps accordingly to overwrite the original settings.
 

docs/source/Contact.rst

@@ -1,10 +1,13 @@
 Contact
 =======
 
-**Peter Robinson**
+**Prof. Peter Robinson**
 
 Peter.Robinson@jax.org
 
+**Dr. Xingmin("Aaron") Zhang**
+Aaron.Zhang@jax.org
+
 The Jackson Laboratory for Genomic Medicine
 
 Farmington, CT

docs/source/Install & running.rst

@@ -8,6 +8,11 @@ To build and run the GUI, use the following command.::
   $ java -jar loinc2hpogui/target/Loinc2HpoGui-jar-with-dependencies.jar
 
 
+If you downloaded the jar releases directly, use the following command.::
+
+  //replace {$PATH} with the path to the jar file
+  $ java -jar {$PATH}/Loinc2HpoGui-jar-with-dependencies.jar
+
 To run the library code, enter the following ::
 
   $ java -jar loinc2hpo-core/target/Loinc2HpoLib.jar download

docs/source/Tutorial.rst

@@ -1,8 +1,6 @@
 Tutorial
 ========
 
-This tutorial is for loinc2hpo V1.1.0.
-
 Overview
 --------
 The app has three tabs.
@@ -131,3 +129,49 @@ To save data, you can click `File` - `Save Session`. This will save your annotat
 
 To export data, you can click `File` - `Export annotations as`. Currently the app only supports .tsv files.
 
+
+Converting Observations to HPO terms
+------------------------------------
+
+The third tab "Loinc2HpoConversion" is responsible for converting patient observations into corresponding HPO terms using the annotation map generated above. Here we will demonstrate a few different ways to do this, which is largely dependent on the source of patient observations.
+
+Convert locally stored observations to HPO terms
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Assuming that you have some json serialization of patient observations, you can click `Local` to select the files (multi selection is supported). The app will import the files into the left text region. Then you can click the `convert` button to get the HPO term.
+
+Download observations from FHIR server and convert to HPO terms
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In real world, patient information and their observations are probably saved in a hospital FHIR server. In order to get the observations for a patient, you have to retrieve the patient first using his or her identification. To do this, click the `Server` button, type in the base url for the FHIR server, and provide relevant information to search for the patient.
+
+To demonstrate the process, we will search for a patient named "Harry Metz" with a phone number "002-837-6481" and postal code "79442-0781" from a test FHIR server (hapi-fhir test server: http://fhirtest.uhn.ca):
+
+  .. image:: images/queryPatientFromServer.png
+     :align: center
+     :scale: 60 %
+
+After clicking `confirm`, the patient is found on the server and loaded into the left text region.
+
+  .. image:: images/downloadObservationsFromServer.png
+     :align: center
+     :scale: 30%
+
+Next, you can get all observations related to this patient by right clicking on the patient record and then clicking "download observations". The observations will be loaded to the left text region. Last, you can click `Convert` button to get a list of HPO terms, one for each observation, on the right text region.
+
+  .. image:: images/convertObservationsToHPO.png
+     :align: center
+     :scale: 30%
+
+Note that you may not be able to find such a patient if you try it yourself. This is because the hapi-fhir test server regularly purges their system so all data will be lost. We actually faked such a patient and uploaded to the test server right before we wrote the tutorial. Next, we are going to show you how we did this.
+
+
+Simulate patients and observations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For demonstration purposes, you can create simulated patients and their observations by clicking the `Simulate` button. You will see a popup window that asks you how many patients you want to simulate. Currently, we choose to create 10 observation for each patient, 5 ``Qn`` type, 4 ``Ord`` type ("presence"/"absence" outcome) and 1 ``Nom`` type. Check the "Upload" checkerbox and click `Confirm`. The app will generate fake patients and observations and then upload to the hapi-fhir test server. If it is successful, the patient information will populate the left text region. You can select a patient and request his/her observations from the server. Since the server has the patient, you are now able to search for it using his/her identifications.
+
+  .. image:: images/simulate.png
+     :align: center
+
+Note that the hapi-fhir server is not always stable. You can avoid uploading to server by not checking the `Upload` choice. If you do that, patient observations will directly populate the left text region.

docs/source/Working logic.rst

@@ -16,12 +16,12 @@ Mapping LOINC to candidate HPO terms
 
 LOINC observations have four main categories, quantitative(Qn), ordinal(Ord), nominal (Nom) and narrative(Nar). The majority of them are ``Qn`` (~50%) and ``Ord`` (~26%) and a smaller percentage are ``Nom`` and ``Nar`` type (<10%). The outcome of Qn and Ord type observations are typically represented with a code from the FHIR interpretation code valueset. Some examples are ``L`` for "low", ``LL`` for "critically low", or ``NEG`` for "negative". The full valueset is included in Table 2. Therefore, if we map those interpretation codes to a candidate HPO term for each LOINC, we will be able to convert an observation into a HPO term.
 
-However, the FHIR interpretation code valueset has 39 codes. It would be a massive task we need to map 39 HPO terms for each LOINC code, let alone there could be more than one interpretation code system for different EHR systems. Fortunately, many interpretation codes are  similar (e.g. FHIR ``L`` and ``LL`` both represent "low" but at different severity). So instead of annotating each each interpretation code, we will first map an interpretation code valueset into a much smaller internal code system with 7 values (Table 1). Then we just need to annotate 7 values for each LOINC code (actually 6 because "U" does not need an annotation) instead of 39. Table 2 shows the map between FHIR interpretation code valueset to our internal code system. Similar maps can be built to deal with other interpretation code systems.
+However, the FHIR interpretation code valueset has 39 codes. It would be a massive task if we need to map 39 HPO terms for each LOINC code, let alone there could be more than one interpretation code system for different EHR systems. Fortunately, many interpretation codes are  similar (e.g. FHIR ``L`` and ``LL`` both represent "low" but at different severity). So we picked 7 FHIR codes as our internal code to simplify the annotation job. Instead of annotating each each interpretation code, we will first map an interpretation code valueset into this internal FHIR subset (Table 1). Then we just need to annotate 7 values for each LOINC code (actually 6 because "U" does not need an annotation) instead of 39. Table 2 shows the map between FHIR interpretation code valueset to the FHIR subset. Similar maps can be built to deal with other interpretation code systems.
 
 Table 1: Internal Code System
 
 +-----------+------------------------+
-|code system|http://jax.org/loinc2hpo|
+|code system|   FHIR                 |
 +-----------+------------------------+
 |Code       |   Meaning              |
 +===========+========================+
@@ -33,19 +33,19 @@ Table 1: Internal Code System
 +-----------+------------------------+
 |H          |   high                 |
 +-----------+------------------------+
-|NP         |   not present          |
+|NEG        |   not present          |
 +-----------+------------------------+
-|P          |   present              |
+|POS        |   present              |
 +-----------+------------------------+
 |U          |   unknown              |
 +-----------+------------------------+
 
 Table 2: FHIR interpretation code set Mapping to internal code system
 
 +-----------------------------------+---------------------------+
-|FHIR interpretation code value set |internal code system       |
+|FHIR interpretation code value set |internal FHIR subset       |
 +-------+---------------------------+---------------------------+
-|system |http://hl7.org/fhir/v2/0078|http://jax.org/loinc2hpo   |
+|system |http://hl7.org/fhir/v2/0078|FHIR                       |
 +-------+---------------------------+--------+------------------+
 |Code   | Meaning                   |Code    | Meaning          |
 +=======+===========================+========+==================+
@@ -57,14 +57,14 @@ Table 2: FHIR interpretation code set Mapping to internal code system
 +-------+---------------------------+--------+------------------+
 |AA     |Critically abnormal        |A       |abnormal          |
 +-------+---------------------------+--------+------------------+
-|AC     |Anti-complementary         |P       |present           |
+|AC     |Anti-complementary         |POS     |present           |
 |       |substances present         |        |                  |
 +-------+---------------------------+--------+------------------+
 |B      |Better                     |N       |normal            |
 +-------+---------------------------+--------+------------------+
 |D      |Significant change down    |L       |low               |
 +-------+---------------------------+--------+------------------+
-|DET    |Detected                   |P       |present           |
+|DET    |Detected                   |POS     |present           |
 +-------+---------------------------+--------+------------------+
 |H      |High                       |H       |high              |
 +-------+---------------------------+--------+------------------+
@@ -92,11 +92,11 @@ Table 2: FHIR interpretation code set Mapping to internal code system
 +-------+---------------------------+--------+------------------+
 |N      |Normal                     |N       |normal            |
 +-------+---------------------------+--------+------------------+
-|ND     |Not Detected               |NP      |not present       |
+|ND     |Not Detected               |NEG     |not present       |
 +-------+---------------------------+--------+------------------+
-|NEG    |Negative                   |NP      |not present       |
+|NEG    |Negative                   |NEG     |not present       |
 +-------+---------------------------+--------+------------------+
-|NR     |Non-reactive               |NP      |not present       |
+|NR     |Non-reactive               |NEG     |not present       |
 +-------+---------------------------+--------+------------------+
 |NS     |Non-susceptible            |U       |unknown           |
 +-------+---------------------------+--------+------------------+
@@ -106,13 +106,13 @@ Table 2: FHIR interpretation code set Mapping to internal code system
 |OBX    |Interpretation qualifiers  |U       |unknown           |
 |       |in separate OBX segments   |        |                  |
 +-------+---------------------------+--------+------------------+
-|POS    |Positive                   |P       |positive          |
+|POS    |Positive                   |POS     |positive          |
 +-------+---------------------------+--------+------------------+
 |QCF    |Quality Control Failure    |U       |unknown           |
 +-------+---------------------------+--------+------------------+
 |R      |Resistant                  |U       |unknown           |
 +-------+---------------------------+--------+------------------+
-|RR     |Reactive                   |P       |present           |
+|RR     |Reactive                   |POS     |present           |
 +-------+---------------------------+--------+------------------+
 |S      |Susceptible                |U       |unknown           |
 +-------+---------------------------+--------+------------------+
@@ -122,7 +122,7 @@ Table 2: FHIR interpretation code set Mapping to internal code system
 +-------+---------------------------+--------+------------------+
 |SYN-S  |Synergy - susceptible	    |U       |unknown           |
 +-------+---------------------------+--------+------------------+
-|TOX    |Cytotoxic substance present|P       |present           |
+|TOX    |Cytotoxic substance present|POS     |present           |
 +-------+---------------------------+--------+------------------+
 |U      |Significant change up      |H       |high              |
 +-------+---------------------------+--------+------------------+
@@ -132,19 +132,18 @@ Table 2: FHIR interpretation code set Mapping to internal code system
 +-------+---------------------------+--------+------------------+
 |W      |Worse                      |A       |abnormal          |
 +-------+---------------------------+--------+------------------+
-|WR     |Weakly reactive            |P       |present           |
+|WR     |Weakly reactive            |POS     |present           |
 +-------+---------------------------+--------+------------------+
 
-We can make the annotation task even easier. If we analyze our internal codes carefully, ``NP`` ("not present"), resenting an outcome that the measured value is below a certain threshold, is actually the same as ``L`` ("low"), while ``P`` ("present") is the opposite and actually the same as ``H`` ("high"). Obviously, ``A`` ("abnormal") is simply the reverse of ``N`` ("normal"). So actually, we only need to map three HPO terms for each LOINC code, representing the outcomes of low value, intermediate value, and high value. The above rules should apply to the majority of LOINC codes, although we cannot rule out the possibility that they do not apply, so we will allow mapping different terms to ``NP`` and ``L``, ``P`` and ``H``, or ``A`` and ``N``.
+We can make the annotation task even easier. If we analyze our internal codes carefully, ``NEG`` ("not present") and "POS", resenting an outcome that the measured value is only applicable to ``Ord`` type of LOINC that has "present" or "absent" as its outcome. On the other hand, "L", "N", "A", "H" only apply to ``Qn`` type of LOINC. "A" and "N" will map to the same HPO term but have opposite negation value. What this means is that we only need to map 2 HPO terms to ``Ord`` LOINC with "presence"/"absence" outcome and 3 HPO terms to ``Qn`` LOINC.
 
 The following graph summarize the annotation process and how the app convert a LOINC observation into HPO terms. We pick three HPO terms for each LOINC code, representing the desired phenotype to assign for the patient when the observation value is low, intermediate, or high. By default, those three terms will be mapped to the internal coding values, which further map to external code values used in real world.
 
     .. image:: images/annotation_scheme.png
 
-* note: A LOINC observation may not have both ``P`` ``NP`` and ``L`` ``H``. For example, annotating ``10449-7`` (``"Glucose in Serum or Plasma -- 1 hour post meal"``) with ``P`` ``NP`` seems unrealistic. Doing this, however, will not affect how the app performs when it tries to convert an observation into HPO terms, as ``P`` and ``NP`` will never be encountered, and it does make the annotation task easier.
+* note:
 
-
-``Ord``, ``Nom`` and ``Nar`` observations can use other coding systems that are more difficult to handle. For example, ``Loinc 600-7`` or "Bacteria identified in Blood by Culture" may use a SNOMED concept to represent the finding that *Staphylococcus aureus* is detected::
+``Ord``(non-"presence"/"absence" outcome), ``Nom`` and ``Nar`` observations can use other coding systems that are more difficult to handle. For example, ``Loinc 600-7`` or "Bacteria identified in Blood by Culture" may use a SNOMED concept to represent the finding that *Staphylococcus aureus* is detected::
 
   "coding":[
   {

loinc2hpo-core/pom.xml

@@ -5,7 +5,7 @@
     <parent>
         <artifactId>Loinc2Hpo</artifactId>
         <groupId>org.monarchinitiative</groupId>
-        <version>1.1.2</version>
+        <version>1.1.3</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

loinc2hpo-core/src/main/java/org/monarchinitiative/loinc2hpo/Constants.java

@@ -0,0 +1,33 @@
+package org.monarchinitiative.loinc2hpo;
+
+public class Constants {
+
+    //missing values are noted as "NA" during serialization
+    public static final String MISSINGVALUE = "NA";
+
+    //folder and file names for serializations
+    //1. TSVSingleFile format
+    public static final String TSVSingleFileFolder = "TSVSingleFile";
+    public static final String TSVSingleFileName = "annotations.tsv";
+    //2. TSVSeperateFiles format
+    public static final String TSVSeparateFilesFolder = "TSVSeparateFiles";
+    public static final String TSVSeparateFilesBasic = "basic_annotations.tsv";
+    public static final String TSVSeparateFilesAdv = "advanced_annotations.tsv";
+    //3. JSON format
+    public static final String JSONFileFolder = "JSON";
+    public static final String JSONfile = "annotations.json";
+
+    //folder for LOINC categories
+    public static final String LOINCCategory = "LOINC CATEGORY";
+
+
+    public static final String LOINCSYSTEM = "http://loinc.org";
+    public static final String HAPIFHIRTESTSERVER = "http://fhirtest.uhn.ca/baseDstu3";
+    public static final String UNITSYSTEM = "http://unitsofmeasure.org";
+
+    public static final String V2OBSERVATIONINTERPRETATION = "http://hl7.org/fhir/v2/0078";
+    public static final String V3OBSERVATIONINTERPRETATION = "http://hl7.org/fhir/v3/ObservationInterpretation";
+
+
+
+}