Improved database/querying (+ additional changes). #14

svandenhoek · 2019-11-05T13:17:26Z

Repository split into 2 major directories: "app" & "database".
App now uses an optimized TDB.
SPARQL construct .rq scripts are included to create optimized TDB from full one.
Designed with DisGeNET v6 in mind with phenotype-disease annotiations from DisGeNET v5 (instead of purely v5).
Orphanet HOOM data is now also used to find relevant information.
Results now only use GDA scores of GDAs matched with relevant diseases instead of all diseases that matched to found genes.
Fixed multiple maven build warnings.
General code improvements (such as for the BiologicalEntityCollection).
Split README into seperate ones (and made adjustments according to the changes).
Some code was commented as it is currently not used (code causing errors f.e. due to existing unit tests were not commented).
Adjustments to outputwriter due to optimized TDB including less information (some information is not stored/retrieved anymore).
Added separate .rq files with SPARQL select queries which can be used with Apache Jena's tdbquery for testing/validation (optimized vs full query/TDB).

…imized TDB.

…d TDB (for speed/equality output).

…ts to associated test classes).

… now uses custom TDB.

… app).

…fic).

…inations.

…eved anymore).

…ame results as original TDB/query.

…recated".

… examples).

…tySets that missing previously) to remove "unchecked cast" warning. Solution might be less efficient.

dennishendriksen · 2019-11-07T08:43:48Z

app/src/main/java/org/molgenis/vibe/formats/BiologicalEntity.java

+    }
+
+    protected void setCompareValue(int compareValue) {
+        this.compareValue = requireNonNull(compareValue);


compareValue is non-null because it is a primitive. The requireNonNull check always passes. Did you intend to check whether compareValue is zero?

It was simply a tendency to overvalidate things without considering it might have been unnecessary. Will fix.

dennishendriksen · 2019-11-07T08:59:06Z

database/README.md

+
+[jena_download]: https://jena.apache.org/download/index.cgi
+[jena_configure]: https://jena.apache.org/documentation/tools/#setting-up-your-environment
+[disgenet_rdf_v6_dump]: http://rdf.disgenet.org/download/v5.0.0/disgenetv5.0-rdf-v5.0.0-dump.tar.gz


Points to v5 resources instead of v6.

dennishendriksen · 2019-11-07T12:54:23Z

database/README.md

+## Creating optimized TDB
+
+1. Create a directory to store optimized `.ttl` files in.
+2. Run `tdbquery --loc=/path/to/initial/TDB/ --query=/path/to/vibe/db_creation/sparql_queries/hpo.rq 1> /path/to/optimized/ttl/hpo.ttl`


db_creation --> database/optimized_construct

Updated to vibe/database/sparql_queries/optimized_construct/.

dennishendriksen · 2019-11-07T12:54:35Z

database/README.md

+
+1. Download the data.
+2. Rename `owlapi.xrdf` to `owlapi.xml` (otherwise `tdbloader2` will give `org.apache.jena.riot.RiotException: Failed to determine the content type`)
+3. Run `tdbloader2 --loc /path/to/initial/TDB /path/to/disgenet_v6/dump/*.ttl /path/to/disgenet_v5/pda.ttl /path/to/disgenet_v5/phenotype.ttl  /path/to/disgenet_v5/void.ttl /path/to/sio-release.owl /path/to/owlapi.xml`


tdbloader for windows

Special paragraph for Windows users was added.

dennishendriksen · 2019-11-07T13:45:50Z

database/README.md

+4. Run `tdbquery --loc=/path/to/initial/TDB/ --query=/path/to/vibe/db_creation/sparql_queries/gene.rq 1> /path/to/optimized/ttl/gene.ttl`
+5. Run `tdbquery --loc=/path/to/initial/TDB/ --query=/path/to/vibe/db_creation/sparql_queries/gda.rq 1> /path/to/optimized/ttl/gda.ttl`
+6. Run `tdbquery --loc=/path/to/initial/TDB/ --query=/path/to/vibe/db_creation/sparql_queries/source.rq 1> /path/to/optimized/ttl/source.ttl`
+7. Run `tdbloader2 --loc /path/to/store/optimized/TDB /path/to/optimized/ttl/*.ttl  /path/to/sio-release.owl`


Add a note that when you receive the following error:

org.apache.jena.atlas.RuntimeIOException: java.nio.charset.MalformedInputException: Input length = 1

then the encoding of the ttls might be incorrect and needs to be changed to UTF-8 manually.

Added F.A.Q. with this information.

dennishendriksen · 2019-11-08T06:26:52Z

app/src/main/java/org/molgenis/vibe/formats/Gene.java

@@ -19,7 +21,7 @@
    /**
     * The HGNC (HUGO Gene Nomenclature Committee) name.
     */
-    private String symbol;
+//    private String symbol; // Currently unused.


Then why not remove it?

I suggest leaving it in for now and doing a general code cleanup when:

A more definitive output writer is implemented.

This project is added to sonarcloud.

dennishendriksen · 2019-11-08T06:27:34Z

...src/main/java/org/molgenis/vibe/io/output/ResultsPerGeneSeparatedValuesFileOutputWriter.java

@@ -81,14 +81,16 @@ public void run() throws IOException {
        BufferedWriter writer = getWriter();

        // Writes header.
-        writer.write("gene" + getSeparator() + "diseases" + getSeparator() + "highest GDA score" +
-                getSeparator() + "DSI" + getSeparator() + "DPI");
+        // Currently dsi & dpi are not retrieved.


Then why not remove it?

See response above.

dennishendriksen · 2019-11-08T06:28:32Z

app/src/test/java/org/molgenis/vibe/formats/GeneDiseaseCollectionTester.java

+        Assert.assertEquals(collection.getByDisease(diseases[1]), null);
+    }
+
+    //TODO: More tests for basic java.util.Collection functionalities!


I suggest to create an issue for your TODOs to prevent them sticking around forever. Better: resolve the todo now.

Forgot to remove the TODO. Methods that are more complicated than simply doing the exact same action on the combinationsMap or combinationsMap.keySet() should now all have at least one unit-test.

svandenhoek added 30 commits September 17, 2019 14:54

Adjustments to support DisGeNET v6.0 (including a simplified query).

2e1772c

Removes step for retrieving additional diseases that match a found gene.

05d7fb8

Moved Java app to a subfolder.

fb777cb

Initial SPARQL construct query for HPOs linked to diseases.

e080dd3

Startup for TDB creation specific README.

af5463c

Updated construct script for updated initial TDB.

146c058

Adjusted directory structure.

12546c5

Renamed SPARQL query file.

67ce9c0

Initial set of SPARQL construct scripts needed to create optimized TDB.

9b8f2c8

Updated README for creating optimized TDB.

733112d

Additional README adjustments for creating database.

e598c3d

Reworked README's.

fed7b69

Minor adjustments to main README.

ed3f240

README adjustments.

c3837b7

Updated READMe links.

be128e8

Added pubmed sources from GDAs to optimized construct queries for opt…

d6417a7

…imized TDB.

Added SPARQL scripts for comparing the original TDB with the optimize…

1fd009d

…d TDB (for speed/equality output).

Fixed faulty comment.

ffb5652

Added construct script for (relevant) sources.

e619035

Updated construct prefixes so that they are all equal.

a08ae38

Updated README.

af83fc6

Now uses URI's for comparison of Sources and Biological Entities.

b2b483a

More tests.

bf35ca8

Adjustments to BiologicalEntity constructors for new TDB (+ adjustmen…

df77e0f

…ts to associated test classes).

Removed information regarding DisGeNET version from option parsing as…

31ff218

… now uses custom TDB.

Removed obsolete classes.

64b4260

Updated SPARQL prefixes.

193837e

Adjustment regarding renamed options parsing methods.

04c41a8

Minor adjustment to SPARQL query (more similar to query to-be used in…

95fa47e

… app).

Initial adjustments SPARQL query (digestion) for custom TDB.

c59e0b2

svandenhoek added 24 commits October 31, 2019 10:58

compareTo now uses same variable as equals for BiologicalEntities.

8590d79

Additional adjustments to SPARQL query digestion.

1231b47

Fixed unit test using wrong HPO term to test on.

76599fc

Removed whitespace.

717a14c

Implemented improved BiologicalEntity ordering (is now subclass-speci…

5931fb7

…fic).

BiologicalEntityCollection can now return ordered list of stored comb…

c4c115f

…inations.

Fixed unit test.

11f1891

Removed outdated tests and some test code adjustments.

9fb6298

Fixed bug in output writer (regarding information currently not retri…

a8b5229

…eved anymore).

Minor verbose adjustments.

61bceeb

Commented some unused code.

8b205c7

Added bash script to compare if the optimized TDB/query outputs the s…

fe77227

…ame results as original TDB/query.

Minor adjustment to bash script usage.

3869d5f

Updated .gitignore file.

0a7ae53

Added encoding information to app/pom.xml (fixed maven warning).

14ef1ec

Fix for maven warning "unchecked conversion".

8ee5be1

Fix for maven warning "unchecked conversion".

852b3d5

Fix for maven warning "Integer(int) in java.lang.Integer has been dep…

d696e4d

…recated".

Merge remote-tracking branch 'molgenis/master' into tdb_design_app

8032a8a

Temporarily removed output file examples from README (need up-to-date…

828db27

… examples).

Improved BiologicalEntityCollection and added more tests for it.

a23ea97

Added retainAll test for BiologicalEntityCollection.

b9f768b

Fix for ConcurrentModificationException.

c881cf0

Reverted remove/removeAll BiologicalEntity code (with added removeEmp…

6809d4c

…tySets that missing previously) to remove "unchecked cast" warning. Solution might be less efficient.

svandenhoek mentioned this pull request Nov 5, 2019

Revamped package structure #15

Merged

dennishendriksen reviewed Nov 8, 2019

View reviewed changes

Made adjustments according to code review.

bf2dc23

dennishendriksen approved these changes Nov 12, 2019

View reviewed changes

dennishendriksen merged commit 37b87e9 into molgenis:master Nov 12, 2019

svandenhoek deleted the tdb_design_app branch December 10, 2019 16:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improved database/querying (+ additional changes). #14

Improved database/querying (+ additional changes). #14

svandenhoek commented Nov 5, 2019

dennishendriksen Nov 7, 2019

svandenhoek Nov 12, 2019

dennishendriksen Nov 7, 2019

svandenhoek Nov 12, 2019

dennishendriksen Nov 7, 2019

svandenhoek Nov 12, 2019

dennishendriksen Nov 7, 2019

svandenhoek Nov 12, 2019

dennishendriksen Nov 7, 2019

svandenhoek Nov 12, 2019

dennishendriksen Nov 8, 2019

svandenhoek Nov 12, 2019

dennishendriksen Nov 8, 2019

svandenhoek Nov 12, 2019

dennishendriksen Nov 8, 2019

svandenhoek Nov 12, 2019

Improved database/querying (+ additional changes). #14

Improved database/querying (+ additional changes). #14

Conversation

svandenhoek commented Nov 5, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment