Fixes for HG38 exception and better logging. #4563

jonn-smith · 2018-03-22T21:53:03Z

Added more logging functionality to better inform users.

Fixed NullPointerException when onTraversalStart throws.
Now logs version number information for data sources.
Now checks minimum data sources version and requires that the user
upgrade if the version is too old.

Fixes #4521 in a roundabout way - data sources need to be re-released for this version to run at all, but these new data sources contain a fix for the HG38 issue.

codecov-io · 2018-03-23T14:47:18Z

Codecov Report

Merging #4563 into master will increase coverage by 0.075%.
The diff coverage is 43.802%.

@@               Coverage Diff               @@
##              master     #4563       +/-   ##
===============================================
+ Coverage     79.857%   79.932%   +0.075%     
- Complexity     17054     17347      +293     
===============================================
  Files           1067      1067               
  Lines          62031     63140     +1109     
  Branches       10039     10311      +272     
===============================================
+ Hits           49536     50469      +933     
- Misses          8582      8696      +114     
- Partials        3913      3975       +62

Impacted Files	Coverage Δ	Complexity Δ
...broadinstitute/hellbender/utils/test/BaseTest.java	`64.444% <0%> (-0.481%)`	`33 <0> (ø)`
...r/dataSources/cosmic/CosmicFuncotationFactory.java	`76.238% <0%> (-4.815%)`	`21 <0> (ø)`
...ncotator/mafOutput/MafOutputRendererConstants.java	`98.98% <100%> (ø)`	`1 <0> (ø)`	⬇️
...tools/funcotator/DataSourceFuncotationFactory.java	`90% <100%> (ø)`	`17 <1> (ø)`	⬇️
...titute/hellbender/tools/funcotator/Funcotator.java	`76.351% <100%> (ø)`	`30 <2> (ø)`	⬇️
.../tools/funcotator/dataSources/DataSourceUtils.java	`56.432% <29.73%> (-11.616%)`	`33 <7> (+7)`
...cotator/dataSources/vcf/VcfFuncotationFactory.java	`85.333% <58.333%> (-4.219%)`	`23 <2> (+1)`
...dataSources/gencode/GencodeFuncotationFactory.java	`81.997% <75%> (+1.209%)`	`146 <1> (+4)`	⬆️
...ataSources/xsv/SimpleKeyXsvFuncotationFactory.java	`87.097% <77.778%> (-0.403%)`	`27 <2> (+1)`
...ataSources/xsv/LocatableXsvFuncotationFactory.java	`84.694% <88.889%> (+0.823%)`	`28 <3> (+2)`	⬆️
... and 20 more

LeeTL1220

Some changes and questions

LeeTL1220 · 2018-03-26T15:11:28Z

src/main/java/org/broadinstitute/hellbender/tools/funcotator/Funcotator.java

@@ -397,10 +397,13 @@ public Object onTraversalSuccess() {
    public void closeTool() {

        for(final DataSourceFuncotationFactory factory : dataSourceFactories) {
-            factory.close();
+            if ( factory != null ) {


Would this be better as Closeable?

Sure, I can change DataSourceFuncotationFactory to be Closeable.

LeeTL1220 · 2018-03-26T15:15:15Z

src/main/java/org/broadinstitute/hellbender/tools/funcotator/dataSources/DataSourceUtils.java

+    }
+
+    @VisibleForTesting
+    static boolean validateVersionInformation(final int major, final int minor, final int year, final int month, final int day) {


No need for final on primitives for method parameters.

I'm inclined to leave it as it has semantic meaning as well.

Except that the codebase does not do this elsewhere.

LeeTL1220 · 2018-03-26T15:18:51Z

src/main/java/org/broadinstitute/hellbender/tools/funcotator/dataSources/DataSourceUtils.java

+                    logger.info("Data sources alternate source: " + alternateSource);
+                }
+            }
+            catch (final Exception ex) {


Please log information about the exception. Generally, you should not be catching Exception, but rather a subclass of Exception.

Sounds good.

I want to catch any exceptions that happen there and continue so that it doesn't actually fail. This is an optional check for logging purposes anyway.

LeeTL1220 · 2018-03-26T15:19:33Z

src/main/java/org/broadinstitute/hellbender/tools/funcotator/dataSources/DataSourceUtils.java

+
+        boolean dataSourcesPathIsAcceptable = true;
+
+        final Path readmePath = dataSourcesPath.resolve(IOUtils.getPath(README_FILE_NAME));


Where is documentation about the format of the README file?

I'll add in an explanation of what is required in the README for these purposes.

LeeTL1220 · 2018-03-26T15:20:21Z

src/test/java/org/broadinstitute/hellbender/tools/funcotator/FuncotatorIntegrationTest.java

        arguments.addVCF(new File(inputFile));
        arguments.addReference(new File("/Users/jonn/Development/references/Homo_sapiens_assembly19.fasta"));

-        arguments.addArgument(FuncotatorArgumentDefinitions.DATA_SOURCES_PATH_LONG_NAME, "/Users/jonn/Development/funcotator_dataSources.v1.1.20180208");
+        arguments.addArgument(FuncotatorArgumentDefinitions.DATA_SOURCES_PATH_LONG_NAME, "/Users/jonn/Development/funcotator_dataSources_latestjnkjjk");


This pathname can't possibly be correct. Must be something that would be on anyone's dev machine.

This test is intermediate and planned to be phased out.

It is checked in so that I can run my local test data and not have to worry about rewriting tests every time I need to debug something.

I added the flag doDebugTests to control for this and a unit test to ensure that doDebugTests is always set to false for the tests to pass.

I want to leave these in for now so I can continue to test locally with different settings easily.

LeeTL1220 · 2018-03-26T15:20:28Z

src/test/java/org/broadinstitute/hellbender/tools/funcotator/FuncotatorIntegrationTest.java

@@ -275,10 +277,12 @@ public void spotCheck2() throws IOException {

        final ArgumentsBuilder arguments = new ArgumentsBuilder();

+        arguments.addArgument("verbosity", "INFO");
+
        arguments.addVCF(new File(inputFile));
        arguments.addReference(new File("/Users/jonn/Development/references/Homo_sapiens_assembly19.fasta"));


See next comment.

LeeTL1220 · 2018-03-26T15:20:58Z

src/test/java/org/broadinstitute/hellbender/tools/funcotator/FuncotatorIntegrationTest.java

@@ -235,6 +235,8 @@ public void spotCheck() throws IOException {

            final ArgumentsBuilder arguments = new ArgumentsBuilder();

+            arguments.addArgument("verbosity", "INFO");
+
            arguments.addVCF(new File("/Users/jonn/Development/M2_01115161-TA1-filtered.vcf"));


Please remove all references to /Users/jonn/Development/ etc

This test is intermediate and planned to be phased out.

It is checked in so that I can run my local test data and not have to worry about rewriting tests every time I need to debug something.

I added the flag doDebugTests to control for this and a unit test to ensure that doDebugTests is always set to false for the tests to pass.

I want to leave these in for now so I can continue to test locally with different settings easily.

But why can't you put these tests into the source tree and then remove if/when no longer needed?
You are asking to put these into master by issuing this PR and we cannot cut a release with these tests -- and master needs to always be release-ready.

There should be an alternative way of doing this.

LeeTL1220 · 2018-03-26T15:27:29Z

src/main/java/org/broadinstitute/hellbender/tools/funcotator/dataSources/DataSourceUtils.java

            final Path p = IOUtils.getPath(pathString);
            if ( !isValidDirectory(p) ) {
-                logger.warn("WARNING: Given path is not a valid directory: " + p.toUri().toString());
+                logger.error("ERROR: Given data source path is not a valid directory: " + p.toUri().toString());


I notice that you have a lot of error messages that do not throw exceptions. Is this correct behavior?

It's an error to have an invalid data source path.

You're right - I'll throw an exception here.

LeeTL1220 · 2018-03-26T15:28:09Z

src/main/java/org/broadinstitute/hellbender/tools/funcotator/dataSources/DataSourceUtils.java

+                String line = reader.readLine();
+                while ((line != null) && ((version == null) || (source == null) || (alternateSource == null))) {
+
+                    if (version == null && line.startsWith(README_VERSION_LINE_START)) {


Why have a README and a config file? Can't we merge those?

@LeeTL1220 I don't understand... there is no config file for the data sources directory - only a README file.

Ok... I thought this code was being run for each datasource. Not the datasources as a whole...

LeeTL1220 · 2018-03-26T15:29:54Z

src/main/java/org/broadinstitute/hellbender/tools/funcotator/dataSources/DataSourceUtils.java

+                            version = versionMajor + "." + versionMinor + "." + versionYear + "" + versionMonth + "" + versionDay;
+                        }
+                        else {
+                            logger.warn("README file has improperly formatted version string: " + line);


README files typically have minimal to zero formatting. This would indicate that we have a stricter format. Is there a good reason not to merge with the config file?

@LeeTL1220 There is no config file for a data sources directory - only a README file.

I guess we can call it a config file, but then no one is likely to read it and it provides information to anyone who would download / use the data sources.

Leave w/ README. Again, I thought this was being run for each datasource. Not once for all datasources.

LeeTL1220 · 2018-03-26T15:31:06Z

@jonn-smith Back to you.

jonn-smith · 2018-03-27T15:21:51Z

OK, @LeeTL1220 - First round comments addressed.

LeeTL1220

@jonn-smith I do not see how we can leave in hardocded paths in master branch. Let's discuss alternatives.

LeeTL1220 · 2018-03-27T15:26:24Z

src/main/java/org/broadinstitute/hellbender/tools/funcotator/dataSources/DataSourceUtils.java

+                String line = reader.readLine();
+                while ((line != null) && ((version == null) || (source == null) || (alternateSource == null))) {
+
+                    if (version == null && line.startsWith(README_VERSION_LINE_START)) {


Ok... I thought this code was being run for each datasource. Not the datasources as a whole...

LeeTL1220 · 2018-03-27T15:26:50Z

src/main/java/org/broadinstitute/hellbender/tools/funcotator/dataSources/DataSourceUtils.java

+                            version = versionMajor + "." + versionMinor + "." + versionYear + "" + versionMonth + "" + versionDay;
+                        }
+                        else {
+                            logger.warn("README file has improperly formatted version string: " + line);


Leave w/ README. Again, I thought this was being run for each datasource. Not once for all datasources.

LeeTL1220 · 2018-03-27T15:29:13Z

src/test/java/org/broadinstitute/hellbender/tools/funcotator/FuncotatorIntegrationTest.java

@@ -235,6 +235,8 @@ public void spotCheck() throws IOException {

            final ArgumentsBuilder arguments = new ArgumentsBuilder();

+            arguments.addArgument("verbosity", "INFO");
+
            arguments.addVCF(new File("/Users/jonn/Development/M2_01115161-TA1-filtered.vcf"));


But why can't you put these tests into the source tree and then remove if/when no longer needed?
You are asking to put these into master by issuing this PR and we cannot cut a release with these tests -- and master needs to always be release-ready.

There should be an alternative way of doing this.

jonn-smith · 2018-03-28T15:58:08Z

@LeeTL1220 that's totally fair. There are a few problems with putting in the tests, though.

I don't know that I can publicly release the data that are being tested because of their provenance (pretty sure I can't).

The second reason is that the whole reference sequences are not checked into git lfs. Without it I can't annotate on a set of variants from around the whole genome, and the whole reference is pretty big.

The last thing I can think of is a data source problem - I don't have all the data sources checked into git lfs (for similar reasons, especially now with dbSNP in there).

Also, these have already made their way into master. So if this is really a problem, we should get them out of there ASAP.

LeeTL1220 · 2018-03-28T16:06:36Z

@jonn-smith Let's discuss offline. Yes, let's get these out of master.

Fixed NullPointerException when onTraversalStart throws. Now logs version number information for data sources. Now checks minimum data sources version and requires that the user upgrade if the version is too old.

Now dbSNP uses the `All` datasource, not the `common` data source. Refactored largeDataValidationTest to use an environment variable to point to the test data files.

funcotatorValidation tests no longer run by default. Need to enable these tests in jenkins.

Had to change the expected data because of new mappings from data source fields to MAF output fields.

…nced.

Added more logging functionality to better inform users. Fixed NullPointerException when onTraversalStart throws. Now logs version number information for data sources. Now checks minimum data sources version and requires that the user upgrade if the version is too old. Fixed a bug that caused __UNKNOWN__ annotations. Refactored the spot checks into largeDataValidationTest. Now dbSNP uses the `All` datasource, not the `common` data source. Refactored largeDataValidationTest to use an environment variable to point to the test data files. Moved funcotatorValidation tests into their own group; funcotatorValidation tests no longer run by default.

jonn-smith added the Funcotator label Mar 22, 2018

jonn-smith assigned LeeTL1220 Mar 22, 2018

jonn-smith requested a review from LeeTL1220 March 22, 2018 21:53

jonn-smith requested a review from lbergelson March 23, 2018 17:10

jonn-smith assigned lbergelson Mar 23, 2018

LeeTL1220 reviewed Mar 26, 2018

View reviewed changes

jonn-smith removed the request for review from lbergelson March 27, 2018 15:01

jonn-smith unassigned lbergelson Mar 27, 2018

LeeTL1220 reviewed Mar 27, 2018

View reviewed changes

jonn-smith force-pushed the jts_funcotator_hg38_error_4521 branch from 1da4c11 to 0134757 Compare March 28, 2018 20:41

jonn-smith added 2 commits March 29, 2018 10:57

Added more logging functionality to better inform users.

0745ea8

Fixed NullPointerException when onTraversalStart throws. Now logs version number information for data sources. Now checks minimum data sources version and requires that the user upgrade if the version is too old.

Addressing comments.

5d85137

jonn-smith force-pushed the jts_funcotator_hg38_error_4521 branch from 0134757 to 5d85137 Compare March 29, 2018 14:57

Fixed a bug that caused __UNKNOWN__ annotations.

5e59201

LeeTL1220 assigned jonn-smith and unassigned LeeTL1220 Mar 30, 2018

jonn-smith added 5 commits March 30, 2018 14:54

Refactored the spot checks into largeDataValidationTest.

34c8ceb

Now dbSNP uses the `All` datasource, not the `common` data source. Refactored largeDataValidationTest to use an environment variable to point to the test data files.

Moved funcotatorValidation tests into their own group.

9742487

funcotatorValidation tests no longer run by default. Need to enable these tests in jenkins.

Fixing a unit test.

fefa1d8

Had to change the expected data because of new mappings from data source fields to MAF output fields.

Minor fixes to tests.

5360b0c

Fixed a bug in the tests with how the environment variable was refere…

4269633

…nced.

jonn-smith merged commit 2db999e into master Apr 6, 2018

jonn-smith deleted the jts_funcotator_hg38_error_4521 branch April 6, 2018 21:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes for HG38 exception and better logging. #4563

Fixes for HG38 exception and better logging. #4563

jonn-smith commented Mar 22, 2018

codecov-io commented Mar 23, 2018 •

edited

LeeTL1220 left a comment

LeeTL1220 Mar 26, 2018

jonn-smith Mar 27, 2018 •

edited

LeeTL1220 Mar 26, 2018

jonn-smith Mar 27, 2018 •

edited

LeeTL1220 Mar 27, 2018

LeeTL1220 Mar 26, 2018

jonn-smith Mar 27, 2018

LeeTL1220 Mar 26, 2018

jonn-smith Mar 27, 2018

LeeTL1220 Mar 26, 2018

jonn-smith Mar 27, 2018

LeeTL1220 Mar 26, 2018

LeeTL1220 Mar 26, 2018

jonn-smith Mar 27, 2018

LeeTL1220 Mar 27, 2018

LeeTL1220 Mar 26, 2018

jonn-smith Mar 27, 2018

LeeTL1220 Mar 26, 2018

jonn-smith Mar 27, 2018 •

edited

LeeTL1220 Mar 27, 2018

LeeTL1220 Mar 26, 2018

jonn-smith Mar 27, 2018

LeeTL1220 Mar 27, 2018

LeeTL1220 commented Mar 26, 2018

jonn-smith commented Mar 27, 2018

LeeTL1220 left a comment

LeeTL1220 Mar 27, 2018

LeeTL1220 Mar 27, 2018

LeeTL1220 Mar 27, 2018

jonn-smith commented Mar 28, 2018

LeeTL1220 commented Mar 28, 2018


		boolean dataSourcesPathIsAcceptable = true;

		final Path readmePath = dataSourcesPath.resolve(IOUtils.getPath(README_FILE_NAME));

Fixes for HG38 exception and better logging. #4563

Fixes for HG38 exception and better logging. #4563

Conversation

jonn-smith commented Mar 22, 2018

codecov-io commented Mar 23, 2018 • edited

Codecov Report

LeeTL1220 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonn-smith Mar 27, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonn-smith Mar 27, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonn-smith Mar 27, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LeeTL1220 commented Mar 26, 2018

jonn-smith commented Mar 27, 2018

LeeTL1220 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonn-smith commented Mar 28, 2018

LeeTL1220 commented Mar 28, 2018

codecov-io commented Mar 23, 2018 •

edited

jonn-smith Mar 27, 2018 •

edited

jonn-smith Mar 27, 2018 •

edited

jonn-smith Mar 27, 2018 •

edited