adding fine control over jexl expression behavior with missing values (…
…#772) * adding JexlMissingValueTreatment to control how values in a jexl expression that are missing in the evaluated context are treated * adding overloads to VariantContextUtils.match to allow control of missing value behavior * minor refactoring in JEXLMap * making enum method package private * responding to comments * updated tests * moving JexlMissingValueTreatment to top level * fixing typo

…#772) * adding JexlMissingValueTreatment to control how values in a jexl expression that are missing in the evaluated context are treated * adding overloads to VariantContextUtils.match to allow control of missing value behavior * minor refactoring in JEXLMap * making enum method package private * responding to comments * updated tests * moving JexlMissingValueTreatment to top level * fixing typo
removed VariantContextWriterFactory (#756)
samtools · lbergelson · magicDGS · droazen · lbergelson · magicDGS
Unified Split

Showing 485 changed files with 8,996 additions and 3,314 deletions.
diff --git a/.codecov.yml b/.codecov.yml
@@ -0,0 +1,29 @@
+codecov:
+  branch: master
+
+coverage:
+  precision: 3
+  round: nearest
+  range: "50...100"
+
+  status:
+    project:
+      default:
+        target: auto
+        threshold: .01
+        branches: null
+
+    patch:
+      default:
+        target: auto
+        branches: null
+
+    changes:
+      default:
+        branches: null
+
+
+comment:
+  layout: "header, diff, changes, sunburst, uncovered, tree"
+  branches: null
+  behavior: default
diff --git a/.travis.yml b/.travis.yml
@@ -10,11 +10,12 @@ cache:
     - $HOME/.m2
 jdk:
   - oraclejdk8
-script: ./gradlew jacocoTestReport testSRA;
+  - openjdk8
+script: ./gradlew test jacocoTestReport;
 after_success:
+  - bash <(curl -s https://codecov.io/bash)
   - echo "TRAVIS_BRANCH='$TRAVIS_BRANCH'";
     echo "JAVA_HOME='$JAVA_HOME'";
-    ./gradlew coveralls;
     if [ "$TRAVIS_BRANCH" == "master" ]; then
        ./gradlew uploadArchives;
     fi
diff --git a/README.md b/README.md
@@ -1,8 +1,9 @@
-[![Coverage Status](https://coveralls.io/repos/github/samtools/htsjdk/badge.svg?branch=master)](https://coveralls.io/github/samtools/htsjdk?branch=master)
+[![Coverage Status](https://codecov.io/gh/samtools/htsjdk/branch/master/graph/badge.svg)](https://codecov.io/gh/samtools/htsjdk)
 [![Build Status](https://travis-ci.org/samtools/htsjdk.svg?branch=master)](https://travis-ci.org/samtools/htsjdk)
 [![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.github.samtools/htsjdk/badge.svg)](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.github.samtools%22%20AND%20a%3A%22htsjdk%22)
 [![License](http://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/samtools/htsjdk)
 [![Language](http://img.shields.io/badge/language-java-brightgreen.svg)](https://www.java.com/)
+[![Join the chat at https://gitter.im/samtools/htsjdk](https://badges.gitter.im/samtools/htsjdk.svg)](https://gitter.im/samtools/htsjdk?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
 
 Status of downstream projects automatically built on top of the current htsjdk master branch. See [gatk-jenkins](https://gatk-jenkins.broadinstitute.org/view/HTSJDK%20Release%20Tests/) for detailed logs. Failure may indicate problems  in htsjdk, but may also be due to expected incompatibilities between versions, or unrelated failures in downstream projects.
 - [Picard](https://github.com/broadinstitute/picard):  [![Build Status](https://gatk-jenkins.broadinstitute.org/buildStatus/icon?job=picard-on-htsjdk-master)](https://gatk-jenkins.broadinstitute.org/job/picard-on-htsjdk-master/)
@@ -15,11 +16,19 @@ common file formats, such as [SAM][1] and [VCF][2], used for high-throughput
 sequencing data.  There are also an number of useful utilities for 
 manipulating HTS data.
 
-Please see the [HTSJDK Documentation](http://samtools.github.io/htsjdk) for more information.
-
 > **NOTE: _HTSJDK does not currently support the latest Variant Call Format Specification (VCFv4.3 and BCFv2.2)._**
 
-#### Building HTSJDK
+### Documentation & Getting Help
+
+API documentation for all versions of HTSJDK since `1.128` are available through [javadoc.io](http://www.javadoc.io/doc/com.github.samtools/htsjdk).
+
+If you believe you have found a bug or have an issue with the library please a) search the open and recently closed issues to ensure it has not already been reported, then b) log an issue.
+
+The project has a [gitter chat room](https://gitter.im/samtools/htsjdk) if you would like to chat with the developers and others involved in the project.
+
+To receive announcements of releases and other significant project news please subscribe to the [htsjdk-announce](https://groups.google.com/forum/#!forum/htsjdk-announce) google group.
+
+### Building HTSJDK
 
 HTSJDK is now built using [gradle](http://gradle.org/).
 
@@ -73,7 +82,7 @@ Example gradle usage from the htsjdk root directory:
  ./gradlew tasks
  ```
 
-#### Create an HTSJDK project in IntelliJ
+### Create an HTSJDK project in IntelliJ
 To create a project in IntelliJ IDE for htsjdk do the following:
 
 1. Select fom the menu: `File -> New -> Project from Existing Sources`
@@ -82,23 +91,26 @@ To create a project in IntelliJ IDE for htsjdk do the following:
 
 From time to time if dependencies change in htsjdk you may need to refresh the project from the `View -> Gradle` menu.
 
-#### Licensing Information
+### Licensing Information
 
-Not all sub-packages of htsjdk are subject to the same license, so a license notice is included in each source file or sub-package as appropriate. Please check the relevant license notice whenever you start working with a part of htsjdk that you have not previously worked with to avoid any surprises. 
+Not all sub-packages of htsjdk are subject to the same license, so a license notice is included in each source file or sub-package as appropriate. 
+Please check the relevant license notice whenever you start working with a part of htsjdk that you have not previously worked with to avoid any surprises. 
+Broadly speaking the majority of the code is covered under the MIT license with the following notable exceptions:
 
-#### Java Minimum Version Support Policy
+* Much of the CRAM code is under the Apache License, Version 2
+* Core `tribble` code (underlying VCF reading/writing amongst other things) is under LGPL
+* Code supporting the reading/writing of SRA format is uncopyrighted & public domain
 
-> **NOTE: _Effective November 24th 2015, HTSJDK has ended support of Java 7 and previous versions. Java 8 is now required_.**
+### Java Minimum Version Support Policy
 
 We will support all Java SE versions supported by Oracle until at least six months after Oracle's Public Updates period has ended ([see this link](http://www.oracle.com/technetwork/java/eol-135779.html)).
 
 Java SE Major Release | End of Java SE Oracle Public Updates | Proposed End of Support in HTSJDK | Actual End of Support in HTSJDK
 ---- | ---- | ---- | ----
 6 | Feb 2013 | Aug 2013 | Oct 2015
 7 | Apr 2015 | Oct 2015 | Oct 2015
-8* | Mar 2017 | Sep 2017 | Sep 2017
+8 | Jul 2018 | Jul 2018 | TBD
 
-* to be finalized
 
 HTSJDK is migrating to semantic versioning (http://semver.org/). We will eventually adhere to it strictly and bump our major version whenever there are breaking changes to our API, but until we more clearly define what constitutes our official API, clients should assume that every release potentially contains at least minor changes to public methods.
 
diff --git a/build.gradle b/build.gradle
@@ -5,35 +5,31 @@ buildscript {
 }
 
 plugins {
-    id "java"
+    id 'java'
+    id 'scala'
     id 'maven'
     id 'signing'
     id 'jacoco'
     id 'com.palantir.git-version' version '0.5.1'
     id 'com.github.johnrengelman.shadow' version '1.2.3'
-    id "com.github.kt3k.coveralls" version "2.6.3"
+    id 'com.github.maiflai.scalatest' version '0.15'
 }
 
 repositories {
     mavenCentral()
 }
 
 jacocoTestReport {
-    dependsOn test
     group = "Reporting"
     description = "Generate Jacoco coverage reports after running tests."
     additionalSourceDirs = files(sourceSets.main.allJava.srcDirs)
 
     reports {
-        xml.enabled = true // coveralls plugin depends on xml format report
+        xml.enabled = true // codecov depends on xml format report
         html.enabled = true
     }
 }
 
-jacoco {
-    toolVersion = "0.7.5.201505241946"
-}
-
 dependencies {
     compile "org.apache.commons:commons-jexl:2.1.1"
     compile "commons-logging:commons-logging:1.1.1"
@@ -42,7 +38,11 @@ dependencies {
     compile "org.tukaani:xz:1.5"
     compile "gov.nih.nlm.ncbi:ngs-java:1.2.4"
 
+    testCompile "org.scala-lang:scala-library:2.12.1"
+    testCompile "org.scalatest:scalatest_2.12:3.0.1"
+    testRuntime 'org.pegdown:pegdown:1.4.2' // Necessary for generating HTML reports with ScalaTest
     testCompile "org.testng:testng:6.9.9"
+    testCompile "com.google.jimfs:jimfs:1.1"
 }
 
 sourceCompatibility = 1.8
@@ -67,76 +67,57 @@ jar {
 
 import org.gradle.internal.os.OperatingSystem;
 
-tasks.withType(Test) {
-    outputs.upToDateWhen { false } // tests will always rerun
-    useTestNG()
+tasks.withType(Test) { task ->
+    task.outputs.upToDateWhen { false } // tests will always rerun
 
-    // set heap size for the test JVM(s)
-    minHeapSize = "1G"
-    maxHeapSize = "2G"
+    // Always run serially because there are some very badly behaved tests in HTSJDK that
+    // will cause errors and even deadlocks if run multi-threaded
+    task.maxParallelForks = 1
 
-    jvmArgs '-Djava.awt.headless=true'  //this prevents awt from displaying a java icon while the tests are running
+    // set heap size for the test JVM(s)
+    task.minHeapSize = "1G"
+    task.maxHeapSize = "2G"
 
-    if (System.env.CI == "true") {  //if running under a CI output less into the logs
-        int count = 0
+    task.jvmArgs '-Djava.awt.headless=true'  //this prevents awt from displaying a java icon while the tests are running
+}
 
-        beforeTest { descriptor ->
-            count++
-            if( count % 100 == 0) {
-                logger.lifecycle("Finished "+ Integer.toString(count++) + " tests")
-            }
-        }
-    } else {
-        // show standard out and standard error of the test JVM(s) on the console
-        testLogging.showStandardStreams = true
-        beforeTest { descriptor ->
-            logger.lifecycle("Running Test: " + descriptor)
-        }
+task findScalaAndJavaTypes(type: Exec) {
+    description = "Check that Scala files only exist in the scala test dir and that java files do not reside in the scala test dir."
+    commandLine './scripts/checkScalaAndJavaFiles.sh'
+}
 
-        // listen to standard out and standard error of the test JVM(s)
-        onOutput { descriptor, event ->
-            logger.lifecycle("Test: " + descriptor + " produced standard out/err: " + event.message )
-        }
-    }
+test {
+    description = "Runs the unit tests other than the SRA tests"
 
     testLogging {
-        testLogging {
-            events "skipped", "failed"
-            exceptionFormat = "full"
-        }
-        afterSuite { desc, result ->
-            if (!desc.parent) { // will match the outermost suite
-                println "Results: ${result.resultType} (${result.testCount} tests, ${result.successfulTestCount} successes, ${result.failedTestCount} failures, ${result.skippedTestCount} skipped)"
-            }
-        }
+        events "failed", "skipped"
     }
-}
 
-test {
-    description = "Runs the unit tests other than the SRA tests"
+    if (System.env.CI == "true") {
+        jvmArgs += '-Dsamjdk.sra_libraries_download=true'
+    }
 
-    useTestNG {
-        if( OperatingSystem.current().isUnix() ){
-            excludeGroups "slow", "broken", "sra"
-        } else {
-            excludeGroups "slow", "broken", "unix", "sra"
-        }
+    tags {
+        exclude "slow"
+        exclude "broken"
+        if (System.env.CI == "false") exclude "sra"
+        if (!OperatingSystem.current().isUnix()) exclude "unix"
     }
-}
+} dependsOn findScalaAndJavaTypes
 
 task testSRA(type: Test) {
-    jvmArgs '-Dsamjdk.sra_libraries_download=true'
+    description = "Run the SRA tests"
+    jvmArgs += '-Dsamjdk.sra_libraries_download=true'
 
-    description "Run the SRA tests"
-    useTestNG {
-        configFailurePolicy 'continue'
-        includeGroups "sra"
+    tags {
+        exclude "slow"
+        exclude "broken"
     }
 }
 
 task wrapper(type: Wrapper) {
     description = "Regenerate the gradle wrapper"
-    gradleVersion = '2.13'
+    gradleVersion = '3.2.1'
 }
 
 // This is a hack to disable the java 8 default javadoc lint until we fix the html formatting
@@ -188,7 +169,7 @@ uploadArchives {
                 authentication(userName: project.findProperty("sonatypeUsername"), password: project.findProperty("sonatypePassword"))
             }
 
-            snapshotRepository(url: "https://artifactory.broadinstitute.org/artifactory/libs-snapshot-local/") {
+            snapshotRepository(url: "https://broadinstitute.jfrog.io/broadinstitute/libs-snapshot-local/") {
                 authentication(userName: System.env.ARTIFACTORY_USERNAME, password: System.env.ARTIFACTORY_PASSWORD)
             }
 
diff --git a/gradle/wrapper/gradle-wrapper.jar b/gradle/wrapper/gradle-wrapper.jar
diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties
@@ -1,6 +1,6 @@
-#Fri May 13 14:00:35 EDT 2016
+#Fri Jan 20 17:10:11 EST 2017
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-2.13-bin.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-3.2.1-all.zip
diff --git a/gradlew b/gradlew
@@ -161,4 +161,9 @@ function splitJvmOpts() {
 eval splitJvmOpts $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS
 JVM_OPTS[${#JVM_OPTS[*]}]="-Dorg.gradle.appname=$APP_BASE_NAME"
 
+# by default we should be in the correct project dir, but when run from Finder on Mac, the cwd is wrong
+if [[ "$(uname)" == "Darwin" ]] && [[ "$HOME" == "$PWD" ]]; then
+  cd "$(dirname "$0")"
+fi
+
 exec "$JAVACMD" "${JVM_OPTS[@]}" -classpath "$CLASSPATH" org.gradle.wrapper.GradleWrapperMain "$@"
diff --git a/scripts/checkScalaAndJavaFiles.sh b/scripts/checkScalaAndJavaFiles.sh
@@ -0,0 +1,17 @@
+#/bin/bash
+
+# Check that Scala files only exist in the scala test dir and
+# that java files do not reside in the scala test dir
+
+if `find src | grep -v '^src/test/scala' | grep -q '\.scala$' ` ; then
+	echo 'Found scala file(s) outside of scala test directory';
+	find src | grep -v '^src/test/scala' | grep '\.scala$'
+	exit 1; 
+fi
+
+if `find src/test/scala | grep -q '\.java$' ` ; then
+        echo 'Found java file(s) in scala test directory';
+	find src/test/scala | grep '\.java$'        
+	exit 1;
+fi
+