Intermittent core dump in native pair hmm

[cmnbroad]

GATK has an intermittent failure that results in a core dump, which I think we've tracked down to native pair HMM. We're seeing it quite frequently on travis lately. AFAIK we have only ever seen this on Java 11 (not 8).

More details are available in broadinstitute/gatk#6649. Attached is a travis log that has the core dump log embedded.

log.txt

[rpomaris]

@cmnbroad Thanks for reporting. I tagged @SnehalA and @Kmannth on the GATK issue.

[Kmannth]

@cmnbroad

The team did a Java 11 test with GKL Master/v0.8.8.

compiled GATK with Java 11 with the Master GKL from our repository and then ran the following tests over the weekend:

• IntelInflaterDeflaterIntegrationTest
• HaplotypeUnitTest
• SmithWatermanIntelAlignerUnitTest
• VectorPairHMMUnitTest
  These 4 tests have executed upwards of 20,000 times and not a single crash was encountered. It's worth noting that the last test (VectorPairHMMUnitTest) was meant to be disabled but didn't actually skip the test due to an incorrect Java version check.

We have been unable to reproduce this with latest code Master/v0.8.8.

Can you think of any other thing we should try ?

[cmnbroad]

@kmmanth I'm not sure what you mean by:

It's worth noting that the last test (VectorPairHMMUnitTest) was meant to be disabled but didn't actually skip the test due to an incorrect Java version check.

Are you saying the code that disables the test is incorrect ? Can you provide more detail ?

[Kmannth]

@cmnbroad
In the current code there is check to skip VectorPairHMMUnitTest with a java version check. On our Centos 7.x dev system the way Java was being check did not work so it ran anyway. I don't know exactly why the Java version check might work on your system but not ours but it is something we noticed.

[Kmannth]

We believe this is fixed by the latest version. Please reopen if you hit it again.