Intermittent failure in GeoWaveBasicIT (0.9.0-RC1)

[dcy2003]

Travis Build: https://travis-ci.org/ngageoint/geowave/jobs/101144205
Raw Output: https://s3.amazonaws.com/archive.travis-ci.org/jobs/101144205/log.txt

Tests run: 20, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 951.121 sec <<< FAILURE! - in mil.nga.giat.geowave.test.GeoWaveITSuite
testIngestAndQuerySpatialPointsAndLines(mil.nga.giat.geowave.test.GeoWaveBasicIT)  Time elapsed: 74.456 sec  <<< FAILURE!
java.lang.AssertionError: expected:<11110> but was:<11109>
    at org.junit.Assert.fail(Assert.java:88)
    at org.junit.Assert.failNotEquals(Assert.java:743)
    at org.junit.Assert.assertEquals(Assert.java:118)
    at org.junit.Assert.assertEquals(Assert.java:555)
    at org.junit.Assert.assertEquals(Assert.java:542)
    at mil.nga.giat.geowave.test.GeoWaveBasicIT.testQuery(GeoWaveBasicIT.java:822)
    at mil.nga.giat.geowave.test.GeoWaveBasicIT.testIngestAndQuerySpatialPointsAndLines(GeoWaveBasicIT.java:143)


Results :

Failed tests: 
  GeoWaveBasicIT.testIngestAndQuerySpatialPointsAndLines:143->testQuery:822 expected:<11110> but was:<11109>

Tests run: 20, Failures: 1, Errors: 0, Skipped: 0

[dcy2003]

Additional information:

Subsequent executions of this integration test reveal erratic behavior. I have observed a mix of:

• Test Passes
• Test fails because spatial-query-with-box-filter expects 178 but returns 177 results
• Test fails because spatial-query-with-polygon-filter expects 11110 but returns 11109 results
• Test fails because spatial-temporal-query-with-polygon-filter expects 130 but returns 129 results

For example, for the spatial-query-with-box-filter, the failure case differs from the success case in that only the following result from the success case is missing from the failure case. All other results match one to one. This exact feature is always the missing feature for the spatial-query-with-box-filter failure case, however sometimes this case passes without issue.

featureId:

tornado_tracks.7583

geometry:

MULTILINESTRING ((-93.38 34.169999999999995, -93.02 34.45))

[datasedai]

I think this is caused by the use of Guava's Interner in SpatialQueryFilter.java, which is causing the JTS geometry 'intersects' method to return invalid values occasionally.

When a query is distributed by accumulo client to each TabletServer, multiple separate scans are initiated (I'm assuming for multiple ranges, but my accumulo experience is limited). In the case of this integration test, 8 queries are sent to each tablet server for a total of 16. These queries are executed in separate threads simultaneously. Each thread creates a spatial query filter, which is deserialized from the package sent from the client (via DistributableFilterList). The 'fromBinary()' method uses the Interner to save memory and performance by only creating the prepared geometry once, and sharing that object across threads. Unfortunately, though, it appears that PreparedGeometry is not threadsafe.

I created a sample test that sets up the same general circumstances as provided in SpatialQueryFilter and ran it with four threads. I received a failure rate of about 1 in every 5.14 calls to the intersects method, which is roughly the failure rate I experienced when debugging this issue manually. The test is attached.

[rwgdrummer]

Can you try ThreadLocal instead of the intern?

On Wed, Jan 20, 2016 at 5:05 PM, datasedai notifications@github.com wrote:

I think this is caused by the use of Guava's Interner in
SpatialQueryFilter.java, which is causing the JTS geometry 'intersects'
method to return invalid values occasionally.

When a query is distributed by accumulo client to each TabletServer,
multiple separate scans are initiated (I'm assuming for multiple ranges,
but my accumulo experience is limited). In the case of this integration
test, 8 queries are sent to each tablet server for a total of 16. These
queries are executed in separate threads simultaneously. Each thread
creates a spatial query filter, which is deserialized from the package sent
from the client (via DistributableFilterList). The 'fromBinary()' method
uses the Interner to save memory and performance by only creating the
prepared geometry once, and sharing that object across threads.
Unfortunately, though, it appears that PreparedGeometry is not threadsafe.

I created a sample test that sets up the same general circumstances as
provided in SpatialQueryFilter and ran it with four threads. I received a
failure rate of about 1 in every 5.14 calls to the intersects method, which
is roughly the failure rate I experienced when debugging this issue
manually. The test is attached
https://github.com/ngageoint/geowave/files/98162/FilterTest.java.txt.

—
Reply to this email directly or view it on GitHub
#564 (comment).

[dcy2003]

Fixed in 0.9.0