Resolved Lucene security vulnerability patch

Resolved spring core security vulnerability patch Bumping lucene forced us to implicitly support long return values. However the Lucene search predicated don't all support long per-page values and some of our pagination implementations need to be able to handle all results in one page. Since we don't anticipate larger than Integer.MAX_VALUE documents from lucene, we're simply going to validate that the returning hit count is less than that and throw an exception if it isn't. (+1 squashed commit)
yahoo · Nov 1, 2018 · 18be7ca · 18be7ca
1 parent 3a60cee
commit 18be7ca
Show file tree

Hide file tree

Showing 3 changed files with 25 additions and 3 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -50,6 +50,14 @@ Current
     * Fixed bad format in error message
     * Moved tests off of serialization of `SimplifiedIntervalList`.  That's turning out to be hard to solve.
 
+- [Bump lucene version to patch vulnerability](https://github.com/yahoo/fili/issues/819)
+    * Bumped dependency version to 7.5.0 
+    * Added error in case of greater than maxint hits from Lucene 
+
+- [Bump spring code to patch vulnerability](https://github.com/yahoo/fili/issues/820)
+    * Bumped dependency version to [5.1.2,) 
+    * Throw validation error if excessive documents are returned from Lucene now that it supports up to long hitcounts 
+
 ### Known Issues:
 
 ### Added:

diff --git a/...ore/src/main/java/com/yahoo/bard/webservice/data/dimension/impl/LuceneSearchProvider.java b/...ore/src/main/java/com/yahoo/bard/webservice/data/dimension/impl/LuceneSearchProvider.java
@@ -88,6 +88,8 @@ public class LuceneSearchProvider implements SearchProvider {
             SYSTEM_CONFIG.getPackageVariableName("lucene_search_timeout_ms"), 600000
     );
 
+    public static String TOO_MANY_DOCUMENTS = "Unexpectedly large response from search provider.  Found %l hits.";
+
     /**
      * The maximum number of results per page.
      */
@@ -700,7 +702,17 @@ private Pagination<DimensionRow> getResultsPage(Query query, PaginationParameter
                         perPage
                 );
                 hits = hitDocs.scoreDocs;
-                documentCount = hitDocs.totalHits;
+                // The change to supprt long document sizes is incompletely supported in Lucene
+                // Since we can't request up to long documents we'll only expect to receive up to Integer.MAX_VALUE
+                // responses, and throw an error if we exceed that.
+                if (hitDocs.totalHits > Integer.MAX_VALUE) {
+                    String message = String.format(TOO_MANY_DOCUMENTS, hitDocs.totalHits);
+                    RowLimitReachedException exception = new RowLimitReachedException(message);
+                    LOG.error(exception.getMessage(), exception);
+                    throw exception;
+                }
+                documentCount = (int) hitDocs.totalHits;
+
                 int requestedPageNumber = paginationParameters.getPage(documentCount);
                 if (hits.length == 0) {
                     if (requestedPageNumber == 1) {

diff --git a/pom.xml b/pom.xml
@@ -78,7 +78,9 @@
         <version.hk2>2.5.0-b36</version.hk2>
         <version.metrics>3.2.2</version.metrics>
         <version.logback>1.2.3</version.logback>
-        <version.lucene>6.6.0</version.lucene>
+        <version.lucene>7.5.0</version.lucene>
+
+        <version.spring>[5.1.2,)</version.spring>
         <version.groovy>2.4.15</version.groovy>
         <version.jackson>2.9.5</version.jackson>
 
@@ -195,7 +197,7 @@
             <dependency>
                 <groupId>org.springframework</groupId>
                 <artifactId>spring-core</artifactId>
-                <version>4.3.4.RELEASE</version>
+                <version>${version.spring}</version>
             </dependency>
 
             <!-- Test -->