Sanity check for compatibility and acceptance blacklists

If a blacklisted test cannot be found in the code an error will be raised

enterprise/cypher/acceptance-spec-suite/src/test/java/cypher/AcceptanceSpecSuiteTest.java

@@ -21,15 +21,20 @@
 
 import cucumber.api.CucumberOptions;
 import cucumber.api.junit.Cucumber;
+import cypher.cucumber.BlacklistPlugin;
+import org.junit.AfterClass;
 import org.junit.experimental.runners.Enclosed;
 import org.junit.runner.RunWith;
 
+import java.util.Set;
+
 import static cypher.SpecSuiteConstants.BLACKLIST_PLUGIN;
 import static cypher.SpecSuiteConstants.CYPHER_OPTION_PLUGIN;
 import static cypher.SpecSuiteConstants.DB_CONFIG;
 import static cypher.SpecSuiteConstants.GLUE_PATH;
 import static cypher.SpecSuiteConstants.HTML_REPORT;
 import static cypher.SpecSuiteConstants.JSON_REPORT;
+import static junit.framework.TestCase.fail;
 
 @RunWith( Enclosed.class )
 public class AcceptanceSpecSuiteTest
@@ -46,6 +51,20 @@ private AcceptanceSpecSuiteTest()
     {
     }
 
+    public static abstract class Base
+    {
+        @AfterClass
+        public static void teardown()
+        {
+            Set<String> diff = BlacklistPlugin.getDiffBetweenBlacklistAndUsedScenarios();
+            if ( !diff.isEmpty() )
+            {
+                fail( "The following scenarios were blacklisted but no test corresponds to that name:\n"
+                        + String.join( "\n", diff ) );
+            }
+        }
+    }
+
     @RunWith( Cucumber.class )
     @CucumberOptions(
             plugin = {
@@ -59,7 +78,7 @@ private AcceptanceSpecSuiteTest()
             tags = { "~@pending" },
             strict = true
     )
-    public static class Default
+    public static class Default extends Base
     {
     }
 
@@ -76,7 +95,7 @@ public static class Default
             tags = { "~@pending" },
             strict = true
     )
-    public static class Cost
+    public static class Cost extends Base
     {
     }
 
@@ -93,7 +112,7 @@ public static class Cost
             tags = { "~@pending" },
             strict = true
     )
-    public static class CostCompiled
+    public static class CostCompiled extends Base
     {
     }
 
@@ -111,7 +130,7 @@ public static class CostCompiled
             tags = { "~@pending" },
             strict = true
     )
-    public static class CostCompiledSourceCode
+    public static class CostCompiledSourceCode extends Base
     {
     }
 
@@ -128,7 +147,7 @@ public static class CostCompiledSourceCode
             tags = { "~@pending" },
             strict = true
     )
-    public static class CostSlotted
+    public static class CostSlotted extends Base
     {
     }
 
@@ -145,7 +164,7 @@ public static class CostSlotted
             tags = { "~@pending" },
             strict = true
     )
-    public static class Compatibility23
+    public static class Compatibility23 extends Base
     {
     }
 
@@ -162,7 +181,7 @@ public static class Compatibility23
             tags = { "~@pending" },
             strict = true
     )
-    public static class Compatibility31
+    public static class Compatibility31 extends Base
     {
     }
 
@@ -179,7 +198,7 @@ public static class Compatibility31
             tags = { "~@pending" },
             strict = true
     )
-    public static class Compatibility32
+    public static class Compatibility32 extends Base
     {
     }
 

enterprise/cypher/acceptance-spec-suite/src/test/resources/blacklists/compatibility-23.txt

@@ -12,18 +12,12 @@ Standalone call to VOID procedure that takes no arguments
   In-query call to VOID procedure that takes no arguments
   In-query call to VOID procedure does not consume rows
 Standalone call to VOID procedure that takes no arguments, called with implicit arguments
-Standalone call to procedure that takes no arguments and yields no results
-  In-query call to procedure that takes no arguments and yields no results
-Standalone call to procedure that takes no arguments and yields no results, called with implicit arguments
 Standalone call to procedure with explicit arguments
   In-query call to procedure with explicit arguments
-Standalone call to procedure with implicit arguments
 Standalone call to procedure with argument of type NUMBER  accepts value of type INTEGER
   In-query call to procedure with argument of type NUMBER  accepts value of type INTEGER
 Standalone call to procedure with argument of type NUMBER  accepts value of type FLOAT
   In-query call to procedure with argument of type NUMBER  accepts value of type FLOAT
-Standalone call to procedure with argument of type INTEGER accepts value of type FLOAT
-  In-query call to procedure with argument of type INTEGER accepts value of type FLOAT
 Standalone call to procedure with argument of type FLOAT   accepts value of type INTEGER
   In-query call to procedure with argument of type FLOAT   accepts value of type INTEGER
 Standalone call to procedure with null argument

enterprise/cypher/acceptance-spec-suite/src/test/resources/blacklists/cost-compiled.txt

@@ -126,9 +126,6 @@ Find relationships of a shortest path
 Find no shortest path when a length limit prunes all candidates
 Find no shortest path when the start node is null
 Find all shortest paths
-Find a shortest path among paths of length 1
-Find a shortest path of length 1 if requested to
-Find a shortest path among paths matched using a non-variable length pattern
 Find a combination of a shortest path and a pattern expression
 Filter with AND/OR
 LIMIT 0 should stop side effects

enterprise/cypher/acceptance-spec-suite/src/test/resources/blacklists/cost-slotted.txt

@@ -90,9 +90,6 @@ Find relationships of a shortest path
 Find no shortest path when a length limit prunes all candidates
 Find no shortest path when the start node is null
 Find all shortest paths
-Find a shortest path among paths of length 1
-Find a shortest path of length 1 if requested to
-Find a shortest path among paths matched using a non-variable length pattern
 Find a combination of a shortest path and a pattern expression
 Filter with AND/OR
 

enterprise/cypher/compatibility-spec-suite/src/test/java/cypher/CompatibilitySpecSuiteTest.java

@@ -21,15 +21,20 @@
 
 import cucumber.api.CucumberOptions;
 import cucumber.api.junit.Cucumber;
+import cypher.cucumber.BlacklistPlugin;
+import org.junit.AfterClass;
 import org.junit.runner.RunWith;
 import org.opencypher.tools.tck.TCKCucumberTemplate;
 
+import java.util.Set;
+
 import static cypher.SpecSuiteConstants.BLACKLIST_PLUGIN;
 import static cypher.SpecSuiteConstants.CYPHER_OPTION_PLUGIN;
 import static cypher.SpecSuiteConstants.DB_CONFIG;
 import static cypher.SpecSuiteConstants.GLUE_PATH;
 import static cypher.SpecSuiteConstants.HTML_REPORT;
 import static cypher.SpecSuiteConstants.JSON_REPORT;
+import static junit.framework.TestCase.fail;
 
 @RunWith( CompatibilitySpecSuiteResources.class )
 public class CompatibilitySpecSuiteTest
@@ -46,6 +51,19 @@ private CompatibilitySpecSuiteTest()
     {
     }
 
+    public static abstract class Base
+    {
+        @AfterClass
+        public static void teardown()
+        {
+            Set<String> diff = BlacklistPlugin.getDiffBetweenBlacklistAndUsedScenarios();
+            if ( !diff.isEmpty() )
+            {
+                fail( "The following scenarios were blacklisted but no test corresponds to that name:\n" + String.join( "\n", diff ) );
+            }
+        }
+    }
+
     @RunWith( Cucumber.class )
     @CucumberOptions(
             plugin = {
@@ -59,7 +77,7 @@ private CompatibilitySpecSuiteTest()
             tags = { "~@pending" },
             strict = true
     )
-    public static class Default
+    public static class Default extends Base
     {
     }
 
@@ -76,7 +94,7 @@ public static class Default
             tags = { "~@pending" },
             strict = true
     )
-    public static class Cost
+    public static class Cost extends Base
     {
     }
 
@@ -93,7 +111,7 @@ public static class Cost
             tags = { "~@pending" },
             strict = true
     )
-    public static class CostCompiled
+    public static class CostCompiled extends Base
     {
     }
 
@@ -111,7 +129,7 @@ public static class CostCompiled
             tags = { "~@pending" },
             strict = true
     )
-    public static class CostCompiledSourceCode
+    public static class CostCompiledSourceCode extends Base
     {
     }
 
@@ -128,7 +146,7 @@ public static class CostCompiledSourceCode
             tags = { "~@pending" },
             strict = true
     )
-    public static class CostInterpretedEnterprise
+    public static class CostInterpretedEnterprise extends Base
     {
     }
 
@@ -145,7 +163,7 @@ public static class CostInterpretedEnterprise
             tags = { "~@pending" },
             strict = true
     )
-    public static class Compatibility23
+    public static class Compatibility23 extends Base
     {
     }
 
@@ -162,7 +180,7 @@ public static class Compatibility23
             tags = { "~@pending" },
             strict = true
     )
-    public static class Compatibility31
+    public static class Compatibility31 extends Base
     {
     }
 

enterprise/cypher/compatibility-spec-suite/src/test/resources/blacklists/compatibility-23.txt

@@ -2,8 +2,6 @@
 Aggregates inside normal functions
 Aggregation of named paths
 Aggregation with `min()`
-Get node degree via length of pattern expression
-Get node degree via length of pattern expression that specifies a relationship type
 Return a var length path of length zero
 Remove multiple node properties
 Remove multiple relationship properties
@@ -73,11 +71,9 @@ Ignore null when setting properties using an appending map
 Ignore null when setting properties using an overriding map
 Ignore null when setting label
 Ignore null when removing label
-Pattern expression inside list comprehension
 Fail when returning properties of deleted nodes
 Fail when returning labels of deleted nodes
 Fail when returning properties of deleted relationships
-Fail when returning type of deleted relationships
 `toFloat()` on mixed number types
 `toString()` handling boolean properties
 `toString()` handling inlined boolean

enterprise/cypher/compatibility-spec-suite/src/test/resources/blacklists/cost-compiled.txt

@@ -123,8 +123,6 @@ Return a var length path of length zero
 Return a named var length path of length zero
 Do not fail when evaluating predicates with illegal operations if the AND'ed predicate evaluates to false
 Do not fail when evaluating predicates with illegal operations if the OR'd predicate evaluates to true
-Fail when comparing strings and integers using > in an AND'd predicate
-Fail when comparing strings and integers using > in a OR'd predicate
 Aggregation with named paths
 Zero-length variable length pattern in the middle of the pattern
 Simple variable length pattern
@@ -243,7 +241,6 @@ Fail when returning labels of deleted nodes
 Fail when returning properties of deleted relationships
 Do not fail when returning type of deleted relationships
 Fail when sorting on variable removed by DISTINCT
-Fail when ordering nodes
 Ordering with aggregation
 Return all variables
 Setting and returning the size of a list property
@@ -384,7 +381,6 @@ Handling mixed relationship patterns 1
 Handling mixed relationship patterns 2
 Handling relationships that are already bound in variable length paths
 NOT and false
-Fail when trying to compare strings and numbers
 Aliasing
 Handle dependencies across WITH
 Handle dependencies across WITH with SKIP

enterprise/cypher/compatibility-spec-suite/src/test/resources/blacklists/cost-slotted.txt

@@ -1,10 +1,8 @@
-Combining string operators' was blacklisted, but succeeded
 Directed match of a simple relationship, count
 Directed match on self-relationship graph, count
 Get rows in the middle by param
 Get rows in the middle
 
-handling long chanins of operators
 Handle ORDER BY with LIMIT 1
 Keeping used expression 1
 Matching and returning ordered results, with LIMIT
@@ -49,7 +47,6 @@ IS NOT NULL with literal maps
 `percentileDisc()` failing on bad arguments
 `percentileDisc()` failing in more involved query
 `type()` failing on invalid arguments
-`labels()` in return clauses
 Find friends of others
 Should only join when matching
 Adding a single label
@@ -73,8 +70,6 @@ Return relationships by fetching them from the path
 Return a var length path
 Return a var length path of length zero
 Return a named var length path of length zero
-Fail when comparing strings and integers using > in an AND'd predicate
-Fail when comparing strings and integers using > in a OR'd predicate
 Aggregation with named paths
 Variable length relationship without lower bound
 Variable length relationship without bounds
@@ -207,7 +202,6 @@ Handling mixed relationship patterns and directions 2
 Handling mixed relationship patterns 1
 Handling mixed relationship patterns 2
 Handling relationships that are already bound in variable length paths
-Fail when trying to compare strings and numbers
 A simple pattern with one bound endpoint
 Nested maps
 Connected components succeeding WITH

enterprise/cypher/spec-suite-tools/src/main/scala/cypher/cucumber/BlacklistPlugin.scala

@@ -22,15 +22,27 @@ package cypher.cucumber
 import java.io.FileNotFoundException
 import java.net.URI
 import java.nio.charset.StandardCharsets
+import java.util
 
 import gherkin.formatter.model.{Match, Result}
 
+import scala.collection.JavaConverters._
+import scala.collection.mutable
 import scala.io.Source
 
 object BlacklistPlugin {
   private var _blacklist: Set[String] = null
+  private val _usedScenarios: mutable.Set[String] = mutable.Set()
 
-  def blacklisted(name: String) = blacklist().contains(normalizedScenarioName(name))
+  def blacklisted(name: String) = {
+    val newName = normalizedScenarioName(name)
+    _usedScenarios.add(newName)
+    blacklist().contains(newName)
+  }
+
+  def getDiffBetweenBlacklistAndUsedScenarios(): util.Set[String] = {
+    blacklist().diff(_usedScenarios).asJava
+  }
 
   def normalizedScenarioName(name: String) = {
     val builder = new StringBuilder