[lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED

A `lit` condition line is now a comma-separated list of boolean expressions. 
Comma-separated expressions act as if each expression were on its own 
condition line:
For REQUIRES, if every expression is true then the test will run. 
For UNSUPPORTED, if every expression is false then the test will run. 
For XFAIL, if every expression is false then the test is expected to succeed. 
As a special case "XFAIL: *" expects the test to fail.

Examples:
# Test is expected fail on 64-bit Apple simulators and pass everywhere else
XFAIL: x86_64 && apple && !macosx
# Test is unsupported on Windows and on non-Ubuntu Linux 
# and supported everywhere else
UNSUPPORTED: linux && !ubuntu, system-windows

Syntax: 
* '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false.
* Each test feature is a true identifier. 
* Substrings of the target triple are true identifiers for UNSUPPORTED 
 and XFAIL, but not for REQUIRES. (This matches the current behavior.)
* All other identifiers are false.
* Identifiers are [-+=._a-zA-Z0-9]+

Differential Revision: https://reviews.llvm.org/D18185

llvm-svn: 292904

llvm/docs/TestingGuide.rst

@@ -387,23 +387,49 @@ depends on special features of sub-architectures, you must add the specific
 triple, test with the specific FileCheck and put it into the specific
 directory that will filter out all other architectures.
 
-REQUIRES and REQUIRES-ANY directive
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Some tests can be enabled only in specific situation - like having
-debug build. Use ``REQUIRES`` directive to specify those requirements.
+Constraining test execution
+---------------------------
+
+Some tests can be run only in specific configurations, such as
+with debug builds or on particular platforms. Use ``REQUIRES``
+and ``UNSUPPORTED`` to control when the test is enabled.
+
+Some tests are expected to fail. For example, there may be a known bug
+that the test detect. Use ``XFAIL`` to mark a test as an expected failure.
+An ``XFAIL`` test will be successful if its execution fails, and
+will be a failure if its execution succeeds.
 
 .. code-block:: llvm
 
-    ; This test will be only enabled in the build with asserts
+    ; This test will be only enabled in the build with asserts.
     ; REQUIRES: asserts
+    ; This test is disabled on Linux.
+    ; UNSUPPORTED: -linux-
+    ; This test is expected to fail on PowerPC.
+    ; XFAIL: powerpc
+
+``REQUIRES`` and ``UNSUPPORTED`` and ``XFAIL`` all accept a comma-separated
+list of boolean expressions. The values in each expression may be:
 
-You can separate requirements by a comma.
-``REQUIRES`` means all listed requirements must be satisfied.
-``REQUIRES-ANY`` means at least one must be satisfied.
+- Features added to ``config.available_features`` by 
+  configuration files such as ``lit.cfg``.
+- Substrings of the target triple (``UNSUPPORTED`` and ``XFAIL`` only).
+
+| ``REQUIRES`` enables the test if all expressions are true.
+| ``UNSUPPORTED`` disables the test if any expression is true.
+| ``XFAIL`` expects the test to fail if any expression is true.
+
+As a special case, ``XFAIL: *`` is expected to fail everywhere.
+
+.. code-block:: llvm
+
+    ; This test is disabled on Windows,
+    ; and is disabled on Linux, except for Android Linux.
+    ; UNSUPPORTED: windows, linux && !android
+    ; This test is expected to fail on both PowerPC and ARM.
+    ; XFAIL: powerpc || arm
 
-List of features that can be used in ``REQUIRES`` and ``REQUIRES-ANY`` can be
-found in lit.cfg files.
 
 Substitutions
 -------------
@@ -520,24 +546,6 @@ their name. For example:
    This program runs its arguments and then inverts the result code from it.
    Zero result codes become 1. Non-zero result codes become 0.
 
-Sometimes it is necessary to mark a test case as "expected fail" or
-XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:``
-on a line near the top of the file. This signals that the test case
-should succeed if the test fails. Such test cases are counted separately
-by the testing tool. To specify an expected fail, use the XFAIL keyword
-in the comments of the test program followed by a colon and one or more
-failure patterns. Each failure pattern can be either ``*`` (to specify
-fail everywhere), or a part of a target triple (indicating the test
-should fail on that platform), or the name of a configurable feature
-(for example, ``loadable_module``). If there is a match, the test is
-expected to fail. If not, the test is expected to succeed. To XFAIL
-everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL``
-line:
-
-.. code-block:: llvm
-
-    ; XFAIL: darwin,sun
-
 To make the output more useful, :program:`lit` will scan
 the lines of the test case for ones that contain a pattern that matches
 ``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number

llvm/utils/lit/lit/BooleanExpression.py

@@ -0,0 +1,251 @@
+import re
+
+class BooleanExpression:
+    # A simple evaluator of boolean expressions.
+    #
+    # Grammar:
+    #   expr       :: or_expr
+    #   or_expr    :: and_expr ('||' and_expr)*
+    #   and_expr   :: not_expr ('&&' not_expr)*
+    #   not_expr   :: '!' not_expr
+    #                 '(' or_expr ')'
+    #                 identifier
+    #   identifier :: [-+=._a-zA-Z0-9]+
+
+    # Evaluates `string` as a boolean expression.
+    # Returns True or False. Throws a ValueError on syntax error.
+    #
+    # Variables in `variables` are true.
+    # Substrings of `triple` are true.
+    # 'true' is true.
+    # All other identifiers are false.
+    @staticmethod
+    def evaluate(string, variables, triple=""):
+        try:
+            parser = BooleanExpression(string, set(variables), triple)
+            return parser.parseAll()
+        except ValueError as e:
+            raise ValueError(str(e) + ('\nin expression: %r' % string))
+
+    #####
+
+    def __init__(self, string, variables, triple=""):
+        self.tokens = BooleanExpression.tokenize(string)
+        self.variables = variables
+        self.variables.add('true')
+        self.triple = triple
+        self.value = None
+        self.token = None
+
+    # Singleton end-of-expression marker.
+    END = object()
+
+    # Tokenization pattern.
+    Pattern = re.compile(r'\A\s*([()]|[-+=._a-zA-Z0-9]+|&&|\|\||!)\s*(.*)\Z')
+
+    @staticmethod
+    def tokenize(string):
+        while True:
+            m = re.match(BooleanExpression.Pattern, string)
+            if m is None:
+                if string == "":
+                    yield BooleanExpression.END;
+                    return
+                else:
+                    raise ValueError("couldn't parse text: %r" % string)
+
+            token = m.group(1)
+            string = m.group(2)
+            yield token
+
+    def quote(self, token):
+        if token is BooleanExpression.END:
+            return '<end of expression>'
+        else:
+            return repr(token)
+
+    def accept(self, t):
+        if self.token == t:
+            self.token = next(self.tokens)
+            return True
+        else:
+            return False
+
+    def expect(self, t):
+        if self.token == t:
+            if self.token != BooleanExpression.END:
+                self.token = next(self.tokens)
+        else:
+            raise ValueError("expected: %s\nhave: %s" %
+                             (self.quote(t), self.quote(self.token)))
+
+    def isIdentifier(self, t):
+        if (t is BooleanExpression.END or t == '&&' or t == '||' or
+            t == '!' or t == '(' or t == ')'):
+            return False
+        return True
+
+    def parseNOT(self):
+        if self.accept('!'):
+            self.parseNOT()
+            self.value = not self.value
+        elif self.accept('('):
+            self.parseOR()
+            self.expect(')')
+        elif not self.isIdentifier(self.token):
+            raise ValueError("expected: '!' or '(' or identifier\nhave: %s" %
+                             self.quote(self.token))
+        else:
+            self.value = (self.token in self.variables or
+                          self.token in self.triple)
+            self.token = next(self.tokens)
+
+    def parseAND(self):
+        self.parseNOT()
+        while self.accept('&&'):
+            left = self.value
+            self.parseNOT()
+            right = self.value
+            # this is technically the wrong associativity, but it
+            # doesn't matter for this limited expression grammar
+            self.value = left and right
+
+    def parseOR(self):
+        self.parseAND()
+        while self.accept('||'):
+            left = self.value
+            self.parseAND()
+            right = self.value
+            # this is technically the wrong associativity, but it
+            # doesn't matter for this limited expression grammar
+            self.value = left or right
+
+    def parseAll(self):
+        self.token = next(self.tokens)
+        self.parseOR()
+        self.expect(BooleanExpression.END)
+        return self.value
+
+
+#######
+# Tests
+
+import unittest
+
+class TestBooleanExpression(unittest.TestCase):
+    def test_variables(self):
+        variables = {'its-true', 'false-lol-true', 'under_score',
+                     'e=quals', 'd1g1ts'}
+        self.assertTrue(BooleanExpression.evaluate('true', variables))
+        self.assertTrue(BooleanExpression.evaluate('its-true', variables))
+        self.assertTrue(BooleanExpression.evaluate('false-lol-true', variables))
+        self.assertTrue(BooleanExpression.evaluate('under_score', variables))
+        self.assertTrue(BooleanExpression.evaluate('e=quals', variables))
+        self.assertTrue(BooleanExpression.evaluate('d1g1ts', variables))
+
+        self.assertFalse(BooleanExpression.evaluate('false', variables))
+        self.assertFalse(BooleanExpression.evaluate('True', variables))
+        self.assertFalse(BooleanExpression.evaluate('true-ish', variables))
+        self.assertFalse(BooleanExpression.evaluate('not_true', variables))
+        self.assertFalse(BooleanExpression.evaluate('tru', variables))
+
+    def test_triple(self):
+        triple = 'arch-vendor-os'
+        self.assertTrue(BooleanExpression.evaluate('arch-', {}, triple))
+        self.assertTrue(BooleanExpression.evaluate('ar', {}, triple))
+        self.assertTrue(BooleanExpression.evaluate('ch-vend', {}, triple))
+        self.assertTrue(BooleanExpression.evaluate('-vendor-', {}, triple))
+        self.assertTrue(BooleanExpression.evaluate('-os', {}, triple))
+        self.assertFalse(BooleanExpression.evaluate('arch-os', {}, triple))
+
+    def test_operators(self):
+        self.assertTrue(BooleanExpression.evaluate('true || true', {}))
+        self.assertTrue(BooleanExpression.evaluate('true || false', {}))
+        self.assertTrue(BooleanExpression.evaluate('false || true', {}))
+        self.assertFalse(BooleanExpression.evaluate('false || false', {}))
+
+        self.assertTrue(BooleanExpression.evaluate('true && true', {}))
+        self.assertFalse(BooleanExpression.evaluate('true && false', {}))
+        self.assertFalse(BooleanExpression.evaluate('false && true', {}))
+        self.assertFalse(BooleanExpression.evaluate('false && false', {}))
+
+        self.assertFalse(BooleanExpression.evaluate('!true', {}))
+        self.assertTrue(BooleanExpression.evaluate('!false', {}))
+
+        self.assertTrue(BooleanExpression.evaluate('   ((!((false) ))   ) ', {}))
+        self.assertTrue(BooleanExpression.evaluate('true && (true && (true))', {}))
+        self.assertTrue(BooleanExpression.evaluate('!false && !false && !! !false', {}))
+        self.assertTrue(BooleanExpression.evaluate('false && false || true', {}))
+        self.assertTrue(BooleanExpression.evaluate('(false && false) || true', {}))
+        self.assertFalse(BooleanExpression.evaluate('false && (false || true)', {}))
+
+    # Evaluate boolean expression `expr`.
+    # Fail if it does not throw a ValueError containing the text `error`.
+    def checkException(self, expr, error):
+        try:
+            BooleanExpression.evaluate(expr, {})
+            self.fail("expression %r didn't cause an exception" % expr)
+        except ValueError as e:
+            if -1 == str(e).find(error):
+                self.fail(("expression %r caused the wrong ValueError\n" +
+                           "actual error was:\n%s\n" +
+                           "expected error was:\n%s\n") % (expr, e, error))
+        except BaseException as e:
+            self.fail(("expression %r caused the wrong exception; actual " + 
+                      "exception was: \n%r") % (expr, e))
+
+    def test_errors(self):
+        self.checkException("ba#d",
+                            "couldn't parse text: '#d'\n" +
+                            "in expression: 'ba#d'")
+
+        self.checkException("true and true",
+                            "expected: <end of expression>\n" +
+                            "have: 'and'\n" +
+                            "in expression: 'true and true'")
+
+        self.checkException("|| true",
+                            "expected: '!' or '(' or identifier\n" +
+                            "have: '||'\n" +
+                            "in expression: '|| true'")
+
+        self.checkException("true &&",
+                            "expected: '!' or '(' or identifier\n" +
+                            "have: <end of expression>\n" +
+                            "in expression: 'true &&'")
+
+        self.checkException("",
+                            "expected: '!' or '(' or identifier\n" +
+                            "have: <end of expression>\n" +
+                            "in expression: ''")
+
+        self.checkException("*",
+                            "couldn't parse text: '*'\n" +
+                            "in expression: '*'")
+
+        self.checkException("no wait stop",
+                            "expected: <end of expression>\n" +
+                            "have: 'wait'\n" +
+                            "in expression: 'no wait stop'")
+
+        self.checkException("no-$-please",
+                            "couldn't parse text: '$-please'\n" +
+                            "in expression: 'no-$-please'")
+
+        self.checkException("(((true && true) || true)",
+                            "expected: ')'\n" +
+                            "have: <end of expression>\n" +
+                            "in expression: '(((true && true) || true)'")
+
+        self.checkException("true (true)",
+                            "expected: <end of expression>\n" +
+                            "have: '('\n" +
+                            "in expression: 'true (true)'")
+
+        self.checkException("( )",
+                            "expected: '!' or '(' or identifier\n" +
+                            "have: ')'\n" +
+                            "in expression: '( )'")
+
+if __name__ == '__main__':
+    unittest.main()