diff --git a/docs/01-what-is-filecheck.rst b/docs/01-what-is-filecheck.rst index a54c9fb..aa0682a 100644 --- a/docs/01-what-is-filecheck.rst +++ b/docs/01-what-is-filecheck.rst @@ -1,7 +1,7 @@ What is FileCheck.py ==================== -**FileCheck.py** is a Python port of the **LLVM's FileCheck**, "flexible +**FileCheck.py** is a Python port of **LLVM's FileCheck**, "flexible pattern matching file verifier" [1_]. LLVM's FileCheck is a command-line tool written in C++ which @@ -15,118 +15,27 @@ of the commands run by LIT. Why Python port? ---------------- -There are software projects that would benefit from having a suite of LLVM LIT +There are software projects that would benefit from having a suite of LIT-based integration tests. Mull mutation testing system is one example [4_]. -The problem is that you have to build FileCheck from LLVM sources which is not a trivial task for 1) people who are not aware with the LLVM infrastructure and 2) Python-based projects that would prefer to not have to build anything from LLVM sources in their CI process. +The problem is that you have to build FileCheck from LLVM sources which is not a trivial task for 1) people who are not familiar with the LLVM infrastructure and +2) Python-based projects which would prefer to not depend on anything C or +C++-related including building dependencies from LLVM sources. The option of having pre-compiled binaries is a workaround, but it is not always -possible to keep third-party binary artifacts in source code, -(see https://github.com/doorstop-dev/doorstop/pull/431#issuecomment-549237579). +possible to keep third-party binary artifacts in source code +(see +https://github.com/doorstop-dev/doorstop/pull/431#issuecomment-549237579). -Simple example --------------- +What's next? +------------ -When writing LIT/FileCheck tests it is common to combine the LIT's ``RUN`` -commands and FileCheck's ``CHECK`` assertions in a single file. +If you are new to FileCheck and integration testing using LIT, we recommend you +to read the tutorials: :doc:`03-tutorial-hello-world` and +:doc:`04-tutorial-lit-and-filecheck`. -Passing test -~~~~~~~~~~~~ - -Test file ``01-pass.c``: - -.. code-block:: c - - /** - ; RUN: clang %s -o %S/hello-world && %S/hello-world | filecheck %s - ; CHECK: Hello world - */ - - #include - int main() { - printf("Hello world\n"); - return 0; - } - -Command: - -.. code-block:: bash - - $ lit 01-pass.c - -Output: - -.. code-block:: text - - -- Testing: 1 tests, single process -- - PASS: :: 01-pass.c (1 of 1) - Testing Time: 0.10s - Expected Passes : 1 - -Failing test -~~~~~~~~~~~~ - -Test file ``02-fail.c``: - -.. code-block:: c - - /** - ; RUN: clang %s -o %S/hello-world && %S/hello-world | filecheck %s - ; CHECK: Wrong line - */ - - #include - int main() { - printf("Hello world\n"); - return 0; - } - - -Command: - -.. code-block:: bash - - $ lit 02-fail.c - -Output: - -.. code-block:: text - - -- Testing: 1 tests, single process -- - FAIL: :: 02-fail.c (1 of 1) - Testing Time: 0.11s - ******************** - Failing Tests (1): - :: 02-fail.c - - Unexpected Failures: 1 - -The verbose version: - -.. code-block:: bash - - $ lit -v 02-fail.c - -Produces more output: - -.. code-block:: text - - -- Testing: 1 tests, single process -- - FAIL: :: 02-fail.c (1 of 1) - ******************** TEST ' :: 02-fail.c' FAILED ******************** - 02-fail.c:3:10: error: CHECK: expected string not found in input - ; CHECK: Wrong line - ^ - :1:1: note: scanning from here - Hello world - ... - ******************** - Testing Time: 0.11s - ******************** - Failing Tests (1): - :: 02-fail.c - - Unexpected Failures: 1 +If you know how FileCheck and LIT work, you can check out the status of the port +on the :doc:`07-roadmap` page. Links ----- diff --git a/docs/02-installation.rst b/docs/02-installation.rst index 890e5b3..8a41a6e 100644 --- a/docs/02-installation.rst +++ b/docs/02-installation.rst @@ -1,7 +1,14 @@ Installation ============ -... +FileCheck.py is a pip package: +.. code-block:: bash + pip install filecheck +LIT is also a pip package: + +.. code-block:: bash + + pip install lit diff --git a/docs/03-tutorial-hello-world.rst b/docs/03-tutorial-hello-world.rst index ad93972..f964995 100644 --- a/docs/03-tutorial-hello-world.rst +++ b/docs/03-tutorial-hello-world.rst @@ -98,11 +98,10 @@ With invalid input: $ echo $? 1 -What next? ----------- +What's next? +------------ `FileCheck` is rarely used alone. The main use case for `FileCheck` is to serve as an assertion matcher tool when it is used in a combination with the -`LLVM LIT Integrated Tester `_ and -this is what our next tutorial is about. Don't stop here and check it out right -away. +LLVM LIT Integrated Tester and this is what our next tutorial is about. Don't +stop here and check it out right away: :doc:`04-tutorial-lit-and-filecheck`. diff --git a/docs/04-tutorial-lit-and-filecheck.rst b/docs/04-tutorial-lit-and-filecheck.rst new file mode 100644 index 0000000..cf187b4 --- /dev/null +++ b/docs/04-tutorial-lit-and-filecheck.rst @@ -0,0 +1,152 @@ +Tutorial: LIT and FileCheck +=========================== + +This tutorial assumes that you have installed ``lit`` and ``filecheck`` and have +them available in your PATH: + +.. code-block:: bash + + $ filecheck + /usr/local/bin/filecheck + not specified + +.. code-block:: bash + + $ lit + ... + lit: error: No inputs specified + +Minimal example +--------------- + +When writing LIT/FileCheck tests it is common, but not required, to combine +LIT's ``RUN`` commands and FileCheck's ``CHECK`` assertions in a single file. + +Let's create a file ``minimal.itest`` with the following contents + +.. code-block:: text + + RUN: echo "Hello world" | filecheck %s + CHECK: Hello world + +LIT expects a config file in a directory from which it is run. Let's create +a minimal one called ``lit.cfg``: + +.. code-block:: text + + import lit.formats + config.test_format = lit.formats.ShTest("0") + +Now we can run ``lit``: + +.. code-block:: bash + + lit minimal.itest + +.. code-block:: bash + + -- Testing: 1 tests, single process -- + PASS: :: test.itest (1 of 1) + Testing Time: 0.10s + Expected Passes : 1 + +Example: testing output of C program +------------------------------------ + +Passing test +~~~~~~~~~~~~ + +Test file ``01-pass.c``: + +.. code-block:: c + + /** + ; RUN: clang %s -o %S/hello-world && %S/hello-world | filecheck %s + ; CHECK: Hello world + */ + + #include + int main() { + printf("Hello world\n"); + return 0; + } + +Command: + +.. code-block:: bash + + $ lit 01-pass.c + +Output: + +.. code-block:: text + + -- Testing: 1 tests, single process -- + PASS: :: 01-pass.c (1 of 1) + Testing Time: 0.10s + Expected Passes : 1 + +Failing test +~~~~~~~~~~~~ + +Test file ``02-fail.c``: + +.. code-block:: c + + /** + ; RUN: clang %s -o %S/hello-world && %S/hello-world | filecheck %s + ; CHECK: Wrong line + */ + + #include + int main() { + printf("Hello world\n"); + return 0; + } + + +Command: + +.. code-block:: bash + + $ lit 02-fail.c + +Output: + +.. code-block:: text + + -- Testing: 1 tests, single process -- + FAIL: :: 02-fail.c (1 of 1) + Testing Time: 0.11s + ******************** + Failing Tests (1): + :: 02-fail.c + + Unexpected Failures: 1 + +The verbose version: + +.. code-block:: bash + + $ lit -v 02-fail.c + +Produces more output: + +.. code-block:: text + + -- Testing: 1 tests, single process -- + FAIL: :: 02-fail.c (1 of 1) + ******************** TEST ' :: 02-fail.c' FAILED ******************** + 02-fail.c:3:10: error: CHECK: expected string not found in input + ; CHECK: Wrong line + ^ + :1:1: note: scanning from here + Hello world + ... + ******************** + Testing Time: 0.11s + ******************** + Failing Tests (1): + :: 02-fail.c + + Unexpected Failures: 1 diff --git a/docs/04-tutorial-llvm-lit-and-filecheck.rst b/docs/04-tutorial-llvm-lit-and-filecheck.rst deleted file mode 100644 index c7dd2e5..0000000 --- a/docs/04-tutorial-llvm-lit-and-filecheck.rst +++ /dev/null @@ -1,29 +0,0 @@ -Tutorial: LLVM LIT and FileCheck -================================ - -This tutorial assumes that you have installed ``filecheck`` and have it -available in your PATH: - -.. code-block:: bash - - $ filecheck - /usr/local/bin/filecheck - not specified - -The FileCheck program expects a path to a check file and a number: - -.. code-block:: text - - filecheck [] - -Create a new file ``hello-world.check`` with the following contents: - -.. code-block:: text - - ; CHECK: Hello world - -.. code-block:: bash - - echo "Hello world" | filecheck hello-world.check - /usr/local/bin/FileCheck - diff --git a/docs/index.rst b/docs/index.rst index 4853f0e..add0167 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -7,7 +7,7 @@ Welcome to FileCheck.py's documentation! 01-what-is-filecheck 02-installation 03-tutorial-hello-world - 04-tutorial-llvm-lit-and-filecheck + 04-tutorial-lit-and-filecheck 05-check-commands 06-options 07-roadmap diff --git a/examples/hello-world-regex.check b/examples/hello-world/hello-world-regex.check similarity index 100% rename from examples/hello-world-regex.check rename to examples/hello-world/hello-world-regex.check diff --git a/examples/hello-world.check b/examples/hello-world/hello-world.check similarity index 100% rename from examples/hello-world.check rename to examples/hello-world/hello-world.check diff --git a/docs/examples/what-is-filecheck/.gitignore b/examples/lit-and-filecheck/.gitignore similarity index 100% rename from docs/examples/what-is-filecheck/.gitignore rename to examples/lit-and-filecheck/.gitignore diff --git a/docs/examples/what-is-filecheck/01-pass.c b/examples/lit-and-filecheck/01-pass.c similarity index 100% rename from docs/examples/what-is-filecheck/01-pass.c rename to examples/lit-and-filecheck/01-pass.c diff --git a/docs/examples/what-is-filecheck/02-fail.c b/examples/lit-and-filecheck/02-fail.c similarity index 100% rename from docs/examples/what-is-filecheck/02-fail.c rename to examples/lit-and-filecheck/02-fail.c diff --git a/examples/lit-and-filecheck/minimal.itest b/examples/lit-and-filecheck/minimal.itest new file mode 100644 index 0000000..53403c9 --- /dev/null +++ b/examples/lit-and-filecheck/minimal.itest @@ -0,0 +1,2 @@ +RUN: echo "Hello world" | filecheck %s +CHECK: Hello world diff --git a/docs/examples/what-is-filecheck/lit.cfg b/examples/lit.cfg similarity index 98% rename from docs/examples/what-is-filecheck/lit.cfg rename to examples/lit.cfg index 6dd2298..1a3d5ed 100644 --- a/docs/examples/what-is-filecheck/lit.cfg +++ b/examples/lit.cfg @@ -1,3 +1,2 @@ import lit.formats config.test_format = lit.formats.ShTest("0") - diff --git a/filecheck/FileCheck.py b/filecheck/FileCheck.py index 4a35ea7..4f8a27e 100755 --- a/filecheck/FileCheck.py +++ b/filecheck/FileCheck.py @@ -234,11 +234,11 @@ def main(): # CHECK and CHECK-NEXT strict_whitespace_match = "" if args.strict_whitespace and args.match_full_lines else " ?" - check_regex = "; {}:{}(.*)".format(check_prefix, strict_whitespace_match) + check_regex = ".*{}:{}(.*)".format(check_prefix, strict_whitespace_match) check_match = re.search(check_regex, line) check_type = CheckType.CHECK if not check_match: - check_regex = "; {}-NEXT:{}(.*)".format(check_prefix, strict_whitespace_match) + check_regex = ".*{}-NEXT:{}(.*)".format(check_prefix, strict_whitespace_match) check_match = re.search(check_regex, line) check_type = CheckType.CHECK_NEXT @@ -263,7 +263,7 @@ def main(): checks.append(check) continue - check_not_regex = "; {}-NOT:{}(.*)".format(check_prefix, strict_whitespace_match) + check_not_regex = ".*{}-NOT:{}(.*)".format(check_prefix, strict_whitespace_match) check_match = re.search(check_not_regex, line) if check_match: match_type = MatchType.SUBSTRING @@ -286,7 +286,7 @@ def main(): checks.append(check) continue - check_empty_regex = "; {}-EMPTY:".format(check_prefix) + check_empty_regex = ".*{}-EMPTY:".format(check_prefix) check_match = re.search(check_empty_regex, line) if check_match: check = Check(check_type=CheckType.CHECK_EMPTY, diff --git a/tests/integration/tests/check_commands/CHECK/one_string/positive-match/02-positive_match_check_without_preceeding_semicolon/filecheck.check b/tests/integration/tests/check_commands/CHECK/one_string/positive-match/02-positive_match_check_without_preceeding_semicolon/filecheck.check new file mode 100644 index 0000000..e7bbab2 --- /dev/null +++ b/tests/integration/tests/check_commands/CHECK/one_string/positive-match/02-positive_match_check_without_preceeding_semicolon/filecheck.check @@ -0,0 +1 @@ +CHECK: hello diff --git a/tests/integration/tests/check_commands/CHECK/one_string/positive-match/02-positive_match_check_without_preceeding_semicolon/filecheck.input b/tests/integration/tests/check_commands/CHECK/one_string/positive-match/02-positive_match_check_without_preceeding_semicolon/filecheck.input new file mode 100644 index 0000000..b6fc4c6 --- /dev/null +++ b/tests/integration/tests/check_commands/CHECK/one_string/positive-match/02-positive_match_check_without_preceeding_semicolon/filecheck.input @@ -0,0 +1 @@ +hello \ No newline at end of file diff --git a/tests/integration/tests/check_commands/CHECK/one_string/positive-match/02-positive_match_check_without_preceeding_semicolon/sample.itest b/tests/integration/tests/check_commands/CHECK/one_string/positive-match/02-positive_match_check_without_preceeding_semicolon/sample.itest new file mode 100644 index 0000000..69aa714 --- /dev/null +++ b/tests/integration/tests/check_commands/CHECK/one_string/positive-match/02-positive_match_check_without_preceeding_semicolon/sample.itest @@ -0,0 +1,2 @@ +; RUN: echo "hello" | %FILECHECK_EXEC %S/filecheck.check | %FILECHECK_TESTER_EXEC %s --match-full-lines +; CHECK: {{^.*}}FileCheck{{(\.py)?$}}