Merge pull request #820 from mull-project/doc-update

docs: update to the latest state (Features, How Mull works, Hello World tutorial)

docs/Features.rst

@@ -1,9 +1,32 @@
 Features
 ========
 
-Key features are:
+- Mull enables mutation testing of C and C++ projects.
+
+- Mull expects a passing test suite to exit with 0. If the test suite fails, it
+  must exit with a non-zero exit code. Any C/C++ test framework that follows
+  this convention is supported by Mull.
+
+- `Supported Mutations <SupportedMutations.html>`_.
+
+- Generate results in various formats:
+
+  - IDE Reporter: compiler-like warnings are printed to standard output
+
+    - ``sample.cpp:15:11: warning: Killed: Replaced >= with < [cxx_ge_to_lt]``
+
+  - SQLite Reporter: SQLite database file.
+
+  - JSON file that conforms `mutation-testing-elements schema <https://github.com/stryker-mutator/mutation-testing-elements/tree/master/packages/mutation-testing-report-schema>`_
+
+  - `Mutation Testing Elements HTML Reporter <https://github.com/stryker-mutator/mutation-testing-elements/tree/master/packages/mutation-testing-elements>`_
+
+- Parallelized execution of tests
+
+- Mull requires test programs to be compiled with Clang/LLVM. Mull supports
+  all LLVM versions starting from LLVM 6.
+
+For a more detailed description of Mull's architecture, see
+`How Mull works <HowMullWorks.html>`_.
+
 
-- Support of a number of unit testing frameworks: `Google Test <https://github.com/google/googletest>`, `CppUTest <http://cpputest.github.io/>`.
-- Support of a number of `mutation operators <SupportedMutations.html>`_.
-- Generate results in a different formats: compiler-like warnings printed to standard output, SQLite DB	and JSON file that conforms `mutation-testing-elements schema <https://github.com/stryker-mutator/mutation-testing-elements/tree/master/packages/mutation-testing-report-schema>`.
-- Parallel running of tests

docs/GettingStarted.rst

@@ -9,7 +9,7 @@ Then, `install Mull <Installation.html>`_ and go through the `tutorials <Tutoria
 As soon as you are comfortable with the basics you may want to learn about various `options and settings <CommandLineReference.html>`_ Mull has,
 as well as pick the right set of available `mutation operators <SupportedMutations.html>`_.
 
-Doing research? There is a special chapter for you: `For Researchers <ForResearchers.html>`_.
+If you are curious about how Mull works under the hood: `How Mull works <HowMullWorks.html>`_.
 
 If you want to dive deeper and look behind the curtains, then we encourage you to `hack on Mull <HackingOnMull.html>`_.
 

docs/HelloWorld.rst

@@ -11,6 +11,9 @@ down to ``Complete example`` below.
 
 ----
 
+Step 1: Checking version
+------------------------
+
 The tutorial assumes that you have `installed <Installation.html>`_ Mull on your system and
 have the `mull-cxx` executable available:
 
@@ -19,11 +22,14 @@ have the `mull-cxx` executable available:
     $ mull-cxx -version
     Mull: LLVM-based mutation testing
     https://github.com/mull-project/mull
-    Version: 0.7.0
-    Commit: 1638698
-    Date: 28 Mar 2020
+    Version: 0.8.0
+    Commit: f94f38ed
+    Date: 04 Jan 2021
     LLVM: 9.0.0
 
+Step 2: Enabling Bitcode
+------------------------
+
 The most important thing that Mull needs to know is the path to your program
 which must be a valid C or C++ executable. Let's create a C program:
 
@@ -41,27 +47,6 @@ and compile it:
 
 We can already try running ``mull-cxx`` and see what happens:
 
-.. code-block:: bash
-
-    $ mull-cxx hello-world
-    mull-cxx: for the -test-framework option: must be specified at least once!
-
-This is the second important thing that Mull needs: we have to specify which
-kind of test framework Mull should assume our program uses.
-
-We specify ``CustomTest``:
-
-.. code-block:: bash
-
-    mull-cxx -test-framework=CustomTest hello-world
-
-``-test-framework=CustomTest`` parameter tells Mull that it should not expect
-a real test framework such as Google Test or any kind of advanced test suite.
-Instead Mull will simply consider that our tests will be simple test functions
-which we will call from the ``main()`` function.
-
-Now the output is different:
-
 .. code-block:: text
 
     [info] Extracting bitcode from executable (threads: 1)
@@ -87,7 +72,7 @@ Let's try again:
 .. code-block:: text
 
     $ clang -fembed-bitcode -g main.cpp -o hello-world
-    $ mull-cxx -test-framework CustomTest hello-world
+    $ mull-cxx hello-world
     [info] Extracting bitcode from executable (threads: 1)
            [################################] 1/1. Finished in 3ms
     [info] Loading bitcode files (threads: 1)
@@ -109,6 +94,9 @@ important part of the output: ``No mutants found. Mutation score: infinitely
 high``. We have our executable but we don't have any code so there is nothing
 Mull could work on.
 
+Step 3: Killing mutants, one survived
+-------------------------------------
+
 Let's add some code:
 
 .. code-block:: c
@@ -146,7 +134,7 @@ verbose.
 .. code-block:: text
 
     $ clang -fembed-bitcode -g main.cpp -o hello-world
-    $ mull-cxx -test-framework=CustomTest -ide-reporter-show-killed hello-world
+    $ mull-cxx -ide-reporter-show-killed hello-world
     [info] Extracting bitcode from executable (threads: 1)
            [################################] 1/1. Finished in 10ms
     [info] Loading bitcode files (threads: 1)
@@ -206,8 +194,8 @@ what the survived mutation is about: Mull has replaced ``age >= 21`` with
 
 Let's add the third test case and see what happens.
 
-Complete example
-----------------
+Step 4: Killing mutants again, all killed
+-----------------------------------------
 
 The code:
 
@@ -246,7 +234,7 @@ The code:
 .. code-block:: text
 
     $ clang -fembed-bitcode -g main.cpp -o hello-world
-    $ mull-cxx -test-framework=CustomTest -ide-reporter-show-killed hello-world
+    $ mull-cxx -ide-reporter-show-killed hello-world
     [info] Extracting bitcode from executable (threads: 1)
            [################################] 1/1. Finished in 4ms
     [info] Loading bitcode files (threads: 1)
@@ -304,7 +292,7 @@ This is a short summary of what we have learned in tutorial:
 
 - Your code has to be compiled with ``-fembed-bitcode -g`` compile flags:
 
-  - Mull expects embedded bitcode files to be present in binary executable
+  - Mull expects embedded bitcode files to be present in a binary executable
     (ensured by ``-fembed-bitcode``).
 
   - Mull needs debug information to be included by the compiler (enabled by
@@ -314,7 +302,3 @@ This is a short summary of what we have learned in tutorial:
 - Mull expects the following arguments to be always provided:
 
   - Your executable program
-
-  - ``-test-framework`` parameter that tells Mull which kind of testing
-    framework to expect. In this tutorial we have been using the ``CustomTest``
-    framework.

docs/ForResearchers.rst → docs/HowMullWorks.rst

@@ -1,5 +1,5 @@
-For researchers
-===============
+How Mull works
+==============
 
 This page contains a short summary of the design and features of Mull. Also
 the advantages of Mull are highlighted as well as some known issues.
@@ -9,35 +9,45 @@ If you want to learn more than we cover here, Mull has a paper:
 
 ----
 
-
 Design
 ------
 
 Mull is based on LLVM and uses its API extensively. The main APIs used are:
-**LLVM IR**, **LLVM JIT**, **Clang AST API**.
+**LLVM IR** and **Clang AST API**.
 
 Mull finds and creates mutations of a program in memory, on the level of LLVM
 bitcode.
 
-Mull uses information about source code obtained via Clang AST API to find which
-mutations in LLVM bitcode are valid (i.e. they trace back to the source code),
-all invalid mutations are ignored in a controlled way.
+All mutations are injected into the original program code and each injected gets
+mutation hidden under a conditional flag that enables that specific mutation.
+The resulting program is compiled into a single binary which is run multiple
+times, one run per mutation. With each run, a condition for a corresponding
+mutation is activated to check how the injection of that particular mutation
+affects the execution of a test suite.
+
+Mull runs the tested program and its mutated versions in child subprocesses so
+that the execution of the tested program does not affect Mull running in a
+parent process.
+
+**Note:** Mull no longer uses LLVM JIT for execution of mutated programs.
+See the
+`Historical note: LLVM JIT deprecation (January 2021)`_.
 
-Mull runs the program and its mutated versions in memory using LLVM JIT. The
-``fork()`` call is used to run mutants in child subprocesses so that their
-execution does not affect Mull as a parent process.
+Mull uses information about source code obtained via Clang AST API to find out
+which mutations in LLVM bitcode are valid (i.e. they trace back to the source
+code), all invalid mutations are ignored in a controlled way.
 
 Mutations search
 ----------------
 
-The default search algorithm simply finds all mutations that can be made on the
+The default search algorithm simply finds all mutations that can be found on the
 level of LLVM bitcode.
 
 The **"black search" algorithm** called Junk Detection uses source code information
 provided by Clang AST to filter out invalid mutations from a set of all possible
 mutations that are found in LLVM bitcode by the default search algorithm.
 
-The **"white search"** algorithm starts with collecting source code information
+The **"white search" algorithm** starts with collecting source code information
 via Clang AST and then feeds this information to the default search algorithm
 which allows finding valid mutations and filtering out invalid mutations
 at the same time.
@@ -59,7 +69,8 @@ See `Supported Mutation Operators <SupportedMutations.html>`_.
 Reporting
 ---------
 
-Mull reports survived/killed mutations to the console by default.
+Mull reports survived/killed mutations to the console by default. The
+compiler-like warnings are printed to standard output.
 
 Mull has an SQLite reporter: mutants and execution results are collected in
 SQLite database. This kind of reporting makes it possible to make SQL queries
@@ -86,20 +97,7 @@ Mull has 3 layers of testing:
 
 1. Unit and integration testing on the level of C++ classes
 2. Integration testing against known real-world projects, such as OpenSSL
-3. Integration testing using LLVM Integrated Tester (in progress)
-
-Current development
--------------------
-
-The current development goals for Mull for Autumn 2019 - Spring 2020 are:
-
-- Stable performance of black and white search algorithms supported by a solid
-  integration test coverage.
-
-- **Incremental mutation testing**. Mull can already run on subsets of program
-  code but the API and workflows are still evolving.
-
-- More mutation operators.
+3. Integration testing using LLVM Integrated Tester (LIT)
 
 Advantages
 ----------
@@ -109,18 +107,16 @@ mutations is very good performance. Combined with incremental mutation testing
 one can get mutation testing reports in the order of few seconds.
 
 Another advantage is language agnosticism. The developers of Mull have been
-focusing on C/C++ as their primary development languages at their jobs but
-the proof of concepts have been developed for the other compiled languages such
-as Rust and Swift.
+focusing on C/C++ as the primary supported languages but the proof of concepts
+for other compiled languages, such as Rust and Swift, have been developed.
 
 A lot of development effort have been put into Mull in order to make it stable
 across different operating systems and versions of LLVM. Combined with the
-growing test coverage and highly modular design the authors are slowly but
-steady getting to the point when they can claim that Mull is a very stable,
-very well tested and maintained system.
+growing test coverage and highly modular design, Mull is a very stable,
+well-tested and maintained system.
 
-Known issues
-------------
+Known issue: Precision
+----------------------
 
 Mull works on the level of LLVM bitcode and from there it gets its strengths
 but also its main weakness: the precision of the information for mutations is
@@ -131,6 +127,15 @@ things the good suite of integration tests is aimed to provide Mull with a good
 contract of supported mutations which are predictable and known to work without
 any side effects.
 
+Historical note: LLVM JIT deprecation (January 2021)
+----------------------------------------------------
+
+The usage of LLVM JIT has been deprecated and all LLVM JIT-related code has been
+removed from Mull by January 2021.
+
+This issue explains the reasons:
+`PSA: Moving away from JIT <https://github.com/mull-project/mull/issues/798>`_.
+
 Paper
 -----
 

docs/index.rst

@@ -11,7 +11,7 @@ Welcome to Mull's documentation!
    Tutorials
    SupportedMutations
    CommandLineReference
-   ForResearchers
+   HowMullWorks
    HackingOnMull
 
 .. Indices and tables