diff --git a/python/servo/mutation/README.md b/python/servo/mutation/README.md index c52953c363cb..bc9c5f2eea6d 100644 --- a/python/servo/mutation/README.md +++ b/python/servo/mutation/README.md @@ -5,7 +5,7 @@ The motivation for mutation testing is to test the breadth coverage of tests for For more info refer [Wiki page](https://en.wikipedia.org/wiki/Mutation_testing). -In this project, mutation testing is used to test the coverage of WPT for Servo's browsing engine. +Here Mutation testing is used to test the coverage of WPT for Servo's browsing engine. ### Mutation Strategy This version of mutation testing consists of a Python script that finds random uses of && in Servo's code base and replaces them by ||. The expectation from the WPT tests is to catch this mutation and result in failures when executed on the corresponding code base. @@ -16,7 +16,7 @@ The mutation test aims to run only tests which are concerned with the mutant. Th #### test_mapping.json The file test_mapping.json is used to map the source code to their corresponding WPT tests. The user must maintain a updated version of this file in the path where mutation testing needs to be performed. Additionally, the test_mapping.json will only consist of maps of source codes that are present in the current directory. Hence, each folder will have a unique test_mapping.json file. Any source code files that may be present in a path but are not mapped to a WPT in test_mapping.json will not be covered for mutation testing. -### Sample test_mapping.json format +#### Sample test_mapping.json format A sample of test_mapping.json is as shown below: ``` @@ -32,14 +32,14 @@ A sample of test_mapping.json is as shown below: Please ensure that each folder that requires a mutant to be generated consists of test_mapping.json file so that the script can function as expected. -### Basic Execution Flow -The implementation of mutation testing is as follows: -1. The script is called from the command line, it searches through the path entered by user for test_mapping.json. -2. If found, it reads the json file and parses one componenent of source file to generate mutants. -3. The corresponding WPT tests are run for each mutant and the test results are logged. -4. Once all WPT are run for the first source file, the mutation continues for other source files mentioned in the json file and runs their corresponding WPT tests. -5. Once it has completed executing mutation testing for the entered path, it repeats the above procedure for sub-paths present inside the entered path. +If we want to run mutation test for a source path then there should be test_mapping.json in that path and all the subdirectories which has source files. +Eg: There should be test mapping in following folders if we run mutation test on 'components/script' path. +* components/script/test_mapping.json +* components/script/dom/test_mapping.json +* components/script/task_source/test_mapping.json +* components/script/dom/bindings/test_mapping.json +* ... ### Running Mutation test The mutation tests can be run by running the below command from the servo directory on the command line interface: @@ -52,4 +52,12 @@ Eg. `python python/servo/mutation/init.py components/script/dom` The CI script for running mutation testing is present in /etc/ci folder. It can be called by executing the below command from the CLI: -`python /etc/ci/mutation_test.py` \ No newline at end of file +`python /etc/ci/mutation_test.py` + +### Execution Flow +1. The script is called from the command line, it searches for test_mapping.json in the path entered by user. +2. If found, it reads the json file and parses it, gets source file to tests mapping. +3. If the source file does not have any local changes then it is mutated at a random line. +4. The corresponding WPT tests are run for this mutant and the test results are logged. +5. Once all WPT are run for the first source file, the mutation continues for other source files mentioned in the json file and runs their corresponding WPT tests. +6. Once it has completed executing mutation testing for the entered path, it repeats the above procedure for sub-paths present inside the entered path. diff --git a/python/servo/mutation/test.py b/python/servo/mutation/test.py index 242d545a305d..1d5a6f48d2ab 100644 --- a/python/servo/mutation/test.py +++ b/python/servo/mutation/test.py @@ -61,4 +61,4 @@ def mutation_test(file_name, tests): subprocess.call('git checkout {0}'.format(file_name), shell=True) else: print "Cannot mutate {0}".format(file_name) - print "-----------------------------------------------------------------\n" + print "-"*80 + "\n"