Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial steps of Mutation testing #18984

Merged
merged 24 commits into from Nov 16, 2017
Merged
Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
abbdcf0
XMLHTTPRequest Mutator - Initial step Mutation testing
dsandeephegde Oct 19, 2017
0d2ad9a
Checking the wpt test failures/success.
dsandeephegde Oct 25, 2017
ba21529
Added Test Mapping framework and running through a path.
dsandeephegde Oct 25, 2017
af605a6
Added Mutation Test to CI
dsandeephegde Oct 25, 2017
445ab9a
Reverting wrong comment.
dsandeephegde Oct 25, 2017
58ab11c
Fixed few tidy errors
dsandeephegde Oct 25, 2017
c2d46c3
Fixed json tidy error
dsandeephegde Oct 25, 2017
6688b8a
Changed method to mutate a line of code
dsandeephegde Oct 25, 2017
2b8b98d
Removed build and wpt-test output from mutation test log and refactor…
dsandeephegde Oct 25, 2017
5366264
Added Readme.md file for mutation testing
panup21091993 Oct 26, 2017
5cc498d
Loging success message on mutation test success
dsandeephegde Oct 27, 2017
ffbabf4
Added randomness to the mutation strategy
dsandeephegde Oct 31, 2017
d24d6ac
Skipping mutation test for file with local changes
dsandeephegde Nov 6, 2017
84f694d
Added more information in Mutation Test Readme
dsandeephegde Nov 8, 2017
f6f4545
fixed tidy error
dsandeephegde Nov 8, 2017
6e66c0d
Corrected typo in Readme
dsandeephegde Nov 11, 2017
b8199e1
Changed readme saying wildcard not allowed in test_mapping.josn
dsandeephegde Nov 11, 2017
cc6c2ee
Added mutation test summary and made it exit with relevant exit code
dsandeephegde Nov 12, 2017
ed47f89
Changed invocation of muatation test in CI to bash script to use virt…
dsandeephegde Nov 13, 2017
2f900a0
Added PS1 variable before activating virtualenv
dsandeephegde Nov 13, 2017
61794f8
Made test summary more descriptive and Updated Readme
dsandeephegde Nov 14, 2017
f817a9c
Refactored mutate random line method
dsandeephegde Nov 14, 2017
a9493d0
Corrected a typo
dsandeephegde Nov 14, 2017
b8c6d14
Reveting accidental mutant commit
dsandeephegde Nov 16, 2017
File filter...
Filter file types
Jump to…
Jump to file
Failed to load files.

Always

Just for now

Next

XMLHTTPRequest Mutator - Initial step Mutation testing

  • Loading branch information
dsandeephegde committed Nov 6, 2017
commit abbdcf0afc822fbce96f3b4ba152ceb9d0063740
@@ -0,0 +1,35 @@
import fileinput
import re
import subprocess
import sys

def mutate_line(file_name, line_number):
lines = open(file_name, 'r').readlines()
lines[line_number - 1] = re.sub(r'\s&&\s', ' || ', lines[line_number - 1])
out = open(file_name, 'w')

This comment has been minimized.

Copy link
@asajeffrey

asajeffrey Oct 23, 2017

Member

This is opening the file twice for each mutation, which is fine for this small test, but might be problematic when you come to scale it up.

out.writelines(lines)
out.close()

def mutation_test(file_name):
lineNumbers = []
for line in fileinput.input(file_name):
if re.search(r'\s&&\s', line):

This comment has been minimized.

Copy link
@asajeffrey

asajeffrey Oct 30, 2017

Member

Again, there's no randomness here. Scaling this up to the whole servo code base is going to require mutating each line with low probability, not trying everything exhaustively.

This comment has been minimized.

Copy link
@dsandeephegde

dsandeephegde Nov 1, 2017

Contributor

I have added randomness, Now mutation test runs only one mutant per file.

This comment has been minimized.

Copy link
@asajeffrey
lineNumbers.append(fileinput.lineno())

for lineToMutate in lineNumbers:
print "Mutating {0} at line {1}".format(file_name, lineToMutate)

This comment has been minimized.

Copy link
@asajeffrey

asajeffrey Oct 23, 2017

Member

This is testing every mutation exhaustively, but only one mutation at a time. Can we randomize this? Test more than one mutation at a time?

This comment has been minimized.

Copy link
@dsandeephegde

dsandeephegde Oct 25, 2017

Contributor

Yes, we could as it would reduce the running time of the test significantly. But for now, we have implemented one strategy to mutate, which replaces && to ||. There could be 20 occurrences of this in one file. Consider, only one of the occurrences is where the test coverage is not present. It might take 20+ test runs to find that particular failure. This could be overseen as an intermittent issue. As of now, we have not thought about how the final test report should look like. If we need any report after the end of each test run, then this randomization would make it difficult to produce a report.

This comment has been minimized.

Copy link
@dsandeephegde

dsandeephegde Oct 25, 2017

Contributor

If we run more than one mutation at a time and trigger the test suite. How do we classify the test failures based on the mutation?

mutate_line(file_name, lineToMutate)
print "compling mutant {0}-{1}".format(file_name, lineToMutate)
sys.stdout.flush()
subprocess.call('python mach build --release', shell=True)
print "running tests for mutant {0}-{1}".format(file_name, lineToMutate)
sys.stdout.flush()
subprocess.call('python mach test-wpt XMLHttpRequest --release', shell=True)
print "mutated file {0} diff".format(file_name)
sys.stdout.flush()
subprocess.call('git --no-pager diff {0}'.format(file_name), shell=True)
print "reverting mutant {0}-{1}".format(file_name, lineToMutate)
sys.stdout.flush()
subprocess.call('git checkout {0}'.format(file_name), shell=True)

mutation_test('components/script/dom/xmlhttprequest.rs')

This comment has been minimized.

Copy link
@asajeffrey

asajeffrey Oct 23, 2017

Member

If we randomized, can we test more than just one .rs file?

This comment has been minimized.

Copy link
@dsandeephegde

dsandeephegde Oct 25, 2017

Contributor

We are currently picking the file name and tests corresponding to it from a test mapping file. We can run it for all the files one by one which is in a directory and has test mapping.

ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.