mutate: custom analyzer fail to get stdout

[pokgak]

I wrote a custom analyzer for the output of my test cases using Python (code below), configured to use the analyzer in .dextool_mutate.toml to use the analyzer.

I ran the test with dextool mutate test --mutant=lcr and the analyzer cannot find the correct test name (see log). I suspect this has something to do with passing stdout into the analyzer because when I ran the analyzer code against the sample output manually I successfully get the correct test case name e.g. passed: xtimer_hang.

See below for analyzer code, sample output, and logs.

Analyzer code

#!/usr/bin/env python3

import re
import sys

stdout = sys.argv[1]
# stdout = open("/path/to/sample/output").read()   # use this with sample output for testing

m = re.search(r"tests_(?P<testname>.*).elf", stdout, re.MULTILINE)
testname = m.group("testname") if m is not None else "not found"

m = re.search(r"(?P<status>\[SUCCESS\])", stdout, re.MULTILINE)
status = "passed" if m is not None else "failed"

print(f"{status}: {testname}")

Sample test output

make: Entering directory '/home/pokgak/git/RIOT-mutate/tests/xtimer_hang'
r
/home/pokgak/git/RIOT-mutate/tests/xtimer_hang/bin/native/tests_xtimer_hang.elf  
RIOT native interrupts/signals initialized.
LED_RED_OFF
LED_GREEN_ON
RIOT native board initialized.
RIOT native hardware initialization complete.

Help: Press s to start test, r to print it is ready
READY
s
START
main(): This is RIOT! (Version: 2020.10-devel-578-g90f5d)
[START]
Testing (  0%)
Testing (  1%)
Testing (  2%)
[....]
Testing ( 99%)
Testing (100%)
[SUCCESS]

make: Leaving directory '/home/pokgak/git/RIOT-mutate/tests/xtimer_hang'

Part of the test log with added annotation

pokgak@mobi39:~/git/RIOT-mutate$ dextool mutate test --mutant=lcr
info: Using /home/pokgak/git/RIOT-mutate/mutate/dextool_mutate.sqlite3
info: Checking that the file(s) on the filesystem match the database
info: Ok
info: Checking the build command
info: Ok
info: Found test commands in ["./mutate/tests/"]:
info: shell command '/home/pokgak/git/RIOT-mutate/mutate/tests/hang.sh'
info: Found new test case(s):
info: not found                       <----------------------------- testcase name not found
info: Resetting alive mutants
info: Measuring the runtime of the test command(s):
shell command '/home/pokgak/git/RIOT-mutate/mutate/tests/hang.sh'
info: 0: Measured runtime 14 secs, 558 ms, 977 μs, and 1 hnsec (fastest 14 secs, 558 ms, 977 μs, and 1 hnsec)
[...]

[pokgak]

Okay, I just noticed that the README says the argument is the path to stdout and not the content of stdout itself. 🤦

The arguments are paths to stdout ($1) and stderr ($2).

Fixed it by using stdout = open(sys.argv[1]).read() instead.

[joakim-brannstrom]

good, good that it worked.