Polished Gem
joakim-brannstrom
released this
08 Feb 11:04
·
376 commits
to master
since this release
This is a long overdue release. The focus have been on polishing existing features and improve the user integration and interaction.
This release have now been battle tested in production for two month without running into any significant problems thus it is time to release an update.
There are still lots of features to fix and improve in the future but those will require some re-engineering so stay tuned for future fun.
Fixes
- Fixed build instructions and cmake config to allow greater control over what
llvm/clang libs are linked. This allows dextool to be built for Fedora-34. - Support llvm/clang 13.
- Fix compilation with latest dmd-compiler (2.097.2)
New features for dextool mutate
- Add a page to HTML report which show the worklist. This is to help a user to
understand what is being worked on by the tool. - Parallel testing of mutants in a schema. Previously the total number of
cores where not fully utilized because only one mutant where tested at a
time. This usually meant that there ended up being idle cores because the
test suite had one or more slow tests that "held back" the testing of
mutants. This is now fixed by running multiple mutants at the same time
which mean that there will probably always be tests that are being
executed on all cores.
From some casual testing this lead to 2-10x faster testing.
Fixes for dextool mutate
- HTML file report could fail. Added code to catch exceptions during report
generation which should fix the problem. - Running parallel instances didn't work that great when testing mutants in
large schemas because an instances didn't update the mutants to test with
those that had already been tested by another instances. This has now been
fixed by polling the database each minute to remove those mutants that have
been tested.
Multiple instances have also been changed to test the mutants in a schema in
random order. This should further reduce the "overlap". The final result is
that for a schema that have 500 mutants and there are 5 instances running in
parallel then each should only need to test ~100 mutants each. - Retest of old mutants could get stuck on those marked with
noCoverage
because the logic would add them to the worklist during initialization and
then be removed by before the test phase started. It
is now changed to only add mutants not with thenoCoverage
status to the
worklist. - Populate the worklist with re-test of old mutants if it is below the
specified retest number. - Fix memory overload check. It failed to parse
/proc/meminfo
. - Block schemas inside constexpr template functions. This reduces the number
of failed schemas thus improve the overall speed.
This fix is only available with clang-12+. - If
test
is run without finishing and thenanalyze
is ran the handling of
timeout mutants will lead to an ever increasing list.
The system basically live lock over time. This is now fixed by tracking the
timeout iteration per mutant instead of globally. - Fix an infinite recursion bug that lead to crash when analyzing. Happened
for example when analyzing llvm. - (html) The trend graph would look like a "hill" /\ when there where a
significant amount of mutants with the timeout status while the timeout
algorithm where still testing them. Because first they where classified as
timeout (killed) and then later on changed to alive when the timeout where
increased. This where detrimental to a teams "motivational drive" to work
with mutation testing and improving the test suite. They expected the first
sample, which showed an improvement, to be "true" and which lead to a
positive feeling. Then later on it started to go down without the team
actually "doing" anything with the test suite.
The change is to only update the trend score graph when there are no timeout
mutants to re-test. - (html) Make the trend graphs prediction color "robust" to small changes
(fluctuations) by using grey color when it is below one 1%. This thus
mean that when it is red the team know that "we have to act", and vice versa
when it is green they know that "it is going well now!". - A binary that contains a mutation schema could sometimes result in a
significant slower test suite. This would lead to a high number of mutants
being initially tagged as "timeout" which slowed down the mutation testing
process significantly. This is now fixed by updating the test suite timeout
by measuring the test suite with the schema injected. - Fixed implementation of pointer aware ROR when generating mutants for
==
and!=
.