Skip to content

Polished Gem

Compare
Choose a tag to compare
@joakim-brannstrom joakim-brannstrom released this 08 Feb 11:04
· 376 commits to master since this release

This is a long overdue release. The focus have been on polishing existing features and improve the user integration and interaction.
This release have now been battle tested in production for two month without running into any significant problems thus it is time to release an update.

There are still lots of features to fix and improve in the future but those will require some re-engineering so stay tuned for future fun.


Fixes

  • Fixed build instructions and cmake config to allow greater control over what
    llvm/clang libs are linked. This allows dextool to be built for Fedora-34.
  • Support llvm/clang 13.
  • Fix compilation with latest dmd-compiler (2.097.2)

New features for dextool mutate

  • Add a page to HTML report which show the worklist. This is to help a user to
    understand what is being worked on by the tool.
  • Parallel testing of mutants in a schema. Previously the total number of
    cores where not fully utilized because only one mutant where tested at a
    time. This usually meant that there ended up being idle cores because the
    test suite had one or more slow tests that "held back" the testing of
    mutants. This is now fixed by running multiple mutants at the same time
    which mean that there will probably always be tests that are being
    executed on all cores.
    From some casual testing this lead to 2-10x faster testing.

Fixes for dextool mutate

  • HTML file report could fail. Added code to catch exceptions during report
    generation which should fix the problem.
  • Running parallel instances didn't work that great when testing mutants in
    large schemas because an instances didn't update the mutants to test with
    those that had already been tested by another instances. This has now been
    fixed by polling the database each minute to remove those mutants that have
    been tested.
    Multiple instances have also been changed to test the mutants in a schema in
    random order. This should further reduce the "overlap". The final result is
    that for a schema that have 500 mutants and there are 5 instances running in
    parallel then each should only need to test ~100 mutants each.
  • Retest of old mutants could get stuck on those marked with noCoverage
    because the logic would add them to the worklist during initialization and
    then be removed by before the test phase started. It
    is now changed to only add mutants not with the noCoverage status to the
    worklist.
  • Populate the worklist with re-test of old mutants if it is below the
    specified retest number.
  • Fix memory overload check. It failed to parse /proc/meminfo.
  • Block schemas inside constexpr template functions. This reduces the number
    of failed schemas thus improve the overall speed.
    This fix is only available with clang-12+.
  • If test is run without finishing and then analyze is ran the handling of
    timeout mutants will lead to an ever increasing list.
    The system basically live lock over time. This is now fixed by tracking the
    timeout iteration per mutant instead of globally.
  • Fix an infinite recursion bug that lead to crash when analyzing. Happened
    for example when analyzing llvm.
  • (html) The trend graph would look like a "hill" /\ when there where a
    significant amount of mutants with the timeout status while the timeout
    algorithm where still testing them. Because first they where classified as
    timeout (killed) and then later on changed to alive when the timeout where
    increased. This where detrimental to a teams "motivational drive" to work
    with mutation testing and improving the test suite. They expected the first
    sample, which showed an improvement, to be "true" and which lead to a
    positive feeling. Then later on it started to go down without the team
    actually "doing" anything with the test suite.
    The change is to only update the trend score graph when there are no timeout
    mutants to re-test.
  • (html) Make the trend graphs prediction color "robust" to small changes
    (fluctuations) by using grey color when it is below one 1%. This thus
    mean that when it is red the team know that "we have to act", and vice versa
    when it is green they know that "it is going well now!".
  • A binary that contains a mutation schema could sometimes result in a
    significant slower test suite. This would lead to a high number of mutants
    being initially tagged as "timeout" which slowed down the mutation testing
    process significantly. This is now fixed by updating the test suite timeout
    by measuring the test suite with the schema injected.
  • Fixed implementation of pointer aware ROR when generating mutants for == and !=.