Limestone

Lots of changes. The user visible highlight is the significant improvements to
the HTML report. Everything has been improved from the overview to the file
mutant navigation. It is now a much more pleasent experience to read the
report.

The significant behavior change with this release is the removal of --mutant.
Allowing test and report to use other mutantion operators than what where
used during the analyze phase both resulted in a higher implementation
complexity and surprising user behavior. The new behavior is that what has been
analyzed is tested and reported. It is both more logical as implemented and how
it is used.

Fixes

• Updated support to llvm-14. This fix should make dextool automatically work
  with future versions of llvm as long as the C bindings that are used by
  dextool are unchanged.
• Bindings updated to support llvm-15.

New features for dextool mutate

• Annotating each line with NOMUT for mutants to ignore may end up being too
  much clutter in the source code. It may even be unfeasible to add on the line
  because it may become too long. Therefor two new annotation types are added.
  NOMUTNEXT to ignore the mutants on the next line and NOMUTBEGIN/NOMUTEND
  to ignore mutants in a whole block of code.
• (html report) Auto detect when javascript is deactivated to print a warning.
  This is mainly for Jenkins servers wherein the security settings may have
  activated CSP. The HTML report look really bad and isn't working as
  expected. For a new user it may seem like it should be "this bad". The auto
  detect thus print show a warning to the user and a hint of how to solve it
  if CSP is the problem.
• Add support for constant replacement of floating point numbers.
• (html report) Improve the sortable tables
• (html report) Improve how mutants that span multiple lines are displayed.
• (html report) Remove buggy treemap.
• (html report) Display an explanation for the mutation operator abbreviations
  when hovering with the mouse.
• (html report) Display how an individual files mutation score have changed
  compared to the last 7 days average. This is to help in identifying negative
  trends.
• (html report) Display coverage information.
• (html report) Move help text and explanations to a popup dialog (I) to
  reduce the clutter.
• (html report) When clicking on a mutant choose the smallest mutants under
  the cursors. This is to fix the annoying behavior wherein the first mutant
  where chosen which could be one that span multiple lines. It is hardly ever
  that one that is of interest. If the mouse is over e.g. a + then the
  expected behavior is to show the AOR mutants under the cursor (-).
• (html report) Deduplicate mutants that have the same source code
  transformation to make it easier to find the relevant mutant to look.
• (html report) Fix scrolling in a file report. Scrolling with the mouse do
  not change the mutant except when the mouse is over the window showing the
  mutants on the current line.
• (html report) Fix the web browsers "back" button when looking at a file report.
• The CLI parameter --mutant has been removed from test and report
  command groups.
• (html report) The test cases in the info box in the file report is now a
  link to the detailed test case report.
• Automatically trigger a full analyze when the configuration is changed.

Fixes for dextool mutate

• (html report) Fix sortable tables in chrome.
• Coverage info could accumulate infinitely in the database, wrong foreign key.
• Changed the checksum of mutants from 128bit to 64bit to simplify the DB
  schema, implementation and slightly improve the performance.
• Fixed a bug when sorting mutants by datetime which resulted in e.g. "testing
  oldest mutant" not working as it should.
• The trend graph for file changes is redesigned to show when a files mutation
  score last changed. It should make it easier to understand why the mutation
  score have changed because it points to the files that are changed.
• --load-behavior halt sometimes failed because the threadpool used for
  tests did not properly shutdown. Added a background thread that triggers after
  10 minutes to forcefully shutdown dextool. This is mainly an issue when
  dextool is running on a shared CI server and it becomes overloaded. It is hard
  to reproduce but the server probably run out of memory.
• Fix coverage bug where a mutant where not re-tested if only tests changed
  such that the mutant is not covered.
• Improve scheman. Moved the schema generator from analyze to test phase to
  make use of the internal worklist. The worklist contains what mutants to test.
  The scheman are now built up from fragments containing mutants that are in the
  worklist. This result in more mutants that are tested for each schema. A rough
  test show a 50% reduction of the number of scheman needed which mean a
  sometimes huge reduction in test time. A side effect of this is that the
  analyze phase use a lot less memory.
• Analyze phase where not stable. The same mutants where not saved to the
  database even if no source code where changed. Luckily the bug where in the
  SQL schema definition which prohibited the same mutant kind to affect one and
  the same source code location.

Deep Steel

Fixes

• Fix compilation with dmd-compiler (2.100.0).
• Removing unused fuzzer plugin.

New features for dextool mutate

• test_cmd_dir where previously always searched recursively but it has turned out to be a problem because some test suites contain executable test data. Thus the new configuration option test_cmd_dir_search is added to allow the search to be configured to be shallow (non-recursive).
• Improved speed when using diff mode by skipping the coverage instrumentation. The user has already specified exactly what to test thus it shouldn't be necessary to run it through the potentially slow coverage run. It is assumed that a diff is usually pretty "small" and the change have entry point coverage thus coverage data wouldn't help.

Fixes for dextool mutate

• Update moment.js to fix security vulnerabilities.
  Impact
  This vulnerability impacts npm (server) users of moment.js, especially if user
  provided locale string, eg fr is directly used to switch moment locale.
  The vulnerability have a low or no impact on dextool as of date but problems should be fixed.
• Analyze never finish if the number of threads are set to one (--threads 1). It is now changed to be a minimum of four.

Cat Gold

Fixes

• Fix compilation with latest dmd-compiler (2.099.0). Required the min version
  to be updated too though because the older one segfaults when compiling
  miniorm.

Fixes for dextool mutate

• Schemas may, and do, affect the runtime of a test suite. This mean that
  the timeout need to be measured and adjusted per schema. This feature exists
  and is done but the timeout where set after all test workers where spawned
  which meant it had no effect. The fix is to send a signal to all test workers
  to update the timeout, if it is needed. The end result is that schemas should
  result in less mutants with timeout status.

New features for dextool mutate

• Add an alternative algorithm for calculating IDs for mutants (relaxed).
  The new algorithm is scope aware. Mutants are unique within a file and
  within a scope. Only mutants within a scope are re-tested if the source code
  in the scope is changed. This thus basically mean that the algorithm works
  like "git diff". It only test mutants in scopes that have changed. It makes it
  faster to test a code change and not lose the report for the whole program
  compared to a "git diff" that only test and report for the changed lines.
  The option is activated via id_algo in the configuration file.
• Changed the default ID algorithm to relaxed. After extensive testing it is
  now mature enough to be used by default. Its main benefit is that diff based
  testing of changes are activated by default which reduces the overall test
  time when small changes are made to the source code.

Polished Gem

This is a long overdue release. The focus have been on polishing existing features and improve the user integration and interaction.
This release have now been battle tested in production for two month without running into any significant problems thus it is time to release an update.

There are still lots of features to fix and improve in the future but those will require some re-engineering so stay tuned for future fun.

---

Fixes

• Fixed build instructions and cmake config to allow greater control over what
  llvm/clang libs are linked. This allows dextool to be built for Fedora-34.
• Support llvm/clang 13.
• Fix compilation with latest dmd-compiler (2.097.2)

New features for dextool mutate

• Add a page to HTML report which show the worklist. This is to help a user to
  understand what is being worked on by the tool.
• Parallel testing of mutants in a schema. Previously the total number of
  cores where not fully utilized because only one mutant where tested at a
  time. This usually meant that there ended up being idle cores because the
  test suite had one or more slow tests that "held back" the testing of
  mutants. This is now fixed by running multiple mutants at the same time
  which mean that there will probably always be tests that are being
  executed on all cores.
  From some casual testing this lead to 2-10x faster testing.

Fixes for dextool mutate

• HTML file report could fail. Added code to catch exceptions during report
  generation which should fix the problem.
• Running parallel instances didn't work that great when testing mutants in
  large schemas because an instances didn't update the mutants to test with
  those that had already been tested by another instances. This has now been
  fixed by polling the database each minute to remove those mutants that have
  been tested.
  Multiple instances have also been changed to test the mutants in a schema in
  random order. This should further reduce the "overlap". The final result is
  that for a schema that have 500 mutants and there are 5 instances running in
  parallel then each should only need to test ~100 mutants each.
• Retest of old mutants could get stuck on those marked with noCoverage
  because the logic would add them to the worklist during initialization and
  then be removed by before the test phase started. It
  is now changed to only add mutants not with the noCoverage status to the
  worklist.
• Populate the worklist with re-test of old mutants if it is below the
  specified retest number.
• Fix memory overload check. It failed to parse /proc/meminfo.
• Block schemas inside constexpr template functions. This reduces the number
  of failed schemas thus improve the overall speed.
  This fix is only available with clang-12+.
• If test is run without finishing and then analyze is ran the handling of
  timeout mutants will lead to an ever increasing list.
  The system basically live lock over time. This is now fixed by tracking the
  timeout iteration per mutant instead of globally.
• Fix an infinite recursion bug that lead to crash when analyzing. Happened
  for example when analyzing llvm.
• (html) The trend graph would look like a "hill" /\ when there where a
  significant amount of mutants with the timeout status while the timeout
  algorithm where still testing them. Because first they where classified as
  timeout (killed) and then later on changed to alive when the timeout where
  increased. This where detrimental to a teams "motivational drive" to work
  with mutation testing and improving the test suite. They expected the first
  sample, which showed an improvement, to be "true" and which lead to a
  positive feeling. Then later on it started to go down without the team
  actually "doing" anything with the test suite.
  The change is to only update the trend score graph when there are no timeout
  mutants to re-test.
• (html) Make the trend graphs prediction color "robust" to small changes
  (fluctuations) by using grey color when it is below one 1%. This thus
  mean that when it is red the team know that "we have to act", and vice versa
  when it is green they know that "it is going well now!".
• A binary that contains a mutation schema could sometimes result in a
  significant slower test suite. This would lead to a high number of mutants
  being initially tagged as "timeout" which slowed down the mutation testing
  process significantly. This is now fixed by updating the test suite timeout
  by measuring the test suite with the schema injected.
• Fixed implementation of pointer aware ROR when generating mutants for == and !=.

Smooth Road

New features for dextool mutate

• Speedup the test phase by skipping mutants inside SDL mutants. Because a SDL
  delete the code it is higly unlikely that mutants inside such a region would
  be killed. This is basically an extension of using coverage to skip mutants
  to test.
• Automatic classification of test cases as either unique, bad or buggy. It
  hopefully makes it easier to read the test case report and fast find the
  interesting test case for further inspection without having to read all test
  cases in detail.
• (html) A page with all mutants, their status etc has been added.
• (html) Allow a user to provide additional metadata about test cases such as
  a link to the implementation and text about the test case. This is in a
  separate json file. The intention is that tools are used to extract the
  information from the implementation of the test cases.

[{"name": "tc_1", "redundant": true, "text": "<p>a text with <br> html <a href=\"foo.html\">links</a>.</p>", "location": {"file": "../../tc_1.cpp", "line": 42}}]

• When test_cmds are executed the host could run out of memry. It resulted
  in either the tool hanging, the linux OOM killer reaping processes etc. In
  summary the system and tool became unstable. The tool now have support for
  specifying the max total used memory of the host via
  max_mem_usage_percentage. If the limit is reached test_cmds will be
  terminated.
• Use __builtin_expect to improve performance when executing binaries with
  schemas by marking all mutant branches as unlikely. It may reduce the test
  suite execution time when running with schemas by up to 20%.
• Somewhat solve the problem of scheman that fail to compile by letting a
  failed schema influence how future scheman are generated. "Adapt" future
  scheman based on passed failures. This allows the schema generator to be
  less pessimistic about pointers because a code base that use "pointer
  tricks" which lead to many failing scheman will over time adjust how the
  scheman are generated.
• Speedup testing of heavy templated libraries by allowing scheman to also
  cover the templated functions. This may initially lead to many failing
  scheman but this will lead to adaption of how the scheman are composed which
  should over time self heal.
• The slowdown mode now reacts faster to a system overload by using loadavg1
  instead of 15.

Fixes for dextool mutate

• Fix invalid scheman when a constexpr is used as a label in a case stmt.
• Fix broken test case suggestion SQL query.
• Fix scattered test case report. Users have reported that the information
  about test cases are spread out in the html report. This constructs an
  individual page per test case and a summary table in index.html for easy
  navigation.
• Updated llvm compatiblity to version 12. The bindings for version 11 worked
  most of the time except for new expressions which could lead to mutant
  schematan that didn't compile.
• Fix bug when the path to the database is a symlink.

Emerald

General

• Fix compatibility with D language frontend 2.079.0+

New features for dextool mutate

• The configuration file for coverage and schema is changed to allow a user to
  specify that the runtime should not be injected by the tool. The user, you,
  manually link it. This is because the automatic injection do not always work
  thus this option fixes the problem by allowing a user to change the build
  system and link with a pre-compiled version of the runtime.
• Install static libraries of the coverage and schema runtimes. See
  README_tutorial_autotools.md for an example of how to use them.
• Reduce which test_cmd are executed when the option test_cmd_checksum is
  activated. This option also activate a checksum table of test_cmd -
  mutation status. If the checksum of the test_cmd match what is already
  known then that mutation status is used instead.

Fixes for dextool mutate

• Fix labels being duplicated in scheman and thus leading to compilation errors.
• Fix that a UTF-8 BOM lead to compilation errors. The tool now find and warn about it.
  It further detect when the source code is in something else than UTF-8 and warn.
• Reduce memory usage by stack allocating AST nodes.
• Fix C++17 if-statement initialization making scheman not compile.
• Fix slow re-analyze on a large project by optimizing the sqlite schema (use indexes).
• Fix GC-crash on long analysis. Dangling object where collected.

Topaz

New features for dextool mutate

• Speedup execution of tests by only running those that are affected by the
  mutant when running the slow source code modification engine. This is done
  by saving the checksum of all test_cmd after build_cmd is executed and
  no mutant is inserted. When a mutant is later on inserted the resulting
  test_cmds are compared to the unmodified. Only those that are different
  are classified as affected and thus need to be executed. If no test_cmd
  is affected by the inserted mutant the mutation is classified as
  equivalent. Set test_cmd_checksum to true to activate the feature. It is
  by default turned off.
• Changed the default test order of mutants to bySize after a thorough
  testing phase. It means that the mutants are tested in random order but
  weighted by how much they affect the program (SUT). This mean that e.g.
  large sdl mutants will be prioritized above aor mutants.

Fixes for dextool mutate

• The captured output from the test cases wasn't possible to limit which could
  lead to a memory explosion and consequential crash of the tool because it
  ran out of memory. This has now been fixed by introducing a configuration
  option (max_test_cmd_output) and default limit of 10 Mbyte.
• Fix console spam when the plugin compile the software under test. The spam
  is replaced by a timestamp that is printed once every minute to indicate to
  a CI service that it is alive and to the user how long it has been
  compiling. If the compilation fail it will automatically re-start but this
  time print all output to the console for the user to understand what went
  wrong.
• Fix lockup when there are test cases with the same name.

Drizzle

New features for dextool mutate

• Check that all files to mutate are writable.
• Check periodically that the test suite, without any mutants, execute OK. If
  not, rollback the last tested mutants.
• Save what test_cmd killed a mutant. This is only activated if test case
  analyzers are used.
• Prioritize testing of unknown mutants.
• Check overload/halt conditions when running a schema for each mutant that is
  tested. Previously it was only done when selecting a new schema.
• Construct larger mutant scheman. Previously they where at most one
  translation unit. Now they can span multiple translation units (e.g. *.cpp).
  This mean that each schema contain more mutants thus the overall mutation
  testing time is reduced.
• Continuously save the result (every 20minute) the tested mutants of a
  schema. This should reduce the overall memory usage and means that if the
  schema is interrupted only the result from the last 20 minutes (at the most)
  are lost.
• Modernize the html layout theme.
• Prioritize mutants to test based on how much they affect the source code in
  order to quicker give actionable feedback to the user. Based on the
  assumption that a large mutant that survive is important to kill.
• Change high interest mutants to reporting those that affect the most src
  code and has survive. Previously it was those that had survived the most
  test suite executions but this was never used by the users. The hope is that
  those that are reported as high interest are actually useful, interesting
  to look at and kill.
  Also allows the number of high interest mutants that are shown to be
  changed.
• Automatically force a re-analyze of all files when a new version of the tool
  is installed and dextool mutate analyze is executed.
• Automatically save the mutation score after each test run when all mutants
  are tested. This is then used to plot a trend.
• The HTML report and console print a SyncStatus. It shows how "in sync" the
  tested mutants are with the code changes and test suite changes.
  The further apart they are the less "trustworthy" is the report. Because
  lets say the test suite is changed to now kill a mutant that is marked as
  alive.

Fixes for dextool mutate

• Fix bugs in schema code generation such that more scheman compile.
• Fix bug where the dependency tracking via included headers used the wrong
  path for headers included from the current directory (#include "foo.hpp").
• Fix bug where a mutant that span multiple lines in the html report is shown
  on one line. It makes it hard to read.
• Removed delete of case branches in switch statements. The language C/C++ and
  the Clang AST make it hard to generate correct mutants and schematan. The
  effort needed to make it work in all cases is still great. It is further a
  bit unnecessary because SDL will delete individual statements in the
  branches thus well. It doesnt't really give that much to be able to delete
  whole branches.
• Not all mutant schematan for binary operators where generated which meant
  that speedup opportunities where lost. With this fix all binary operators
  should have a schema generated for them which speeds up dcr, ror, aor mutant
  operators.
• Fix the size of the database when schematas are saved by compressing them on
  the fly. Reduced the size up to 90% of a database.
• Add support for C++17 structural binding in schematas.
• Enable scheman which mutate const variables in C/C++.
• Fix mutation of C++ lamba expressions.
• Fix libclang bindings. The enum CXCursorKind is not backward or forward
  compatible. Because of this dextool now have a binding for each version it
  supports. What happend where that the analyzed AST could, if the wrong
  binding is used, contain seemingly random nodes which mean that the
  mutations where waaaay off.
• Fix schematas inside return-statements and when a condition contain pointers.
• Measuring the runtime of the test suite is now done multithreaded. It is too
  slow and pessimistic to do single threaded. It was previously changed to
  single threaded because the load of the host computer could vary and lead to
  falsely classify mutants as timeout. But with the feature --load-behavior slowdown this is no longer a problem. Thus changing back to multithreaded
  to speedup both the start and overall mutation testing.
• Remove returnFalse/returnTrue mutants because they are redundant, overlap
  with normal true/false.
• Stop generating the equivalent mutant sdl when it deletes empty scopes.

Nice Weather

This release, as the previous once, has focused on the mutation testing plugin.
The focus has been on improving the mutation testing speed for software that
have just started to write tests to those that have a full test suite with 100%
statement coverage. The road has been long but we are finally at the finish line.

New features for dextool mutate

• Replaced restrict in the configuration with include and exclude to
  allow a user to both box in what to test and to selectively exclude parts.
• Automatically run SQL vacuum on the database to reduce its size. Currently
  triggers when there is a new tool release or schematan are removed.
• An estimation (trend) is calculated using a kalman filter which is ran over
  the historical data of when a mutant where discovered. The intention is to
  give the user a feeling of how the mutation score and thus test suite
  quality is trending based on the latest code changes.
• Add an admin command to reset a mutant subtype.
• Add an admin command to clear the worklist.
• Allow a user to specify retest of old mutants as a percentage of the total.
  This is to allow a user to easier predict when all mutants have been
  re-tested.
• Reduce the database size by only saving those mutants to the database that
  the user has specified.
• Save the test suites execution time in the database to reduce the number of
  times it needs to be executed during the warmup phase of testing mutants.
• Save the mutation testing score in the database when all mutants have been tested (worklist is zero).
  Use this mutation score show a bar graph of how it has changed over time.
• Add --load-threshold and --load-behavior to easier integrate the plugin
  in Jenkins as a good citizen by allowing the plugin to detect when the
  build server is overloaded and either slow down or stop mutation testing.
  Mainly intended to be used for mutation testing of the master branch which
  should have a lower priority than pull requests.
• Add test case stats, uniqueness and test cases to the JSON report.
• Save the exit code of the test suite to the database. This will be used in
  the future to improve the reports because users are interested in the
  difference between them such as when the test suite SEGFAULT verses exits
  with a failure.
• Add tracking of the test suite to allow dextool to know if old mutants need
  to be tested because the test suite have changed.
• Use coverage information to quickly mark mutants that exists in functions
  that are never executed by the test suite as alive.
• Add AORs as a simplified AOR. Instead of producing four mutants when an
  arithmetic operator is used it only produce one.
• Add dependency tracking to reduce the number of files that need to be
  re-analysed. It is expensive to analyze modern C++ thus by tracking the
  #include it is possible to see exactly which files need to be analysed
  instead of as previously, always analyse every file.
• Enable analysis of code that isn't parsed correctly by the clang frontend by
  the analysis to proceed even though there are compilation errors.

Fixes for dextool mutate

• Fix the table sorting in the HTML report to be numerical instead of lexical.
• Fix the mutation score constantly "resetting" because tests are added/removed.
• Fix schemata bug for switch statements. The scheman where not equivalent to
  the source code mutants.
• Fix bug where the admin command to stop the timeout testing didn't actually work.
• Removed case branch delete from mutation operator DCR because it is a
  duplicate of what SDL do for case branches.
• Fix memory usage when analysing. A pool is now used for the mutant AST where
  all nodes are allocated. The pool is then disposed as soon as it isn't
  needed.
• Fix bug in MemFree which calls malloc_trim and GC free to reduce the heap.
• Always interpret the build command as a shell command to execute. Reduced or
  removes the need for a build.sh because the most common use where just to
  change directory and call make.
• Separately record the time it took to test a mutant as in compile and
  execute test suite time.
• Print what is wrong in the configuration file when it fails to parse.
• Fix SQL query for retrieving the oldest mutants by telling it to interpret
  as a datetime and not string.
• Remove mutants that only delete whitespaces. These are obviously equivalent
  mutants thus not interesting.
• Fix AOR mutation operator using the modulo operator when either side is a
  floating point type by not generating this type of mutants.
• Fix bug when counting the number of mutants a test case killed. Mutants
  where duplicated.
• Fix mutation checksum calculation which thus repaired the previously broken
  comment robustness.
• Fix UOI negation by using double negation instead of removing !. This is
  because ! changes the type to a boolean and C++ is picky about the types.

Breaking changes to dextool mutate

• Removed the mutation operator COR and DCC.
• Removed the deprecated --level for the report subcommand.
• Remove restrict from the configuration file.

Camp Fire

This release is mainly a preparation of introducing mutation schemata by refactoring the analyse phase into more steps. A side effect of this is that mutation schematas requires the mutants to be valid. A problem that has existed is that the mutants produced by the SDL mutation operator had a medium probability of introducing either invalid code or undefined behaviour. This has been handled by the refactored analyser.

New features

• introduce profiling of the analyse and report stage. This can be used by a user to understand "why" certain operations take time.
• Print a progress message when removing orphaned mutants. This can take a long time thus this printing avoid e.g. a Jenkins timeout and helps a user not get impatient.
• Mutations that would occur inside a C/C++ macro is now removed except a small list of hard coded exceptions such as TRUE and FALSE.
• The test command runner has been able to execute the tests in parallel since the v2.0.0. It is now possible to configure it to stop execution as soon as one of the test commands signal failure. This can reduce the time it takes to test a mutant at the expense of the report of the test cases being less accurate.
• The test command runner automatically prioritize test cases to execute by how often they kill mutants. It means that it will over time execute the best test commands first.

Fixes

• speedup database access by adding indexes. How much it helps depends on the size of the database but empirical data shows between a 20-90% speedup.
• don't generate SDL mutants that remove returns in functions that has a return type other than void.
• slow down the analyse worker threads if the database thread that store the result is overloaded.
• set the mutations operators to use to "any" if the user doesn't specify an operator.
• Reset the timeout iterator to zero when a mutation operator has finished testing all its mutants. Previously the value where kept thus it created a strange timeout behaviour when switching between operators.
• Eliminate all SDL mutants that resulted in undefined behaviour.
• Eliminate useless mutants generated from the UOI mutation operator. Instead of fully implementing the academical definition of UOI it now only removes negation. This makes UOI useful because most mutants that it generates are productive.

Deprecation

• remove the COR mutation operator. It was rarely used and produced hard to understand mutants.