Release 3.21

• table-generator computes scores according to SV-COMP'24 scoring scheme.
  This changes only the scoring for witness-validation results.
• Support for property files at HTTP(S) URLs in table-generator.
  Tables can already be produced not only from local result files, but also from files that are downloaded on the fly.
  Now this also works for results with property files.
• Fix for tool-info module witnesslint.

Release 3.20

• Two tool-info modules improved.

Release 3.19

• Tool-info modules can now provide URLs that will be used for links in HTML tables.
  There are two new methods, project_url() and url_for_version() that can be implemented, and table-generator will put links to these URLs in the summary table of the HTML tables (for the tool name and the version). Most existing tool-info modules were extended with project_url().
• Many new and improved tool-info modules.

Release 3.18

The big change in this release is the long-awaited support for cgroups v2!
Please refer to the installation instructions for how to use it (on Ubuntu/Debian, installing our package is enough). Note that this of course has not been tested yet on as many different systems as our support for cgroups v1, so there might still be some rough edges. Please provide feedback if you encounter any problems or have questions. On systems with cgroups v1, everything should work the same way as before.
Thanks a lot to @globin for providing the implementation of this!

There are also some other minor improvements:

• If the system administrator has configured a lower frequency limit than what the CPU supports, BenchExec now reports this limit as the CPU speed.
• Fixes in table-generator for result files that conform to the specification but were not produced by benchexec.
• A few minor improvements for better integration with Podman containers.

Release 3.17

• Even more robust handling of child cgroups created within a run
  Despite the improvements from BenchExec 3.13 there could be crashes because sometimes cgroups vanish while we iterate over them. Avoiding this did not fully work, maybe due to some delays in the kernel, so now we handle this.
• Actually fix handling of non-printable characters in environment variables.
  The fix that was part of BenchExec 3.12 only worked if a non-printable character was the first character of the value of the environment variable.

Release 3.16

This release works only on Python 3.7 and newer!

• Improve performance of the "Merging results" step of table-generator.
  In the common case of all run sets having the same set of tasks, this step is now much faster.
• Avoid deadlock in table-generator for cases with many child processes and heavy system load.
• On systems with many cores and if there are many input files or columns, table-generator will now use more than 32 child processes.
• Support the scoring schema used in SV-COMP'23 for witness validators, and show the witness category as part of the task identifier in tables.
  Note that just like for the category correct-unconfirmed external postprocessing of the results is necessary for this.
• If the pattern inside a <propertyfile> tag in the benchmark definition does not match a file, BenchExec now terminates with an error instead of just logging a warning.
• Improved logging in table-generator.
  The log messages about missing run results and missing properties, which could occur many times in certain use cases, are now omitted. Instead, some numbers about the size of the resulting tables and the number of missing results will be logged.

Release 3.15

• Updated installation instructions for Debian.
  Manual installation of the package is needed because dpkg on Debian is incompatible with our PPA on Launchpad.
• Some improvements to tool-info modules.

This release does not change the minimum supported Python version,
but we would like to remind you that BenchExec will soon stop supporting Python 3.6.

Release 3.14

• Added a workaround for the known glibc deadlock described in #656.
  In most cases BenchExec should detect the deadlock and continue benchmarking after a timeout of 60s.
• We noticed a performance problem in Python that affects benchexec in container mode on machines with many cores and added an optimization that reduces its impact.
• Improved handling of non-existent mount points.
  This makes BenchExec easier to use within Podman containers, and we now recommend Podman over Docker in our documentation.
• table-generator no longer attempts to spawn a large number of threads (failing due to the limit on open files) on machines with many cores.
• Many new and improved tool-info modules.

This release does not change the minimum supported Python version, but we would like to remind you that BenchExec will soon stop supporting Python 3.6.

Release 3.13

• More robust handling of child cgroups created within a run
  Note that it is not safe to execute an untrusted tool inside a BenchExec run and give it access to cgroups (e.g., with --no-container or --full-access /sys/fs/cgroups).
  However, some users want to execute trusted tools with cgroup access (e.g., nesting runexec) and we aim to support this.
  It was found that BenchExec could hang or crash in certain cases where child cgroups were created by the process within a run and used to freeze processes (done by nested runexec, for example), or were made inaccessible using file-system permissions.
  This lead to incomplete run cleanup and processes left running or hanging in an unkillable state.
  This release fixes such situations and users who allow cgroup access within runs are strongly recommended to upgrade.
• Fix incomplete rewriting of numbers into roman numerals in the new LaTeX output for statistics.

This release does not change the minimum supported Python version, but we would like to remind you that BenchExec will soon stop supporting Python 3.6.

Release 3.12

• Compatibility with Python 3.10
  Note that it is expected that this is one of the last releases that supports Python 3.6.
• Export of statistics as LaTex commands in table-generator:
  All those values that exist in the statistics table on the Summary tab of our HTML tables can be exported as a series of LaTeX commands with --format=statistics-tex.
  These can then be included in a paper and the commands can be used to retrieve the statistics values (documentation).
  The script contrib/statistics-tex.py that provided a subset of this functionality so far is removed.
  Thanks to @Sowasvonbot for implementing this!
• Slight improvements for mounting overlayfs:
  • Avoid redundant mount points that could prevent certain nested mounts.
  • Avoid a warning in the kernel log.
• Fix handling of non-printable characters in environment variables.
• Slight improvements for the HTML tables:
  • Use of color for categories in filters
  • More informative tooltip in quantile plots (thanks @leventeBajczi)
• Added and improved tool-info modules