Line-based testcase reducer
Clone or download
jschwartzentruber Fix CheckOnly strategy.
The result of main() should reflect whether or not the testcase was
interesting.
Latest commit cd8688c Jul 11, 2018

README.md

Build Status Build Status codecov

Using Lithium

Lithium is an automated testcase reduction tool developed by Jesse Ruderman.

Most of what you need to know to use Lithium is in one of these pages:

Lithium's algorithm

By default, Lithium uses a clever algorithm that's efficient at reducing most large testcases. For a testcase with 2048 lines, it will try removing each chunk of size 1024, permanently removing it if it is still 'interesting'. It then does the same for each chunk of size 512, then 256, all the way down to chunks of size 1. It then does as many additional rounds at chunk size 1 as necessary until it completes a round without removing anything, at which point the file is 1-minimal (removing any single line from the file makes it 'uninteresting').

If n is the size of the testcase and m is the size of the 1-minimal testcase found by Lithium, then Lithium usually performs O(m ⋅ lg(n)) tests with a total test size of O(m ⋅ n). See the analysis of Lithium's algorithm for more information and proofs.

To keep m small, make sure Lithium's smallest removals won't introduce fatal syntax errors into the file it is trying to reduce. For example, don't use --char when trying to reduce a long sequence of JavaScript statements, and don't feed XHTML to Lithium. (Convert it to HTML first and let Firefox's tag-soup parser sort out the invalidity, or use serializeDOMAsScript.)

Command line syntax

pip install lithium-reducer
python -m lithium [options] interestingness-test.py [arguments for interestingness test]

Command line options

--testcase=filename
Tells Lithium which file to reduce. By default, it will assume the last argument to the interestingness test is the file to reduce.
--char (-c)
By default, Lithium treats lines as atomic units. This is great if each line is a JavaScript statement, but sometimes you want to go further. Use this option to tell Lithium to treat the file as a sequence of characters instead of a sequence of lines.
--strategy=[check-only,minimize,minimize-balanced,replace-properties-by-globals,replace-arguments-by-globals,minimize-around]
"minimize" is the default, the algorithm described above. "check-only" tries to run Lithium to determine interestingness, without reduction. For the other strategies, check out this GitHub PR.
--repeat=[always, last, never].
By default, Lithium only repeats at the same chunk size if it just finished the last round (e.g. chunk size 1). You can use --repeat=always to tell it to repeat any chunk size if something was removed during the round, which can be useful for non-deterministic testcases or non-monotonic situations. You can use --repeat=never to tell it to exit immediately after a single round at the last chunk size, which can save a little time at the risk of leaving a little bit extra in the file.
--max=n. default: about half of the file.
--min=n. default: 1.
What chunk sizes to test. Must be powers of two. --max is useful if you're restarting Lithium after it has already gone through a few rounds. --min is useful if you're reducing HTML and want to do the final by hand.
--chunksize=n
Shortcut for "repeat=never, min=n, max=n". --chunksize=1 is a quick way to determine whether a file is 1-minimal, for example after making a change that you think might make some lines unnecessary.

Hints

If you find a non-deterministic bug, don't despair. Lithium will do fine as long as you make the bug happen at least 70% of the time. You can repeat the test either within the application, by adding a loop or reload in the testcase (outside of the DDBEGIN/DDEND markers!), or outside of the application, by adding a loop to the "interestingness test" script.

Requirements

Lithium is written in Python and requires Python 2.7 or 3.4+.

Various versions of Lithium have been used successfully with:

  • Windows 7 / 10 via MozillaBuild 3.0 and up. This comes with 2.7 and 3.6.
  • macOS Sierra 10.12 + Python 2.7
  • Ubuntu Linux 14.04 / 16.04 + Python 2.7 / Python 3.4+

It may or may not work in Windows XP/Vista anymore.

Credits