AssertionError if initial run is not interesting

[Technius]

When I try to run picire with an uninteresting initial run, it crashes with an assertion failure:

Initial test contains 4 lines
Iteration #0
Run #0
	Config size: 4
Traceback (most recent call last):
  File "REDACTED/picire/.venv/bin/picire", line 8, in <module>
    sys.exit(execute())
  File "REDACTED/picire/picire/cli.py", line 286, in execute
    out_src = reduce(args.src,
  File "REDACTED/picire/picire/cli.py", line 249, in reduce
    min_set = dd(list(range(len(src))))
  File "REDACTED/picire/picire/dd.py", line 63, in __call__
    assert self._test_config(config, (f'r{run}', 'assert')) is Outcome.FAIL
AssertionError

I expected picire to exit and state that the initial run is uninteresting.

Steps to reproduce

Here is the example I used:

(.venv) picire % cat foo.txt
a
b
c
d
(.venv) picire % cat dumb_tester.sh
#!/usr/bin/env bash

exit 1
(.venv) picire % picire -i foo.txt --test ./dumb_tester.sh

I'm using commit 63668bb0409f4078ff84564065d161661bd388ae

Misc

I'm currently using the following patch as a workaround:

diff --git a/picire/dd.py b/picire/dd.py
index 888679d..ffeef28 100644
--- a/picire/dd.py
+++ b/picire/dd.py
@@ -60,7 +60,12 @@ class DD(object):
             for run in itertools.count():
                 logger.info('Run #%d', run)
                 logger.info('\tConfig size: %d', len(config))
-                assert self._test_config(config, (f'r{run}', 'assert')) is Outcome.FAIL
+                outcome = self._test_config(config, (f'r{run}', 'assert'))
+                if run == 0 and outcome is Outcome.PASS:
+                    # Initial run is not interesting, so we can't expect any
+                    # reductions or subsets to be interesting either.
+                    break
+                assert outcome is Outcome.FAIL

                 # Minimization ends if the configuration is already reduced to a single unit.
                 if len(config) < 2:

but it cannot distinguish between an uninteresting initial run and an interesting run that requires all inputs.

P.S. This tool is great! It's saved me a lot of debugging time.

EDIT: updated the patch

[renatahodovan]

Hi @Technius !

I'm glad to hear the you use and like picire! :)

The assert you hit is a sanity check at the beginning of every run to see if the config aimed to be reduced (still) reproduces the expected behaviour. If it doesn't, then there is no point to continue. This assert usually fires if the original input doesn't repro or if the input is flaky test case.

If picire is integrated in some framework and the assertion failure breaks the workflow, then I suggest to validate the input and the tester to see if they reproduce the expected behaviour and run picire afterwards only. Or, if you would like to run picire anyhow, simply disable the asserts with PYTHONOPTIMIZE=1.