Always show seed when in verbose mode or in CI

[brandon-leapyear]

IMO it's much easier to debug a failure by passing in a --hypothesis-seed flag instead of editing source code to reflect the values in the "Falsifying example" output. Currently, the report outputting the seed is hidden when a test "fails normally":

hypothesis/hypothesis-python/src/hypothesis/core.py

Lines 756 to 766 in 32bf7aa

	# The test failed by raising an exception, so we inform the
	# engine that this test run was interesting. This is the normal
	# path for test runs that fail.
	tb = get_trimmed_traceback()
	info = data.extra_information
	info.__expected_traceback = format_exception(e, tb)
	info.__expected_exception = e
	verbose_report(info.__expected_traceback)
	self.failed_normally = True

hypothesis/hypothesis-python/src/hypothesis/core.py

Lines 1216 to 1222 in 32bf7aa

	if not (state.failed_normally or generated_seed is None):
	if running_under_pytest:
	report(
	f"You can add @seed({generated_seed}) to this test or "
	f"run pytest with --hypothesis-seed={generated_seed} "
	"to reproduce this failure."
	)

It would be nice to have an option that always shows this output (if there is a seed).

[Zac-HD]

Hmm. I get the surface-level reason for this, and I would definitely like to make sure that reproducing errors is easy.

To make sure that we solve the right problem, can you say a bit more about when this arises? Specifically, I would expect the database to solve this replay problem for you. If it doesn't, how about "always print the seed if settings.verbosity >= verbose?

[brandon-leapyear]

✨ This is an old work account. Please reference @brandonchinn178 for all future communication ✨

---

Oh I'm still not sure how the database system works. Does it require additional configuration? When an error occurs, is it automatically saved in the database? Once it's saved to the database is it automatically included in the list of tests to retry (even if we have phases=(generate,) only)?

Regardless, the issue comes up when an error happens in CI and we need to debug it locally. Is the idea that we should be sharing some database with CI? I think even if that's the intended usage, it'd be nice to make this process easy if you don't have the infrastructure to share databases between CI and local.

If it doesn't, how about "always print the seed if settings.verbosity >= verbose?

Sure, that could work too. But I don't see why this might be preferred over adding another option to explicitly opt in. We probably don't mind making hypothesis verbose in CI, but it seems like it'd be better to have independent flags

[Zac-HD]

Oh I'm still not sure how the database system works. Does it require additional configuration? When an error occurs, is it automatically saved in the database? Once it's saved to the database is it automatically included in the list of tests to retry (even if we have phases=(generate,) only)?

The database is fully automatic, with previously-failing examples replayed in the reuse phase.

Is the idea that we should be sharing some database with CI? I think even if that's the intended usage, it'd be nice to make this process easy if you don't have the infrastructure to share databases between CI and local.

You certainly don't need to share a database. If you already have Redis set up I'd share the database; if you already have CI caching set up I'd cache it; but otherwise don't bother.

Sure, [printing the seed in verbose mode] could work too. But I don't see why this might be preferred over adding another option to explicitly opt in. We probably don't mind making hypothesis verbose in CI [where the issue arises], but it seems like it'd be better to have independent flags

Reasons not to add an option unless we really need to: (1) there is a cost to users and maintainers from having more code and options to understand; (2) some people won't use it who would have needed it, just because they won't know in advance that it would have been useful.

Instead of adding an option, let's additionally print the seed if we're in CI, or in verbose mode.

[Zac-HD]

On further thought, this seems to be duplicating the print_blob setting, which is a more reliable solution to your problem than the seed. Since printing the example and the seed and the blob message seems over the top, I'm closing this as won'tfix.

[brandon-leapyear]

✨ _**This is an old work account. Please reference @brandonchinn178 for all future communication**_ ✨ <!-- updated by mention_personal_account_in_comments.py -->

--- Hm I don't think I was seeing that output in CI, but I can debug that. Just to be clear, this still doesn't address the issue I originally raised:

IMO it's much easier to debug a failure by passing in a --hypothesis-seed

flag instead of editing source code to reflect the values in the "Falsifying example" output. The @reproduce_failure decorator would still involve editing source code, right?

…

On Sat, May 14, 2022, 9:41 PM Zac Hatfield-Dodds ***@***.***> wrote: On further thought, this seems to be duplicating the print_blob setting <https://hypothesis.readthedocs.io/en/latest/settings.html#hypothesis.settings.print_blob>, which is a more reliable solution to your problem than the seed. Since printing the example *and* the seed *and* the blob message seems over the top, I'm closing this as won'tfix. — Reply to this email directly, view it on GitHub <#3312 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGUC75LIISPPP4VZXZKUJUDVKB56JANCNFSM5UQWTYIA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Zac-HD]

The @reproduce_failure decorator would still involve editing source code, right?

Yeah, if you want a totally hands-off approach then the best solution would be to set up a networked database as show here; at that point it's fully automatic.

[brandon-leapyear]

✨ This is an old work account. Please reference @brandonchinn178 for all future communication ✨

---

So under what circumstances would you expect one to see the seed and/or use --hypothesis-seed?