-
Notifications
You must be signed in to change notification settings - Fork 108
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Output quickcheck seed on timeout #359
Comments
What about passing timeout to QuickCheck via |
@Bodigrim I'm not sure that applies to this issue. The goal isn't to set a timeout, the goal is for tasty-quickcheck to register an action that runs when tasty times out. |
If you use |
Are you suggesting that we should add Although maybe tasty-quickcheck can automatically find the current timeout and add |
Yes, this should be possible. |
It appears that Also, applying All in all, we can add a new command line argument |
We had a quickcheck test timeout (using tasty's
--timeout
option), but we can't repro it because we don't have the seed. If we had the seed, we could see what quickcheck generated and see if it generated a pathological input that we haven't handled.One possible solution that doesn't break the public API:
asy
) toapplyTimeout
TastyTimeout
exception to the test thread if the test thread takes too longasy
returns a result within 1 second (or some other short time), return that resultasy
and return a generic "Timed out" resulttasty-quickcheck
'srun
implementation to catchTastyTimeout
and return aResult
with the seed added to the descriptiontimeoutResult
definition intomkTimeoutResult :: TastyTimeout -> Result
A possible solution that changes the
IsTest
interface (I know maintainers don't want to make breaking changes, but I think this would maketasty
more flexible that can enable other future improvements):TestMeta
type containing hooks for test providers; currently, it would contain a single field:resultOnTimeout :: Maybe Result
IORef TestMeta
parameter torun
thatrun
can modify as it executesrun :: test -> IO Result
torun :: test -> IO (TestMeta, IO Result)
Old solution that doesn't work
Add a new
TastyTimeout
exception containing information of the timeout, which should be thrown instead of the exception thrown bytimeout
Break out
timeoutResult
intomkTimeoutResult :: TastyTimeout -> Result
Wrap the
tasty-quickcheck
runner withThis doesn't work because the test is run in a separate thread from the driver, and the timeout exception is handled in the driver thread and doesn't make it to the test thread. The test thread does get the
AsyncCancelled
exception thrown to it, but the driver thread never looks at the result of the test thread after the timeout, so it'll just get thrown away.The text was updated successfully, but these errors were encountered: