Add a runTests function

[erikd]

And actually use it in the Hedhehog tests.

Closes: #97

@thumphries @jystic @nhibberd

[nhibberd]

Probably good to keep the same syntax style here, start of list on previous line that is

[erikd]

👍 Fixed the formatting in the comment for runTests as well.

[thumphries]

I feel a bit weird exporting a function that calls exitWith from the top level, especially with an innocuous name like runTests. Wondering if it would make more sense from Hedgehog.Main or something like that? Or a name with suite or main in it?

(not requesting changes per se, just wondering how you two feel about it)

[erikd]

I'm not married to the name, I just don't want to have to redefine this function in every test suite I write. I also think what you said makes a lot of sense. So how about:

• We rename it runTestsuie.
• Move it to Hedgehog.Main and not export if from Hegdehog requiring people to explicitly import Hedgehog.Main.
• Since if calls exitFailure on any failure, maybe it should also call exitSuccess on success. Ie this function always terminates.

[erikd]

Hmm, now that I think about it, I'm mot so sure about the last part. I could imagine wanting to do:

   runTests
        [ Test.Wibble.One
        ]
  -- Only run the following tests if the previous one passed.
   runTests
        [ Test.Wibble.Two
        , Test.Wibble.Three
        ]

[moodmosaic]

Move it to Hedgehog.Main and not export if from Hegdehog requiring people to explicitly import Hedgehog.Main.

Hmm... Isn't this a little bit weird? Perhaps it could be exposed as 'internal' from Hedgehog itself so we keep importing only the essential stuff:

import           Hedgehog
import qualified Hedgehog.Gen as Gen
import qualified Hedgehog.Range as Range

[erikd]

The idea of putting this in a separate Hedgehog.Main is growing on me. Maybe we should also provide more than one runner. Hedgehog.Mainmight provide fhe functions:

setLineBufferring :: IO ()

-- Run the tests and return `True` if they all passed, `False` otherwise.
runTestsuite :: [IO Bool] -> IO Bool

-- If any test fails, exit with `ExitFailure`.
runTestsuiteExitOnFail :: [IO Bool] -> IO ()

-- This _always_ exits, `ExitFailure` if any test fails, `ExitSuccess` otherwise.
runTestsuiteExit :: [IO Bool] -> IO a

[moodmosaic]

Are we 100% sure about the Main part? Not that I can currently come up with an alternative...

[erikd]

@moodmosaic The import you have in your example are the ones you would put in an actual test suite. I usually keep actual tests out of the top level test runner. With this new functionality, my top level test runner would look like:

import qualified Hedgehog.Main as HMain

import qualified Test.Wibble.One
import qualified Test.Wibble.Two
import qualified Test.Wibble.Three

main :: IO ()
main = do
  HMain.setLineBuffering
  HMain.runTestsuiteExit
    [ Test.Wibble.One.tests
    , Test.Wibble.Two.tests
    , Test.Wibble.Three.tests
    ]

I'm not married to the name Main. Might be Run or Runner instead. Or maybe with these new names and type signatures we can just export them from Hedgehog.

Update: I think the new naming and type signatures are specific enough for these to be exported from Hedgehog.

Update 2: Exposing setLineBuffering is probably a waste. Its a cheap operation and these test runners should just set line buffering regardless.

[gwils]

I don't like the name Hedgehog.Main, but Run or Runner I could get behind. It just feels a bit wrong to me for a library (not an executable) to export a module called Main. I would prefer that this be exported from Hedgehog rather than a separate module regardless.

My preference is the version that calls exitFailure on failure but does not call exitSuccess on success. As Erik has pointed out, you may wish to call many test runs sequentially.

[erikd]

My preference is the version that calls exitFailure on failure but does not call exitSuccess on success.

This is why my (strawman) proposal now has three versions of the runner function :).

[moodmosaic]

Nitpick: there seems to be an extra space between sequence.
Nitpick: will occurs two times in a row on that sentence.

[erikd]

Yeah, this PR will not go ahead as it is anyway.

[thumphries]

Closing this based on Slack discussions we had back in March.

The decision was to not rush into exporting a top-level runner, since public exports are forever and we all had some reservations about the current design. There is also the question of terminal encoding, Windows, etc to consider.

[jacobstanley]

💯