Add a way to run tests in parallel

[safareli]

I was thinking on adding describePar and describeParOnly. so Description is marked as parallel and during execution in _run we are run producers in parallel. it would be very useful in case ps-spec is used for running integration tests and some tests don't effect each other.

so if you have:

describe "delay" do
  it "can delay for 100ms" do
    delay $ Milliseconds $ 100.0 * 1.0
  it "can delay for 200ms" do
    delay $ Milliseconds $ 100.0 * 2.0
  it "can delay for 300ms" do
    delay $ Milliseconds $ 100.0 * 3.0
  it "can delay for 400ms" do
    delay $ Milliseconds $ 100.0 * 4.0
  it "can delay for 500ms" do
    delay $ Milliseconds $ 100.0 * 5.0

which takes 1500 ms, you can apply this change:

-describe "Foo" do
+describePar "Foo" do

and it will take 500ms.

Main issue would be logging from tests which are executing in parallel, as you would have no way to know from which one a logged message is coming from.

[owickstrom]

Hspec has a parallel combinator that I think you can wrap any Spec with: https://hspec.github.io/parallel-spec-execution.html

Maybe could borrow some design from there?

Regarding logs, are you thinking of stuff printed by the code under test, rather than purescript-spec's output?

[safareli]

yes spec logs just fine if you fork tests, issue is when the code under test is logging stuff

[safareli]

I like the parallel and sequential contaminators from hspec, and will try to do similar thing instead of desribePar*.

Tut the issue with logging still is there, I think instead of it :: String -> Aff Unit -> Spec Unit
we can do something like this:
it :: String -> WriterT String Aff Unit -> Spec Unit or
it :: String -> ReaderT (String -> Eff Unit) Aff Unit -> Spec Unit

so you can still log to console but it has test name as prefix, or you just get log when test finishes/fails, instead of output being polluted by random strings from all parallel tests

[owickstrom]

When it comes to logging from the code under test, I think it's a slippery slope to provide functionality in purescript-spec, and that we shouldn't implement any special support for it. Rather, we can document parallel and note that any code that prints to the console will need to synchronize, or perhaps log to different files or with different prefixes.

[owickstrom]

@felixmulder, any thoughts on logging?

[safareli]

If ps-spec itself is not providing that then probably Spec should become be something like this:

type TestName = String
type Spec = Spec'  Aff
type Spec' m t = State (Array (Group (ReaderT TestName m Unit))) t

this way one an use define there own spec:

type SpecWithCustomLoging = Spec'  (ReaderT (String -> Effect Unit) Aff)

[felixmulder]

This one's hard because one can redirect stdout/stderr in application code so taking over them is something that can have adverse effects... Let me marinate on this

…

On Mon, 10 Dec 2018 at 20:04, Irakli Safareli ***@***.***> wrote: If ps-spec itself is not providing that then probably Spec should become be something like this: type TestName = String type Spec = Spec' Aff type Spec' m t = State (Array (Group (ReaderT m TestName Unit))) t this way one an use define there own spec: type SpecWithCustomLoging = Spec' (ReaderT Aff (String -> Effect Unit)) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#80 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABdYwW2wx6zAArc7ugkvSCm14lpT_M6Vks5u3rA7gaJpZM4ZKDKL> .

[safareli]

I think if we add parallel/sequencial we should also provide a way for users to use custom monad then just Aff so it's possible for advanced users to use some custom logging, for example if we add m param to Spec, then _run will have type like this:

_run
  :: forall m
   . Config
  -> Spec' m Unit
  -> (NonEmptyArray Name -> m ~> Aff)
  -> Producer Event Aff (Array (Group Result))

This way someone can use ReaderT (String -> Effect Unit) Aff as m and in _run use Name as prefix of all logs, or use WriterT [String] Aff as m and just log everything after each test is finished/failed.

if we can provide hoistSpec :: forall m g. (m ~> g) -> Spec' m ~> Spec' g or similar that could might also help (I was not able to write this as you can't change type of state in State)

[safareli]

so i have added hoistSpec #82 and par&seq functions (like in hspec)

take a look at this example for how it can be used to for logging (nave version vs writer vs reader)

https://github.com/purescript-spec/purescript-spec/blob/905769151190fef6cbce5168167b5f815f38a937/test/Test/Spec/HoistSpec.purs

and here you can see travis logs https://travis-ci.org/purescript-spec/purescript-spec/builds/466439951#L1285