Integrate Hedgehog into Scalatest

[sageserpent-open]

BACKGROUND

I have used Scalacheck a lot in the past, and have gravitated over to the Scalatest flavour of it, using GeneratorDrivenPropertyChecks and native Scalatest assertions as opposed to Scalacheck properties.

This is all very nice, but I have the usual gripes about writing complex generators and finding that shrinkage doesn't preserve the test data invariants, so I end up disabling shrinkage and having to debug test failures with some quite complex failing test cases, or having to hand-roll shrinkers that I hope maintain the same invariants as the original generators.

ENTER HEDGEHOG

Having heard of Hedgehog and ZIO Test, I've given both a spin, seen that they do indeed offer shrinkage that maintains test case invariants, and have plumped for Hedgehog after writing a nasty inefficient test case generator to stress both frameworks.

THE STORY

What I'd like is to have Hedgehog integrated into Scalatest for the one-stop shopping experience that I'm used to with Scalacheck & Scalatest. It doesn't have to be a verbatim translation of the original tests, but should only require a straightforward translation - maybe a few method name changes, extra arguments, but no big rearrangement of existing test code.

THE SPIKE

I've produced the following PR to show how a simple integration could work and to give a feel for the 'before and after' look of the tests (slightly complicated by one of the tests being in legacy Scalacheck style rather than in Scalatest style, but there is another that is a direct translation).

See here: sageserpent-open/americium#1. Sorry about the cheesy title, I was in a skittish mood.

If there is interest, I'd be happy to help moving this PR over into this project, or breaking it out on its own.

I'm going to continue working on the spike anyway for now as I wish to use it anyway on some other projects, so I shall be filling out some syntax helpers to provide generators matching the usual Scalacheck generators / arbitraries - think of 'nonEmptyList', arbInt etc.

Please comment in this issue / ping via GitHub if there is interest.

[steinybot]

Great idea!

I also added some before/after style to the more traditional runner/ sbt test framework over in https://github.com/BotTech/scala-hedgehog-spines. There are a whole bunch of generators there too.

[sageserpent-open]

@steinybot Nice work there - you've already offered a set of readymade generators for integers, characters, strings etc, so no need for me to do that.

To avoid potential confusion, I should point out that when I used the phrase 'before and after', I was referring to how the tests looked in their Scalacheck form versus being cutover to Hedgehog - as opposed to supporting factored test setup and teardown activities.

Thinking about it, if the spike in Americium were to be brought over into Hedgehog, it should probably be split into the Scalatest integration - that could go into its own subpackage - and the generator automatic derivation piece.

Not everyone will want to use Scalatest as their default approach to testing - they may be used to the 'core' Scalacheck property style, which is what Hedgehog itself offers right now as a core concept. There are a lot of 'laws'-style tests in various Scalaz and Cats projects that use the property-based approach, so I would regard Scalatest style assertions as being more of an optional buy-in, hence the relegation to a subpackage.

That would also play well with another proposal here: #160 which I presume would have its own philosophy for writing tests.

Automatic generator derivation is something that I would imagine has universal appeal; I'd propose adding it to the core code.

Thoughts?

[charleso]

@dwestheide @steinybot @dwestheide I thought I might just put my 2c out, although given how I'm not really contributing on the project it should be taken with a massive grain of salt.

In theory I'd love to host/include the source code of all the sensible/possible integrations in this repository! It would avoid slightly different multiple versions/copies springing up, and ease friction. However, I've watched the binary compatibilities play out with ScalaCheck/ScalaTest (and previously way back with Specs2 and Scalaz) with horror and I don't know if there's a good solution using the current tooling. I'd hate to get in a position where people are complaining that we broke compatibility, or that it doesn't work on a different version. I'm a little naive in this though, and it might be a solved problem, or not really an issue in practice. I also realise that Hedgehog will never be as popular as ScalaCheck/ScalaTest so maybe it's a good problem to have.

All that said, it's probably still worth adding separate modules for those integrations, but this is the reason I've been a little hesitant in just embracing it. If people are keen and there is some interest on this ticket please send the PRs! My only request is we have a module for each major dependency (ie hedgehog-scalatest, hedgehog-minitest, etc).

PS Just as aside, having spent lots of time in the Haskell world too, using source builds really makes this issue more feasible to solve/avoid. You can effectively support every possible version (with some ugly uses of CPP tags) from a single release. I would love if Scala (and the JVM) used source builds as the primary/popular dependency management method, but I realise that's a very long way off.

PPS @sageserpent-open Sorry I have to run. I'm actually personally not a fan of automatic generator derivation, but I know others love it. having a separate module would be fine by me, but perhaps not in core (I just want to keep it as slim as possible).

[dwestheide]

Thanks for your input on this @charleso. There have indeed been some problems with binary compatibility with such integration modules. With ScalaTest and ScalaCheck, for example, you're often effectively tied to using exactly the versions of ScalaCheck and ScalaTest that were used to compile the released integration module. I agree that source dependencies may help alleviate this. However, we're still far from seeing Fury as the main build tool for Scala projects. :)

I think having such integration modules has its merits, despite the problems mentioned. I agree that they should always be in separate sbt sub modules. I'm going to open a PR for minitest a bit later which uses that approach.

As for automatic generator derivation, I would also prefer to see that in a separate sub module and keep the core as slim as it currently is.

[sageserpent-open]

@charleso Indeed, regarding being the nexus of a bunch of third party dependencies, it had also occurred to me subsequent to my last comment that perhaps consumers of Hedgehog might now want the large transitive dependency set that all these integrations would bring in, especially when they are only after a specific chunk of the offered functionality.

I suppose I could petition the Scalatest folk to put the integration in there alongside the Scalatest one - although a similar dependency argument (albeit of lesser magnitude) would apply there.

I'll play safe for now and leave it in Americium, along with the automatic generator derivation; I can make use of it right now that way and it doesn't rock the boat on the Hedgehog side. Americium is essentially my own mini-Guava dumping ground of utilities, so that fits my own use case well.

Perhaps I (or for that matter anyone else) could hive it out of Americium and make small, independent and focussed artifacts for the integration and for the automatic generator derivation. That was how Scalacheck-Shapeless was packaged, and that worked out fine for me as a past end-user.

I'll keep an eye an #160 and see how that pans out - perhaps these artifacts could be built as SBT subprojects in Hedgehog, or perhaps as completely separate SBT main projects in their own repositories.

If anyone wants to jump the gun on this, feel free, though.

On a tangential note, I noticed that 'GenT[X]' is invariant on its type parameter 'X'. Does that have to be the case - a casual glance seems to indicate that it should be covariant. I'll take a look to see whether this is possible with a view on raising a separate PR - would be useful for me, but if folk already know why this can't / shouldn't be done, let me know...

[kevin-lee]

Thank you so much @sageserpent-open @dwestheide and @steinybot for all your work and interest in Hedgehog!

As a user of Hedgehog for about two years, I agree with Charles. I like to see Hedgehog's integration with other test frameworks yet it would be much better if we have them as separate projects. As we all know, just like sbt Community-Plugins. 🙂

[dwestheide]

@kevin-lee As I understood @charleso he was arguing for separate modules (as in build artefacts, possibly sbt sub projects?), not for entirely separate projects with separate repositories, but maybe he can clarify. Personally, I think that the the overhead of a separate Github repository and pipeline is too big for a small integration module consisting of a single Scala source file — also, it would be nice to be able to release them together with the core module.

However, if you are strongly opposed to sbt sub projects in the same repo, we can do have a separate scala-hedgehog-minitest repo, too. I might need some help setting up the build and release pipeline, though. :)

[kevin-lee]

@dwestheide Oh I thought it was even better for you and others as you could have the full control of the project. 🙂
Yeah I might be wrong and @charleso can clarify.

Anyway, if you need help for setting up the release pipeline for your repo, I'm happy to help.
Just let me know please when you need it. 🙂

[dwestheide]

@kevin-lee I guess it might be better, I really don't know. :)

There are arguments for and against both approaches. For example, if it's a separate repo, I will have to keep track of changes in the core releases and release my own versions to keep up with it. I like projects that release all of their modules, including the optional ones, in lockstep, so all of them always have the same version number.

On the other hand, I guess that an optional integration modules will not see a lot of changes. As a sub project, it would constantly be released together with the rest, even if nothing has changed.

I'm okay with either approach. And thanks for offering to help out with setting up the release pipeline, if we decide to go that way. That's very much appreciated. :)

[charleso]

As I understood @charleso he was arguing for separate modules (as in build artefacts, possibly sbt sub projects?),

Yep, that's all I meant. I definitely don't think we should create any other "scala-hedgehog-xyz" github repositories for all the reasons that @dwestheide mentioned. There might be some (hypothetical) things we don't want to include at all on some grounds but that's a separate concern.

[kevin-lee]

That's good. Agree with @dwestheide on all the benefits from not having separate repos.
I probably thought too much about the authours rather than thinking about the project. 🙂

[dwestheide]

Nice! Having clarified this, it seems like #161 is already going in the desired direction, introducing an sbt sub project hedgehog-minitest. It would be great if someone could take a look at it. 🙂

[sageserpent-open]

This is no longer a pressing concern for me; please feel free to close (I'll leave it open for now in case others are actively interested).