Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add labelling of test cases #262

Merged
merged 2 commits into from Apr 23, 2019
Merged

Conversation

@ruhatch
Copy link
Contributor

ruhatch commented Mar 29, 2019

This adds label and collect similar to QuickChecks coverage support. This is achieved using an extra constructor on the existing Log type, which collects Labels. You could similarly add a Class constructor to implement QuickChecks classify and I will happily make a PR for that too if this approach seems reasonable.

Display of labels could also use source information to be printed under the label command, similar to annotates existing behaviour. If this gets merged I can add that and the other extension mentioned above as issues.

@ruhatch ruhatch force-pushed the ruhatch:ruhatch/labels branch from 2ddbe90 to 52054b0 Mar 29, 2019
@HuwCampbell

This comment has been minimized.

Copy link
Contributor

HuwCampbell commented Mar 30, 2019

Thanks for submitting this :)

Just a heads up that it overlaps quite a bit with #253, which is currently not in a fully working state, but is slowly wending through the review process.

-- of test runs that produced each label.
--
label :: MonadTest m => String -> m ()
label =

This comment has been minimized.

Copy link
@HuwCampbell

HuwCampbell Mar 30, 2019

Contributor

I think this is a better design choice.

This comment has been minimized.

Copy link
@moodmosaic
@ruhatch

This comment has been minimized.

Copy link
Contributor Author

ruhatch commented Apr 1, 2019

@HuwCampbell thanks for the heads up! Looks like #253 wouldn't create too many merge conflicts and covers (haha) the other parts of coverage testing that this PR misses. I think that PR could move the Classifiers into a Log constructor and then do the Classifier monoid operations at the end, which would align it with this approach 🤷‍♂

@charleso charleso mentioned this pull request Apr 21, 2019
@jacobstanley jacobstanley force-pushed the ruhatch:ruhatch/labels branch from 52054b0 to 1df0f13 Apr 22, 2019
@jacobstanley

This comment has been minimized.

Copy link
Member

jacobstanley commented Apr 22, 2019

@HuwCampbell I'm going to take over this PR and integrate the two labelling approaches. I've rebased and solved the merge conflicts and will build on top of that.

@jacobstanley jacobstanley force-pushed the ruhatch:ruhatch/labels branch from 5051aeb to 64928b6 Apr 22, 2019
@jacobstanley

This comment has been minimized.

Copy link
Member

jacobstanley commented Apr 22, 2019

Everything is now being recorded in the log and coverage is calculated prior to reporting.

A screenshot of the coverage example in action:
Screenshot 2019-04-22 at 7 01 53 PM

@moodmosaic

This comment has been minimized.

Copy link
Member

moodmosaic commented Apr 22, 2019

Crazy stuff!

@jacobstanley

This comment has been minimized.

Copy link
Member

jacobstanley commented Apr 22, 2019

I have pondered whether we should hide the coverage results if all conditions are satisfied (the last test prop_cover_bool), just to make the output a bit quieter when everything is successful.

@moodmosaic

This comment has been minimized.

Copy link
Member

moodmosaic commented Apr 22, 2019

I think verbose output should be fine in case everything is successful, like in that test for example. It's nice that it shows what the distribution was. Perhaps we can have an option to turn this on/off if needed (RunnerConfig comes to mind).

@jacobstanley

This comment has been minimized.

Copy link
Member

jacobstanley commented Apr 22, 2019

Yeah i was just pondering doing something better than IO Bool for test types. It is great for ghci though, which is the main place I use it.

I suppose we could do something awful with IORef 🙈

@jacobstanley jacobstanley force-pushed the ruhatch:ruhatch/labels branch from 64928b6 to b5c0137 Apr 23, 2019
@jacobstanley jacobstanley merged commit 3dd679d into hedgehogqa:master Apr 23, 2019
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
charleso added a commit to hedgehogqa/scala-hedgehog that referenced this pull request Aug 22, 2019
Issue 71: #71

Taken from a combination of the following PRs:

- hedgehogqa/haskell-hedgehog#253
- hedgehogqa/haskell-hedgehog#262
charleso added a commit to hedgehogqa/scala-hedgehog that referenced this pull request Aug 22, 2019
Issue 71: #71

Taken from a combination of the following PRs:

- hedgehogqa/haskell-hedgehog#253
- hedgehogqa/haskell-hedgehog#262
charleso added a commit to hedgehogqa/scala-hedgehog that referenced this pull request Aug 22, 2019
Issue 71: #71

Taken from a combination of the following PRs:

- hedgehogqa/haskell-hedgehog#253
- hedgehogqa/haskell-hedgehog#262
charleso added a commit to hedgehogqa/scala-hedgehog that referenced this pull request Aug 22, 2019
Issue 71: #71

Taken from a combination of the following PRs:

- hedgehogqa/haskell-hedgehog#253
- hedgehogqa/haskell-hedgehog#262
charleso added a commit to hedgehogqa/scala-hedgehog that referenced this pull request Aug 22, 2019
Issue 71: #71

Taken from a combination of the following PRs:

- hedgehogqa/haskell-hedgehog#253
- hedgehogqa/haskell-hedgehog#262
charleso added a commit to hedgehogqa/scala-hedgehog that referenced this pull request Aug 22, 2019
Issue 71: #71

Taken from a combination of the following PRs:

- hedgehogqa/haskell-hedgehog#253
- hedgehogqa/haskell-hedgehog#262
charleso added a commit to hedgehogqa/scala-hedgehog that referenced this pull request Aug 22, 2019
Issue 71: #71

Taken from a combination of the following PRs:

- hedgehogqa/haskell-hedgehog#253
- hedgehogqa/haskell-hedgehog#262
charleso added a commit to hedgehogqa/scala-hedgehog that referenced this pull request Aug 22, 2019
Issue 71: #71

Taken from a combination of the following PRs:

- hedgehogqa/haskell-hedgehog#253
- hedgehogqa/haskell-hedgehog#262
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.