Skip to content

Loading…

QuickCheck-like fact support #82

Closed
AlexBaranosky opened this Issue · 21 comments

3 participants

@AlexBaranosky
Collaborator

Maybe it could look something like:

(generated [a (midje/string) b (midje/string)]
  (fact
    (str a b) => (has-prefix a)))

Inspired by examples from ScalaCheck's site:
http://code.google.com/p/scalacheck/

@AlexBaranosky
Collaborator

Midje could come prepackaged with some generators (like midje/string), and also supply a Generator protocol that could be used to supply a generator for any type of data a user has.

@AlexBaranosky
Collaborator

It looks like ClojureCheck has already done a ton of the legwork on this.
https://bitbucket.org/kotarak/clojurecheck/src/85c314b1bd2b/src/main/clojure/clojurecheck/core.clj

I'm going to see if maybe we could utilize ClojureCheck as a library for this feature.

@AlexBaranosky
Collaborator

This Agitator product for Java looks interesting. I wonder how it could be incorporated into Midje to improve the way TDD is done? http://www.agitar.com/pdf/Paper-Agitar-ODT-TDD.pdf [PDF]

@AlexBaranosky
Collaborator

ClojureCheck:
http://kotka.de/blog/2010/06/ClojureCheck_is_back.html

I've contacted the maintainer of ClojureCheck about possibly pulling out the generator part of ClojureCheck into a separate namespace to use as a library in Midje.

Say you have a generative example like this, where any two strings concatenated together (str) always starts with the first string:

(generative [?a (midje/string) ?b (midje/string)]
  (fact
     (str ?a ?b) => (has-prefix ?a)))

Besides the generative testing approach, I'm also interested in being able to run diagnostics on an expression. What if by swapping the first word (the macro) we could send feedback to standard out about useful tidbits of knowledge about the function under test? (Or maybe run the suite in diagnose mode or something)

(diagnose [?a (midje/string) ?b (midje/string)]
  (fact
     (str ?a ?b) => (has-prefix ?a)))

This is the feature of Agitator that seemed like it was thinking out of the box, and potentially really neat.

@AlexBaranosky
Collaborator

Might make sense to use test.generative.generators as the library to use for Midje's (theoretical) generator library, since it seems better maintained:
https://github.com/clojure/test.generative/blob/master/src/main/clojure/clojure/test/generative/generators.clj

@AlexBaranosky
Collaborator

"Yes, the code is doing what you want.
By the way, what else is it doing?"

@AlexBaranosky
Collaborator

"It has been observed that TDD is an excellent methodology for developing “clean code that works” (Ron
Jeffries). Its many strengths include the ability to create code that does only what you want and to create
a thorough set of automated tests. However, by itself, TDD is incomplete as a coding and testing
methodology because of the fact that it creates code that does only what you want without taking into
account unintended side effects. ODT fills that void, using automated tools to test the behavior of code
and providing developers with actionable observations about possible unintended side effects."

@AlexBaranosky
Collaborator
;; This first swipe at generative-style testing in Midje was super easy:

;; first a use of it

(defn make-string []
  (rand-nth ["a" "b" "c" "d" "e" "f" "g" "i"]))

(formula [a (make-string) b (make-string)]
  (str a b) => (has-prefix a))

;; How its defined

(def ^:dynamic *num-generations* 100)

(defmacro formula [bindings & body]
  (macro-for [_ (range *num-generations*)]
    `(let ~bindings
       (midje.sweet/fact
          ~@body))))

;; NOTES:
;; * just borrow some generator functions from test.generative, and voila
;; a stripped down generative test framework.
;; * this version makes a fact for every generation which would quickly
;; make the fact count in the report meaningless.
;; * doc-strings can be added later.

;; QUESTIONS:
;; * how can we make the reporting of these really nice?
;; * is this it? There has to be more to this kind of thing than simply
;; generating 100 of each formula. What do you think?
@AlexBaranosky
Collaborator

I have looked into this further. What would it take to move this implementation from overly simplistic, to real-world usable?

1) store up the results of the 100 test runs, and then create only ONE report from them on failure
2) if there is a failure in one of the 100 runs, retry the fact execution with the inputs shrunken, repeat this process until it stops failing, then use the most shrunken failure case in the report output.

Both of these changes require adding interesting changes to Midje's flow of logic. Currently it assumes one fact per report, and that no fact would ever be re-ran.

@marick
Owner

Re: "interesting changes to Midje's flow of logic". Can generative tests be considered tabular tests?

@AlexBaranosky
Collaborator

Tabular tests will report a test run for each row of the table.

Each generative test would have 100 runs, so this will quickly report an insane # of test runs. Using tabular for generative testing wold also make it hard to do failure cases shrinking. (https://github.com/AlexBaranosky/Shrink) Because ideally it would be intelligent enough to stop when it sees a failure then go into a mode where it keeps calling shrink, until it no longer fails.

@marick
Owner

How about using metaconstant notation instead of a logic variable notation? Consider: a metaconstant describes a value about which nothing is known except what's stated in a provided clause. A variable to be replaced by a value generator is one about which nothing is known except (in some vague sense) its type. So how about this test:

(fact
  (.startsWith ^String ..a.. ^String ..b..) ..a..) => true))

The existence of a tagged metaconstant gives Midje license to generate N tests.

@AlexBaranosky
Collaborator

It's definitely an approach to consider. It is less boilerplate, which I like.

@AlexBaranosky
Collaborator

@marick, I like your idea for a metaconstant syntax. But I think the logic variable notation is simpler to implement. My plan is to (attempt to) get it working using the logic syntax, then we can discuss further how to improve the syntax.

@AlexBaranosky AlexBaranosky referenced this issue
Commit has since been removed from the repository and is no longer available.
@AlexBaranosky AlexBaranosky added a commit that referenced this issue
@AlexBaranosky AlexBaranosky [Issue #82] first version of formula: only reports 0-1 times per form…
…ula, regardless of # of generated fact runs
c5213ac
@AlexBaranosky AlexBaranosky added a commit that referenced this issue
@AlexBaranosky AlexBaranosky [Issue #82] first version of formula: only reports 0-1 times per form…
…ula, regardless of # of generated fact runs
923d921
@AlexBaranosky
Collaborator

Shrinking is harder than I erroneously imagined. To shrink we need to have a record of the inputs to the fact, but for normal facts (Which we piggy back over) there is no such record, as they are just baked into the fact code.

Ideally we need to be able to notice a failure in unprocessed.clj and then shrink the inputs and run another related fact.

I need to think on this.

@AlexBaranosky
Collaborator

@marick:
I'd like to merge the formulas branch into master (except for the crufty last commit I accidentally pushed) [1bb5b25]

It is alpha, and it doesn't shrink yet, but it is functional. I can mark the docstring ALPHA, but it seems worth getting it out in the open for people to try.

One question is d we want to suggest people use the generators built into test.generative (at least for now)? I personally have no problem with that, and think it might be the right final choice anyway (rather than recreating the wheel)...

@AlexBaranosky AlexBaranosky added a commit that referenced this issue
@AlexBaranosky AlexBaranosky Revert "[Issue #82] work in progress"
This reverts commit 1bb5b25.
8a30cfa
@AlexBaranosky AlexBaranosky added a commit that referenced this issue
@AlexBaranosky AlexBaranosky [Issue #82] rollback dynamic bindability of number of fact generation…
…s per formula until I can cover it in test properly.
915733b
@AlexBaranosky
Collaborator

I merged the formulas branch into master. The current backlog for formulas-related stories is here:
https://github.com/marick/Midje/blob/master/FORMULAS-BACKLOG-AND-FEATURE-IDEAS.md

@AlexBaranosky AlexBaranosky added a commit that referenced this issue
@AlexBaranosky AlexBaranosky [Issue #82] refactored such that last fact done in formula is just a …
…signal fact to signal the end of the formula, and holds no other purpose
988b8a5
@AlexBaranosky
Collaborator

does anyone have any strong feeling about if I force formulas to only have one check in them? The way I've implemented formulas piggy backs on the fact's ability to send extra params down to unprocessed. With extra checks this method of implementing them becomes impossible... well I guess not IMPOSSIBLE, but harder.

At least for initial formula versions I'm going to limit the scope of the feature to only working for one check at a time.

@AlexBaranosky AlexBaranosky added a commit that referenced this issue
@AlexBaranosky AlexBaranosky [Issue #82] added opt-map to formulas with :num-trials key to specify…
… number of trials on a per formula basis
d6c488a
@AlexBaranosky AlexBaranosky added a commit that referenced this issue
@AlexBaranosky AlexBaranosky [Issue-#82] riffing on the idea of making t-formulas more of an as-do…
…cumentation style... but kept in the usual location.
47265ca
@AlexBaranosky AlexBaranosky added a commit that referenced this issue
@AlexBaranosky AlexBaranosky [Issue #82] `with-num-trials` provides a nice syntax for multiple for…
…mulas to generate n trials each
d4d01bb
@josephwilk

Hows this coming along?
Curious to give it a try, is it documented in the wiki or anywhere else?

@marick
Owner

The basics work now. Alex can say more. There's no documentation, but the tests can help:

https://github.com/marick/Midje/blob/master/test/behaviors/t_formulas.clj

@marick
Owner

The original request exists: see formula. So I'll close this pending someone to take it on.

@marick marick closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.