QuickCheck-like fact support #82

Closed
AlexBaranosky opened this Issue Jan 3, 2012 · 21 comments

Comments

Projects
None yet
3 participants
@AlexBaranosky
Collaborator

AlexBaranosky commented Jan 3, 2012

Maybe it could look something like:

(generated [a (midje/string) b (midje/string)]
  (fact
    (str a b) => (has-prefix a)))

Inspired by examples from ScalaCheck's site:
http://code.google.com/p/scalacheck/

@AlexBaranosky

This comment has been minimized.

Show comment
Hide comment
@AlexBaranosky

AlexBaranosky Jan 3, 2012

Collaborator

Midje could come prepackaged with some generators (like midje/string), and also supply a Generator protocol that could be used to supply a generator for any type of data a user has.

Collaborator

AlexBaranosky commented Jan 3, 2012

Midje could come prepackaged with some generators (like midje/string), and also supply a Generator protocol that could be used to supply a generator for any type of data a user has.

@AlexBaranosky

This comment has been minimized.

Show comment
Hide comment
@AlexBaranosky

AlexBaranosky Jan 3, 2012

Collaborator

It looks like ClojureCheck has already done a ton of the legwork on this.
https://bitbucket.org/kotarak/clojurecheck/src/85c314b1bd2b/src/main/clojure/clojurecheck/core.clj

I'm going to see if maybe we could utilize ClojureCheck as a library for this feature.

Collaborator

AlexBaranosky commented Jan 3, 2012

It looks like ClojureCheck has already done a ton of the legwork on this.
https://bitbucket.org/kotarak/clojurecheck/src/85c314b1bd2b/src/main/clojure/clojurecheck/core.clj

I'm going to see if maybe we could utilize ClojureCheck as a library for this feature.

@AlexBaranosky

This comment has been minimized.

Show comment
Hide comment
@AlexBaranosky

AlexBaranosky Jan 5, 2012

Collaborator

This Agitator product for Java looks interesting. I wonder how it could be incorporated into Midje to improve the way TDD is done? http://www.agitar.com/pdf/Paper-Agitar-ODT-TDD.pdf [PDF]

Collaborator

AlexBaranosky commented Jan 5, 2012

This Agitator product for Java looks interesting. I wonder how it could be incorporated into Midje to improve the way TDD is done? http://www.agitar.com/pdf/Paper-Agitar-ODT-TDD.pdf [PDF]

@AlexBaranosky

This comment has been minimized.

Show comment
Hide comment
@AlexBaranosky

AlexBaranosky Jan 5, 2012

Collaborator

ClojureCheck:
http://kotka.de/blog/2010/06/ClojureCheck_is_back.html

I've contacted the maintainer of ClojureCheck about possibly pulling out the generator part of ClojureCheck into a separate namespace to use as a library in Midje.

Say you have a generative example like this, where any two strings concatenated together (str) always starts with the first string:

(generative [?a (midje/string) ?b (midje/string)]
  (fact
     (str ?a ?b) => (has-prefix ?a)))

Besides the generative testing approach, I'm also interested in being able to run diagnostics on an expression. What if by swapping the first word (the macro) we could send feedback to standard out about useful tidbits of knowledge about the function under test? (Or maybe run the suite in diagnose mode or something)

(diagnose [?a (midje/string) ?b (midje/string)]
  (fact
     (str ?a ?b) => (has-prefix ?a)))

This is the feature of Agitator that seemed like it was thinking out of the box, and potentially really neat.

Collaborator

AlexBaranosky commented Jan 5, 2012

ClojureCheck:
http://kotka.de/blog/2010/06/ClojureCheck_is_back.html

I've contacted the maintainer of ClojureCheck about possibly pulling out the generator part of ClojureCheck into a separate namespace to use as a library in Midje.

Say you have a generative example like this, where any two strings concatenated together (str) always starts with the first string:

(generative [?a (midje/string) ?b (midje/string)]
  (fact
     (str ?a ?b) => (has-prefix ?a)))

Besides the generative testing approach, I'm also interested in being able to run diagnostics on an expression. What if by swapping the first word (the macro) we could send feedback to standard out about useful tidbits of knowledge about the function under test? (Or maybe run the suite in diagnose mode or something)

(diagnose [?a (midje/string) ?b (midje/string)]
  (fact
     (str ?a ?b) => (has-prefix ?a)))

This is the feature of Agitator that seemed like it was thinking out of the box, and potentially really neat.

@AlexBaranosky

This comment has been minimized.

Show comment
Hide comment
@AlexBaranosky

AlexBaranosky Jan 5, 2012

Collaborator

Might make sense to use test.generative.generators as the library to use for Midje's (theoretical) generator library, since it seems better maintained:
https://github.com/clojure/test.generative/blob/master/src/main/clojure/clojure/test/generative/generators.clj

Collaborator

AlexBaranosky commented Jan 5, 2012

Might make sense to use test.generative.generators as the library to use for Midje's (theoretical) generator library, since it seems better maintained:
https://github.com/clojure/test.generative/blob/master/src/main/clojure/clojure/test/generative/generators.clj

@AlexBaranosky

This comment has been minimized.

Show comment
Hide comment
@AlexBaranosky

AlexBaranosky Jan 7, 2012

Collaborator

"Yes, the code is doing what you want.
By the way, what else is it doing?"

Collaborator

AlexBaranosky commented Jan 7, 2012

"Yes, the code is doing what you want.
By the way, what else is it doing?"

@AlexBaranosky

This comment has been minimized.

Show comment
Hide comment
@AlexBaranosky

AlexBaranosky Jan 7, 2012

Collaborator

"It has been observed that TDD is an excellent methodology for developing “clean code that works” (Ron
Jeffries). Its many strengths include the ability to create code that does only what you want and to create
a thorough set of automated tests. However, by itself, TDD is incomplete as a coding and testing
methodology because of the fact that it creates code that does only what you want without taking into
account unintended side effects. ODT fills that void, using automated tools to test the behavior of code
and providing developers with actionable observations about possible unintended side effects."

Collaborator

AlexBaranosky commented Jan 7, 2012

"It has been observed that TDD is an excellent methodology for developing “clean code that works” (Ron
Jeffries). Its many strengths include the ability to create code that does only what you want and to create
a thorough set of automated tests. However, by itself, TDD is incomplete as a coding and testing
methodology because of the fact that it creates code that does only what you want without taking into
account unintended side effects. ODT fills that void, using automated tools to test the behavior of code
and providing developers with actionable observations about possible unintended side effects."

@AlexBaranosky

This comment has been minimized.

Show comment
Hide comment
@AlexBaranosky

AlexBaranosky Jan 15, 2012

Collaborator
;; This first swipe at generative-style testing in Midje was super easy:

;; first a use of it

(defn make-string []
  (rand-nth ["a" "b" "c" "d" "e" "f" "g" "i"]))

(formula [a (make-string) b (make-string)]
  (str a b) => (has-prefix a))

;; How its defined

(def ^:dynamic *num-generations* 100)

(defmacro formula [bindings & body]
  (macro-for [_ (range *num-generations*)]
    `(let ~bindings
       (midje.sweet/fact
          ~@body))))

;; NOTES:
;; * just borrow some generator functions from test.generative, and voila
;; a stripped down generative test framework.
;; * this version makes a fact for every generation which would quickly
;; make the fact count in the report meaningless.
;; * doc-strings can be added later.

;; QUESTIONS:
;; * how can we make the reporting of these really nice?
;; * is this it? There has to be more to this kind of thing than simply
;; generating 100 of each formula. What do you think?
Collaborator

AlexBaranosky commented Jan 15, 2012

;; This first swipe at generative-style testing in Midje was super easy:

;; first a use of it

(defn make-string []
  (rand-nth ["a" "b" "c" "d" "e" "f" "g" "i"]))

(formula [a (make-string) b (make-string)]
  (str a b) => (has-prefix a))

;; How its defined

(def ^:dynamic *num-generations* 100)

(defmacro formula [bindings & body]
  (macro-for [_ (range *num-generations*)]
    `(let ~bindings
       (midje.sweet/fact
          ~@body))))

;; NOTES:
;; * just borrow some generator functions from test.generative, and voila
;; a stripped down generative test framework.
;; * this version makes a fact for every generation which would quickly
;; make the fact count in the report meaningless.
;; * doc-strings can be added later.

;; QUESTIONS:
;; * how can we make the reporting of these really nice?
;; * is this it? There has to be more to this kind of thing than simply
;; generating 100 of each formula. What do you think?
@AlexBaranosky

This comment has been minimized.

Show comment
Hide comment
@AlexBaranosky

AlexBaranosky Feb 11, 2012

Collaborator

I have looked into this further. What would it take to move this implementation from overly simplistic, to real-world usable?

  1. store up the results of the 100 test runs, and then create only ONE report from them on failure
  2. if there is a failure in one of the 100 runs, retry the fact execution with the inputs shrunken, repeat this process until it stops failing, then use the most shrunken failure case in the report output.

Both of these changes require adding interesting changes to Midje's flow of logic. Currently it assumes one fact per report, and that no fact would ever be re-ran.

Collaborator

AlexBaranosky commented Feb 11, 2012

I have looked into this further. What would it take to move this implementation from overly simplistic, to real-world usable?

  1. store up the results of the 100 test runs, and then create only ONE report from them on failure
  2. if there is a failure in one of the 100 runs, retry the fact execution with the inputs shrunken, repeat this process until it stops failing, then use the most shrunken failure case in the report output.

Both of these changes require adding interesting changes to Midje's flow of logic. Currently it assumes one fact per report, and that no fact would ever be re-ran.

@marick

This comment has been minimized.

Show comment
Hide comment
@marick

marick Feb 13, 2012

Owner

Re: "interesting changes to Midje's flow of logic". Can generative tests be considered tabular tests?

Owner

marick commented Feb 13, 2012

Re: "interesting changes to Midje's flow of logic". Can generative tests be considered tabular tests?

@AlexBaranosky

This comment has been minimized.

Show comment
Hide comment
@AlexBaranosky

AlexBaranosky Feb 13, 2012

Collaborator

Tabular tests will report a test run for each row of the table.

Each generative test would have 100 runs, so this will quickly report an insane # of test runs. Using tabular for generative testing wold also make it hard to do failure cases shrinking. (https://github.com/AlexBaranosky/Shrink) Because ideally it would be intelligent enough to stop when it sees a failure then go into a mode where it keeps calling shrink, until it no longer fails.

Collaborator

AlexBaranosky commented Feb 13, 2012

Tabular tests will report a test run for each row of the table.

Each generative test would have 100 runs, so this will quickly report an insane # of test runs. Using tabular for generative testing wold also make it hard to do failure cases shrinking. (https://github.com/AlexBaranosky/Shrink) Because ideally it would be intelligent enough to stop when it sees a failure then go into a mode where it keeps calling shrink, until it no longer fails.

@marick

This comment has been minimized.

Show comment
Hide comment
@marick

marick Feb 13, 2012

Owner

How about using metaconstant notation instead of a logic variable notation? Consider: a metaconstant describes a value about which nothing is known except what's stated in a provided clause. A variable to be replaced by a value generator is one about which nothing is known except (in some vague sense) its type. So how about this test:

(fact
  (.startsWith ^String ..a.. ^String ..b..) ..a..) => true))

The existence of a tagged metaconstant gives Midje license to generate N tests.

Owner

marick commented Feb 13, 2012

How about using metaconstant notation instead of a logic variable notation? Consider: a metaconstant describes a value about which nothing is known except what's stated in a provided clause. A variable to be replaced by a value generator is one about which nothing is known except (in some vague sense) its type. So how about this test:

(fact
  (.startsWith ^String ..a.. ^String ..b..) ..a..) => true))

The existence of a tagged metaconstant gives Midje license to generate N tests.

@AlexBaranosky

This comment has been minimized.

Show comment
Hide comment
@AlexBaranosky

AlexBaranosky Feb 13, 2012

Collaborator

It's definitely an approach to consider. It is less boilerplate, which I like.

Collaborator

AlexBaranosky commented Feb 13, 2012

It's definitely an approach to consider. It is less boilerplate, which I like.

@AlexBaranosky

This comment has been minimized.

Show comment
Hide comment
@AlexBaranosky

AlexBaranosky Feb 18, 2012

Collaborator

@marick, I like your idea for a metaconstant syntax. But I think the logic variable notation is simpler to implement. My plan is to (attempt to) get it working using the logic syntax, then we can discuss further how to improve the syntax.

Collaborator

AlexBaranosky commented Feb 18, 2012

@marick, I like your idea for a metaconstant syntax. But I think the logic variable notation is simpler to implement. My plan is to (attempt to) get it working using the logic syntax, then we can discuss further how to improve the syntax.

AlexBaranosky added a commit that referenced this issue Feb 19, 2012

[Issue #82] first version of formula: only reports 0-1 times per form…
…ula, regardless of # of generated fact runs

AlexBaranosky added a commit that referenced this issue Feb 19, 2012

AlexBaranosky added a commit that referenced this issue Feb 19, 2012

[Issue #82] first version of formula: only reports 0-1 times per form…
…ula, regardless of # of generated fact runs

AlexBaranosky added a commit that referenced this issue Feb 19, 2012

@AlexBaranosky

This comment has been minimized.

Show comment
Hide comment
@AlexBaranosky

AlexBaranosky Feb 19, 2012

Collaborator

Shrinking is harder than I erroneously imagined. To shrink we need to have a record of the inputs to the fact, but for normal facts (Which we piggy back over) there is no such record, as they are just baked into the fact code.

Ideally we need to be able to notice a failure in unprocessed.clj and then shrink the inputs and run another related fact.

I need to think on this.

Collaborator

AlexBaranosky commented Feb 19, 2012

Shrinking is harder than I erroneously imagined. To shrink we need to have a record of the inputs to the fact, but for normal facts (Which we piggy back over) there is no such record, as they are just baked into the fact code.

Ideally we need to be able to notice a failure in unprocessed.clj and then shrink the inputs and run another related fact.

I need to think on this.

AlexBaranosky added a commit that referenced this issue Feb 22, 2012

@AlexBaranosky

This comment has been minimized.

Show comment
Hide comment
@AlexBaranosky

AlexBaranosky Feb 24, 2012

Collaborator

@marick:
I'd like to merge the formulas branch into master (except for the crufty last commit I accidentally pushed) [https://github.com/marick/Midje/commit/1bb5b25fbe30498000691ed1acb02a6ef47f8fe6]

It is alpha, and it doesn't shrink yet, but it is functional. I can mark the docstring ALPHA, but it seems worth getting it out in the open for people to try.

One question is d we want to suggest people use the generators built into test.generative (at least for now)? I personally have no problem with that, and think it might be the right final choice anyway (rather than recreating the wheel)...

Collaborator

AlexBaranosky commented Feb 24, 2012

@marick:
I'd like to merge the formulas branch into master (except for the crufty last commit I accidentally pushed) [https://github.com/marick/Midje/commit/1bb5b25fbe30498000691ed1acb02a6ef47f8fe6]

It is alpha, and it doesn't shrink yet, but it is functional. I can mark the docstring ALPHA, but it seems worth getting it out in the open for people to try.

One question is d we want to suggest people use the generators built into test.generative (at least for now)? I personally have no problem with that, and think it might be the right final choice anyway (rather than recreating the wheel)...

AlexBaranosky added a commit that referenced this issue Feb 25, 2012

AlexBaranosky added a commit that referenced this issue Feb 25, 2012

AlexBaranosky added a commit that referenced this issue Feb 25, 2012

[Issue #82] rollback dynamic bindability of number of fact generation…
…s per formula until I can cover it in test properly.
@AlexBaranosky

This comment has been minimized.

Show comment
Hide comment
@AlexBaranosky

AlexBaranosky Feb 25, 2012

Collaborator

I merged the formulas branch into master. The current backlog for formulas-related stories is here:
https://github.com/marick/Midje/blob/master/FORMULAS-BACKLOG-AND-FEATURE-IDEAS.md

Collaborator

AlexBaranosky commented Feb 25, 2012

I merged the formulas branch into master. The current backlog for formulas-related stories is here:
https://github.com/marick/Midje/blob/master/FORMULAS-BACKLOG-AND-FEATURE-IDEAS.md

@AlexBaranosky

This comment has been minimized.

Show comment
Hide comment
@AlexBaranosky

AlexBaranosky Mar 9, 2012

Collaborator

does anyone have any strong feeling about if I force formulas to only have one check in them? The way I've implemented formulas piggy backs on the fact's ability to send extra params down to unprocessed. With extra checks this method of implementing them becomes impossible... well I guess not IMPOSSIBLE, but harder.

At least for initial formula versions I'm going to limit the scope of the feature to only working for one check at a time.

Collaborator

AlexBaranosky commented Mar 9, 2012

does anyone have any strong feeling about if I force formulas to only have one check in them? The way I've implemented formulas piggy backs on the fact's ability to send extra params down to unprocessed. With extra checks this method of implementing them becomes impossible... well I guess not IMPOSSIBLE, but harder.

At least for initial formula versions I'm going to limit the scope of the feature to only working for one check at a time.

AlexBaranosky added a commit that referenced this issue Mar 10, 2012

AlexBaranosky added a commit that referenced this issue Mar 11, 2012

AlexBaranosky added a commit that referenced this issue Mar 14, 2012

AlexBaranosky added a commit that referenced this issue Mar 15, 2012

AlexBaranosky added a commit that referenced this issue Mar 15, 2012

AlexBaranosky added a commit that referenced this issue Mar 15, 2012

AlexBaranosky added a commit that referenced this issue Mar 17, 2012

AlexBaranosky added a commit that referenced this issue Mar 19, 2012

[Issue-#82] riffing on the idea of making t-formulas more of an as-do…
…cumentation style... but kept in the usual location.

AlexBaranosky added a commit that referenced this issue Mar 21, 2012

AlexBaranosky added a commit that referenced this issue Mar 26, 2012

AlexBaranosky added a commit that referenced this issue Mar 26, 2012

@josephwilk

This comment has been minimized.

Show comment
Hide comment
@josephwilk

josephwilk Mar 20, 2013

Contributor

Hows this coming along?
Curious to give it a try, is it documented in the wiki or anywhere else?

Contributor

josephwilk commented Mar 20, 2013

Hows this coming along?
Curious to give it a try, is it documented in the wiki or anywhere else?

@marick

This comment has been minimized.

Show comment
Hide comment
@marick

marick Mar 20, 2013

Owner

The basics work now. Alex can say more. There's no documentation, but the tests can help:

https://github.com/marick/Midje/blob/master/test/behaviors/t_formulas.clj

Owner

marick commented Mar 20, 2013

The basics work now. Alex can say more. There's no documentation, but the tests can help:

https://github.com/marick/Midje/blob/master/test/behaviors/t_formulas.clj

@marick

This comment has been minimized.

Show comment
Hide comment
@marick

marick May 7, 2013

Owner

The original request exists: see formula. So I'll close this pending someone to take it on.

Owner

marick commented May 7, 2013

The original request exists: see formula. So I'll close this pending someone to take it on.

@marick marick closed this May 7, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment