Maybe it could look something like:
(generated [a (midje/string) b (midje/string)]
(str a b) => (has-prefix a)))
Inspired by examples from ScalaCheck's site:
Midje could come prepackaged with some generators (like midje/string), and also supply a Generator protocol that could be used to supply a generator for any type of data a user has.
It looks like ClojureCheck has already done a ton of the legwork on this.
I'm going to see if maybe we could utilize ClojureCheck as a library for this feature.
This Agitator product for Java looks interesting. I wonder how it could be incorporated into Midje to improve the way TDD is done? http://www.agitar.com/pdf/Paper-Agitar-ODT-TDD.pdf [PDF]
I've contacted the maintainer of ClojureCheck about possibly pulling out the generator part of ClojureCheck into a separate namespace to use as a library in Midje.
Say you have a generative example like this, where any two strings concatenated together (str) always starts with the first string:
(generative [?a (midje/string) ?b (midje/string)]
(str ?a ?b) => (has-prefix ?a)))
Besides the generative testing approach, I'm also interested in being able to run diagnostics on an expression. What if by swapping the first word (the macro) we could send feedback to standard out about useful tidbits of knowledge about the function under test? (Or maybe run the suite in diagnose mode or something)
(diagnose [?a (midje/string) ?b (midje/string)]
(str ?a ?b) => (has-prefix ?a)))
This is the feature of Agitator that seemed like it was thinking out of the box, and potentially really neat.
Might make sense to use test.generative.generators as the library to use for Midje's (theoretical) generator library, since it seems better maintained:
"Yes, the code is doing what you want.
By the way, what else is it doing?"
"It has been observed that TDD is an excellent methodology for developing “clean code that works” (Ron
Jeffries). Its many strengths include the ability to create code that does only what you want and to create
a thorough set of automated tests. However, by itself, TDD is incomplete as a coding and testing
methodology because of the fact that it creates code that does only what you want without taking into
account unintended side effects. ODT fills that void, using automated tools to test the behavior of code
and providing developers with actionable observations about possible unintended side effects."
;; This first swipe at generative-style testing in Midje was super easy:
;; first a use of it
(defn make-string 
(rand-nth ["a" "b" "c" "d" "e" "f" "g" "i"]))
(formula [a (make-string) b (make-string)]
(str a b) => (has-prefix a))
;; How its defined
(def ^:dynamic *num-generations* 100)
(defmacro formula [bindings & body]
(macro-for [_ (range *num-generations*)]
;; * just borrow some generator functions from test.generative, and voila
;; a stripped down generative test framework.
;; * this version makes a fact for every generation which would quickly
;; make the fact count in the report meaningless.
;; * doc-strings can be added later.
;; * how can we make the reporting of these really nice?
;; * is this it? There has to be more to this kind of thing than simply
;; generating 100 of each formula. What do you think?
I have looked into this further. What would it take to move this implementation from overly simplistic, to real-world usable?
1) store up the results of the 100 test runs, and then create only ONE report from them on failure
2) if there is a failure in one of the 100 runs, retry the fact execution with the inputs shrunken, repeat this process until it stops failing, then use the most shrunken failure case in the report output.
Both of these changes require adding interesting changes to Midje's flow of logic. Currently it assumes one fact per report, and that no fact would ever be re-ran.
Re: "interesting changes to Midje's flow of logic". Can generative tests be considered tabular tests?
Tabular tests will report a test run for each row of the table.
Each generative test would have 100 runs, so this will quickly report an insane # of test runs. Using tabular for generative testing wold also make it hard to do failure cases shrinking. (https://github.com/AlexBaranosky/Shrink) Because ideally it would be intelligent enough to stop when it sees a failure then go into a mode where it keeps calling shrink, until it no longer fails.
How about using metaconstant notation instead of a logic variable notation? Consider: a metaconstant describes a value about which nothing is known except what's stated in a provided clause. A variable to be replaced by a value generator is one about which nothing is known except (in some vague sense) its type. So how about this test:
(.startsWith ^String ..a.. ^String ..b..) ..a..) => true))
The existence of a tagged metaconstant gives Midje license to generate N tests.
It's definitely an approach to consider. It is less boilerplate, which I like.
@marick, I like your idea for a metaconstant syntax. But I think the logic variable notation is simpler to implement. My plan is to (attempt to) get it working using the logic syntax, then we can discuss further how to improve the syntax.
[Issue #82] first version of formula: only reports 0-1 times per form…
…ula, regardless of # of generated fact runs
[Issue #82] removed unused code
[Issue #82] adding backlog file for the formula epic
[Issue #82] now reports the first failure's normal report
[Issue #82] formula can now take an optional docstring
[Issue #82] updating formulas backlog
[Issue #82] no longer counts failures as 2 fact runs.
Shrinking is harder than I erroneously imagined. To shrink we need to have a record of the inputs to the fact, but for normal facts (Which we piggy back over) there is no such record, as they are just baked into the fact code.
Ideally we need to be able to notice a failure in unprocessed.clj and then shrink the inputs and run another related fact.
I need to think on this.
[Issue #82] work in progress
I'd like to merge the formulas branch into master (except for the crufty last commit I accidentally pushed) [1bb5b25]
It is alpha, and it doesn't shrink yet, but it is functional. I can mark the docstring ALPHA, but it seems worth getting it out in the open for people to try.
One question is d we want to suggest people use the generators built into test.generative (at least for now)? I personally have no problem with that, and think it might be the right final choice anyway (rather than recreating the wheel)...
Revert "[Issue #82] work in progress"
This reverts commit 1bb5b25.
[Issue #82] fixed formulas so that they evaluate their generators, an…
…d bodies 100 times each
[Issue #82] adjustments after checking vs all versions of clojure
[Issue #82] more ideas added to the formula backlog
[Issue #82] rollback dynamic bindability of number of fact generation…
…s per formula until I can cover it in test properly.
I merged the formulas branch into master. The current backlog for formulas-related stories is here:
[Issue #82] adding syntax validations for formula macro
[Issue #82] syntax error if less than 2 elements in the binding
[Issue #82] generations per formula is now dynamically rebindable
[Issue #82] improving documentation and the backlog
[Issue #82] validates that num generations var is bound to number >= 2
[Issue #82] removed test.generative -- its not 1.2.1 compatible
[Issue #82] refactored such that last fact done in formula is just a …
…signal fact to signal the end of the formula, and holds no other purpose
[Issue #82] updated backlog with new insights
[Issue #82] once a formula detects a failure, won't run any more gene…
[Issue #82] updated backlog; organized t-formula
[Issue #82] formulas can eb set to 1 or more generations now, not 2 o…
[Issue #82] try block is a little more stable
[Issue #82] updating backlog
does anyone have any strong feeling about if I force formulas to only have one check in them? The way I've implemented formulas piggy backs on the fact's ability to send extra params down to unprocessed. With extra checks this method of implementing them becomes impossible... well I guess not IMPOSSIBLE, but harder.
At least for initial formula versions I'm going to limit the scope of the feature to only working for one check at a time.
[Issue #82] updated formula backlog
[Issue #82] added fact covering simple use of `provided` in a formulas
[Issue #82] bunch of partial work
[Issue #82] dynamic binding of *num-generations-per-formula* works again
[Issue #82] fixing shrinking to happen at runtime not macroexpand time
[Issue #82] pulled duplicated code into private fn
[Issue #82] formulas only allow one check per formula
[Issue #82] formulas can have multiple bindings that each can be shru…
[Issue #82] updates to formula backlog
[Issue #82] arrows in against-background in a formula are ignored whe…
…n looking for check arrows
[Issue #82] a lot more syntax validations for formula
[Issue #82] added notion of a future-formula/pending-formula/incipien…
[Issue #82] added opt-map to formulas with :num-trials key to specify…
… number of trials on a per formula basis
[Issue #82] renamed *num-generations-per-formula* -> *num-trials*
[Issue #82] validate that only valid opt-map keys is :num-trials
[Issue #82] validate that :num-trials can only be set to 1+
[Issue #82] added one more validation case taking the opts-map into a…
[Issue #82] added opts-map? to :arglists and docstring
[Issue #82] 'or' makes for cleaner code
[Issue #82] better test case
[Issue #82] use valid-let over when-valid
[Issue #82] clarifying renames
[Issue #82] clarifying renames - part #2
[Issue #82] introduce fn pop-opts-map
[Issue-#82] broadened test to cover two bindingvars that both shrink.
[Issue-#82] riffing on the idea of making t-formulas more of an as-do…
…cumentation style... but kept in the usual location.
[Issue #82] `with-num-trials` provides a nice syntax for multiple for…
…mulas to generate n trials each
[Issue #82] min length of all shrunken cases seqs determines how many…
… shrunken cases tried
[Issue #82] missed a comment
Hows this coming along?
Curious to give it a try, is it documented in the wiki or anywhere else?
The basics work now. Alex can say more. There's no documentation, but the tests can help:
The original request exists: see formula. So I'll close this pending someone to take it on.