Handle printing of clojure.spec.test.alpha/check results

[kennyjwilli]

The output from clojure.spec.test.alpha/check is unwieldy. Seems like Expound could provide a way to cleanly print these results.

[kennyjwilli]

For example, take this simple function:

(defn fabulous
  [x y]
  nil)

(s/fdef fabulous
        :args (s/cat :x number? :y number?)
        :ret string?)

Running (clojure.spec.test.alpha/check fabulous) results in this mountain of text: https://pastebin.com/bxcaiYzn (put in pastebin because of its length).

We can do a little better by passing some data from the returned list to expound:

(expound/printer
  (ex-data (get-in (first (st/check `fabulous)) [:clojure.spec.test.check/ret :result-data :clojure.test.check.properties/error])))
-- Function spec failed -----------

  nil

returned an invalid value

  nil

should satisfy

  string?



-------------------------
Detected 1 error
=> nil

It looks like expound.alpha/spec-name is not correctly resolving the spec name, thus the first nil. The map passed to expound.alpha/printer has :clojure.spec.alpha/failure set as :check-failed which causes spec-name to return nil.

[bhb]

@kennyjwilli Thanks for reporting this! I haven't used clojure.spec.test.alpha/check myself, so I hadn't considered this use case, but it's a good idea. I'll look into it.

[aviflax]

OMG yes please!

[bhb]

@kennyjwilli @aviflax I've added a few functions: explain-results, explain-result, explain-results-str and explain-result-str. Here's an example:

(require '[clojure.spec.alpha :as s])
(require '[clojure.spec.test.alpha :as st])
(require '[expound.alpha :as expound])
(defn fabulous
  [x y]
  nil)

(s/fdef fabulous
        :args (s/cat :x number? :y number?)
        :ret string?)

(set! s/*explain-out* expound/printer)

(expound/explain-results (st/check `fabulous))
;;== Checked expound.alpha/fabulous ===========
;;
;;-- Function spec failed -----------
;;
;;  nil
;;
;;returned an invalid value
;;
;;  nil
;;
;;should satisfy
;;
;;  string?
;;
;;
;;
;;-------------------------
;;Detected 1 error

As you can see, you can pass the results from check to this function to have it print the results with expound. Does this work for your use case? I think the error messages themselves need some work (the above message isn't very good), but I'm wondering about the general approach. What do you think?

If you're so inclined, you can try it by installing "0.5.1-SNAPSHOT"

[kennyjwilli]

The approach seems good - that's is exactly how I'd use it.

I know you said the error messages need some work, but I thought I'd still let you know of a couple cases that aren't reported correctly.

This example fails with an ClassCastException but Expound thinks it was successful.

(defn my-add2
  [x y]
  (+ (str x) y))

(s/fdef my-add2
        :args (s/cat :x number? :y number?)
        :ret string?)

(expound/explain-results (st/check `my-add2))
== Checked user/my-add2 =

Success!
=> nil

Sometimes test.check may fail to generate. In this case we get an exception that says "Unable to construct gen at: [:f] for: fn?". This exception is thrown from text.check and is stored in the path [:clojure.spec.test.check/ret :result].

(defn takes-fn
  [f]
  (f 1))

(s/fdef takes-fn
        :args (s/cat :f fn?)
        :ret any?)

(st/check `takes-fn)
=>
({:spec #object[clojure.spec.alpha$fspec_impl$reify__2451
                0x2c417d47
                "clojure.spec.alpha$fspec_impl$reify__2451@2c417d47"],
  :clojure.spec.test.check/ret {:result #error{:cause "Unable to construct gen at: [:f] for: fn?",
                                               :data #:clojure.spec.alpha{:path [:f],
                                                                          :form clojure.core/fn?,
                                                                          :failure :no-gen}}},
  :sym user/takes-fn,
  :failure #error{:cause "Unable to construct gen at: [:f] for: fn?",
                  :data #:clojure.spec.alpha{:path [:f], :form clojure.core/fn?, :failure :no-gen}}})

(expound/explain-results (st/check `takes-fn))
== Checked user/takes-fn 

=> nil

[bhb]

@kennyjwilli This is extremely helpful feedback. Thanks!

[bhb]

@kennyjwilli @aviflax Thanks for the detailed help on how failure modes of check. I've released another version of 0.5.1-SNAPSHOT that covers the cases you've mentioned above. Can you give it another try? Thanks!

[kennyjwilli]

Looks good. Here's another example that isn't working correctly.

(defn my-broken-add
  [x y]
  (+ (str x) y))

(s/fdef my-broken-add
        :args (s/cat :x number? :y number?)
        :ret string?)

(expound/explain-results (st/check `my-broken-add))
== Checked user/my-broken-add 

Success!
=> nil

Running (st/check my-broken-add) does not pass. This applies for any function that throws an exception while executing:

(defn ithrow
  [x]
  (throw (ex-info "ithrow" {})))

(s/fdef ithrow
        :args (s/cat :x number?)
        :ret string?)

(expound/explain-results (st/check `ithrow))
== Checked user/ithrow ==

Success!
=> nil

[bhb]

@kennyjwilli That's weird, I see different behavior locally. When I try, I get

expound(expound/explain-results (st/check `ithrow))
== Checked user/ithrow ======================

  (user/ithrow -1)

 threw error

#error {
 :cause "ithrow"
 :data {}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "ithrow"
   :data {}
   :at [clojure.core$ex_info invokeStatic "core.clj" 4739]}]
 :trace

Is it possible that your local version of 0.5.1-SNAPSHOT is out of date or cached? I just updated it a few days ago.

[bhb]

Perhaps try rm -rf ~/.m2/repository/expound/expound/0.5.1-SNAPSHOT and try again to force a re-download?

[kennyjwilli]

Running with [expound "0.5.1-20180423.185123-3"] yields the same results I stated earlier. I also tried deleting the expound folder and am still getting the same results.

[bhb]

@kennyjwilli Hmm, this is very strange. I fear that I've done something wrong here, or there is some context that I'm missing. I appreciate your help on this!

What happens if you do the following?

clj -Srepro -Sdeps '{:deps {expound {:mvn/version "0.5.1-20180423.185123-3"} org.clojure/test.check {:mvn/version "0.9.0"} org.clojure/clojure {:mvn/version "1.9.0"}}}'

And then, in the REPL, you do:

(require '[expound.alpha :as expound] 
              '[clojure.spec.test.alpha :as st] 
              '[clojure.spec.alpha :as s]
              '[clojure.test.check])
(set! s/*explain-out* (expound/custom-printer {:theme :figwheel-theme}))
(defn ithrow
  [x]
  (throw (ex-info "ithrow" {})))

(s/fdef ithrow
        :args (s/cat :x number?)
        :ret string?)

(expound/explain-results (st/check `ithrow))

Do you see the same behavior as before?

Thanks for your continued help!

[bhb]

@kennyjwilli What's weird is that I thought I saw the behavior you reported at one point, but now I cannot reproduce. I wonder if there is something weird with check behaving in non-deterministic ways, or expound missing some specific error case. I'll continue to look.

[kennyjwilli]

clj is great for debugging these sorts of problems! Running with the clj deps and the above code, I do get the expected result. Perhaps Boot is not using the correct Expound jar for some reason. Gotta love SNAPSHOTs... 🙃

[bhb]

@kennyjwilli Excellent, glad to hear it worked. Thanks for all your help!

[bhb]

@kennyjwilli @aviflax This feature is in expound 0.6.0. Thanks for all your help!

[kennyjwilli]

@bhb Thanks for your responsiveness and great work. Looking forward to using this feature!