Tutorial Playground

[CodaFi]

No description provided.

[CodaFi]

@rnapier I know you stressed pre-reqs in the last pull request. Is there anything you would add/remove from this list? Any better links than wikipedia would be helpful too.

[buildreactive]

I'm trying to get SwiftCheck working with DateTime (https://github.com/zbeckman/DateTimeKit). I get that I need to create an extension DateTime : Arbitrary but I'm a bit lost as to how to do this right. If I want SwiftCheck to support DateTime just as thoroughly as an atomic type, how would I go about that?

Seeing a tutorial on how to add support for NSDate would probably be a great example (and it would be a nice addition to SwiftCheck).

[CodaFi]

NSDate cannot be used directly (dumbass Self requirement restrictions on protocol instances). You have to wrap it.

[CodaFi]

There, I included it in the tutorial. It's a pretty great way to introduce modifiers.

[buildreactive]

Excellent. That's another major real world problem we run into all the time. (Also the fact that NSDate kinda sucks which is why we are using DateTime instead...)

[jtbandes]

This should work too:

func glue5(l: String)(m: String)(m2: String)(m3: String)(r: String) {
    return l + m + m2 + m3 + r
}

[CodaFi]

Absolutely. I'll use that.

[CodaFi]

This is a dumb and lazy and inaccurate simplification. I think I'll use a diagram.

[CodaFi]

I'd like a list of external links here too

[rnapier]

I like where this is going, but I'm struggling to use it in my own code. You have a lot of nice examples like emailGen, but don't show how you would use that with forAll (or more generally, how to use it with property). With experimentation, I think I've discovered that these can't be passed to forAll and have to instead be generated inside the closure. It's not clear whether this is correct or not.

The specific case I was trying to implement was "array of UInt8 of length 8."

[CodaFi]

I can see 2 ways of going about your specific example:

1. Pass the generator to forAll

let arrayGen = UInt8.arbitrary.proliferateSized(8)

property("...") <- forAll(arrayGen) { x in
    /// ... 
}

2. Use implication to discard bad arrays:

property("...") <- forAll { (x : [UInt8]) in
    return (x.count == 8) ==> {
        /// ...
    }
}

The former is more conducive to explanation than the latter.

[CodaFi]

This evil is only temporary.

[rnapier]

Does the filtering (implication) process at up my 100 tests? I mean, if I pick 100 arrays and only 2 of them are the right length, do I run just those 2, or do I keep implicating until get I get 100 things to try?

I didn't realize the forAll(generator)... form. That probably answers my next question about how to create non-empty strings.

[CodaFi]

@zbeckman Does this further your understanding of why your use-cases are a little strange w.r.t. this library?

[buildreactive]

Hm. I see your points.

However – let's say you wanted to write a test that ensures a given list of data is populated with at least 100 (or more?) "valid" sets of data. We could say a valid piece of data is a "positive Int" or a "well formed email" or a "password/salt pair." Doesn't really matter – one thing I run into in the real world is a two phase test plan:

1. Test that your preconditions are valid, e.g., that you have actually create 100 viable and usable properties within the definition of your spec.
2. Run those viable properties through your program to see if it fails.

The above example was a bit contrived but that's what it was getting at... Validate the properties, and THEN verify the program.

[CodaFi]

Then you would make part 1 a Generator and part 2 the actual property.

A spec is not the place to be generating data, a spec is a place where you validate your assumptions against your model. The testing block works best as a function from filtered inputs to truthy and falsy outcomes.

[buildreactive]

One thing to understand about how tools like SwiftCheck get applied... and I'm saying this from about 30 years in SQA, from the Defense Department to little startups... it gets applied in a very "practical" way. Teams that are striving (and under the gun) to deliver working software end up applying it to solve their immediate need. That includes things like:

1. Generate a set of test data for a given program component.
2. Validate that the test data is, actually, valid test data. Often the generation of the data set is just as complicated as testing the code or even writing the code.
3. FIGURING OUT how to test a program... because they are not mathematicians or professional quality assurance people. Or perhaps they are profession QA people... but that tends to mean they are NOT programmers.
4. Applying reproductions to ensure regressions in a UI or UI component or program component do not occur.
5. Actually GENERATING real data for the program – either in a quality assurance setting (e.g. load a QA server with a lot of test data that looks "real") or even in production (use it to populate required values in a running system – more rare, but it happens).

I realize some of these fall outside the scope of what SwiftCheck/QuickCheck was really designed to do, but when you give a programmer a tool and a problem... ;-)

[CodaFi]

I know it's a very strange thing to be told to throw out that kind of experience for this newfangled abstract nonsense, but there are very serious implications to this whole thing. When you get down to properties, I mean really get down to properties, you only have 2 steps:

1. What are my assumptions?
2. When do they hold?

Step 1 lets you break down your problem into properties, your assumptions become inputs to functions.
Step 2 lets you actually write some "laws". SwiftCheck may not be able to tell you with 100% certainty that they hold (something something entscheidungsproblem), but like a Scientific Law we take repeated observation of success to mean success.

[buildreactive]

(And yes, making a custom shrinker makes sense – however, I can pretty much guarantee that 80% of the people that use this library will be utterly confused when they write an email generator that explicitly does NOT include spaces... and then they see failure cases that do – that is the only reason I'm suggesting you go down this path – to avoid further confusion).

[CodaFi]

Utterly, no. I catch your meaning tho. I've carved out room for another example under the modifier types one. I will solve your problem there.

[CodaFi]

There, that should suffice.

[buildreactive]

Looking back at our conversation, I have a question: How would you write a generator that would create a list of 100 things that you know meet a certain program specification? (Perhaps email addresses that you know pass the email =~ validEmailPattern or whatever)? I'm curious to see your implementation of a generator that verifies such a rule and produces a list. I think this would be another good real world case. The key here is making it clear to the read:

1. How to generate a list of things.
2. How to size the list to whatever desired size.
3. How to verify that each thing meets a program specification.
   (I think you've touched on all of these but I don't think you have put it all together in one place – which makes me wonder if I'm doing it the right way).

[CodaFi]

It's more efficient to be correct by construction than to be correct by filtering out the totality of strings SwiftCheck can generate. I wouldn't make a generator of strings with a suchThat clause using ~= as a filter because Gen will spin as long as it needs trying to find a passing example. Convert the body of the relation ~= into the blocks necessary to build the generator, build it, and feed it to a forAll. Then, you can use the relation ~= as your spec!

let g = Gen.sized >>- { n in
    return myGen.proliferateSized(n)
} >>- Gen.fromElementsOf

[buildreactive]

How would I go about having one test run a larger set of property tests (say 1000 rather than 100)?

[CodaFi]

That is accomplished in QuickCheck by modifying the stdArgs options. I haven't explored a satisfactory way of doing the same in Swift (the last version's syntax couldn't allow for multiple arguments in a subscript). If you'd like to try, all the machinery is there.

[buildreactive]

What about adding a few props to Testable (maxRun, maxFail, things like that)?

[buildreactive]

(I can give it a try but I'm no expert on this syntax).

[CodaFi]

No, don't do that!

There's an internal struct called Argument that is passed to quickCheckWithResult. You need to make that public and probably find a way to pass it through the property function.

[CodaFi]

With that, I'm done. I will absolutely accept modifications to this in the form of pull requests, but any future questions about any part of the framework need to be separate issues from here on out.

Thank you all for your help with this 💕

⛵

[buildreactive]

Awesome! This is a big gain the project. Question: Can you generate an HTML page from the tutorial, stick it into the project, and link to it from the readme.md so it's on the wiki? That would be great if it could be done.

By the way... my comment on adding props to Testable where motivated by ScalaCheck, where I can do this (and which I really like):

property("Unicode email addresses validate correctly") <- forAll(Generators.emailAddress) { (e : String) in
        f.text = e
        return f.isValid
}.minSize(200).maxDiscard(50)

Anyhow, food for future thought...

[CodaFi]

Why HTML? The playground comes bundled with the project now, so you have access to it even if you build with carthage. The tutorial is designed to be interactive.

As for the other stuff, like I said, it's all there, it just needs a public interface.

[buildreactive]

So that it's on the wiki. Our team doesn't get the Carthage checkout since frameworks are handled by me and then committed in git. Nothing like good docs on the web.

[CodaFi]

Do you know of any tools to extract that kind of information from a playground?

[buildreactive]

No afraid not... will keep my eyes open for something though.