Property Test Discussion #1121
Comments
sksamuel
added
discussion
enhancement ✨
|
Deterministic Re-runs I run property test, and my test fails. I know the seed of the failure, so I want to execute values from that seed as I know they'll fail. I then make the test pass. After the test passes, I'll always use the same seed, and that's not exactly very good, as I won't have enough randomness from now on (i'll always execute on the exact same 1000 values). For this reason, I think that the seed should generate, for instance, 1000 cases, but we should still have randomness to the test, so maybe an additional 1000 random cases should be generated as well. And at some point I could have like As property testing should be used mostly in Unit tests, adding more test cases shouldn't be bad, and this could be optional any way. I just am not sure on the syntax |
|
Exhaustive
How would we know if it's supported? Could we split between I don't really like doing magic based on a parameter. For example, Without looking at the code, I must know if |
|
The generator itself would know if it's supported and return the
appropriate sequence.
…On Fri, 13 Dec 2019, 19:47 Leonardo Colman Lopes, ***@***.***> wrote:
*Exhaustive*
Auto - Exhaustive if possible and supported, otherwise random.
How would we know if it's supported? Could we split between interface
ExhaustiveGen and interface RandomGen? Or maybe something inbetween?
I don't really like doing magic based on a parameter. For example, Without
looking at the code, I must know if Gen.int() is exhaustible or not. But
how could we do that?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#1121>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFVSGRRIIWXDS7HVXJQZ2DQYQ3KNANCNFSM4J2MINTA>
.
|
|
@Kerooker regarding the deterministic re runs, I think what should be done is by default leave the seed null (changing everytime), and if a test failed, print to System.err the seed that was used for the failing test. You can then apply that same seed to reproduce the failing case, fix your code so the tests pass, and then remove the seed again to resume using random seeds. |
|
@xgouchet That's better than having more than one seed, probably |
|
Distribution
I think so, yes. |
In that case I'm not sure of the value of the "built in" ones. Could just be an interface and each gen provides zero or more implementations. |
Yes I think so. I'm not a fan of multiple seeds. |
How would I know it's supported? For example, I want to test all integers. How can I know if it supports exhaustiveness until I try to execute it? |
|
You know the iteration count and you know your own range.
…On Sat, 14 Dec 2019, 18:37 Leonardo Colman Lopes, ***@***.***> wrote:
The generator itself would know if it's supported and return the
appropriate sequence.
How would I know it's supported? For example, I want to test all integers.
How can I know if it supports exhaustiveness until I try to execute it?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#1121>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFVSGTERMGRQL6LLW3RM33QYV34HANCNFSM4J2MINTA>
.
|
|
Take a look at the Gen.int implementation I included
…On Sat, 14 Dec 2019, 19:06 Stephen Samuel (Sam), ***@***.***> wrote:
You know the iteration count and you know your own range.
On Sat, 14 Dec 2019, 18:37 Leonardo Colman Lopes, <
***@***.***> wrote:
> The generator itself would know if it's supported and return the
> appropriate sequence.
>
> How would I know it's supported? For example, I want to test all
> integers. How can I know if it supports exhaustiveness until I try to
> execute it?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <#1121>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAFVSGTERMGRQL6LLW3RM33QYV34HANCNFSM4J2MINTA>
> .
>
|
I would like a way to have this informed without having to look at the code. When dealing with a big code base, having to look if the Perhaps splitting exhaustivity and randomness into different interfaces might be better for this purpose. But I don't have a clue on how to make that syntax clear when using the gen. |
|
That's what Earth Citizen was saying with Series and Gen but you'll need a
way to make it work.
In the test method you're going to say something like forAll(1000) { ... }.
You probably won't know if the gen is exhaustive without looking at the
impl. For example a Gen for enums. You'd have to look at the impl to know
how many there are.
So if you ask for a Series it's kind of redundant to also ask for the
number of iterations. That's built into a Series but not a random Gen.
So you could move the iteration count and declare it in a different way
(omitting it entirely for Series).
It might work but we need to see a concrete syntax suggestion.
…On Sat, 14 Dec 2019, 19:18 Leonardo Colman Lopes, ***@***.***> wrote:
Take a look at
I would like a way to have this informed without having to look at the
code. When dealing with a big code base, having to look if the Gen is
exhaustive or not might be a little bit troublesome. And even more
problematic if the Gen is very complex.
Perhaps splitting exhaustivity and randomness into different interfaces
might be better for this purpose. But I don't have a clue on how to make
that syntax clear when using the gen.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#1121>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFVSGWBSDTBUABADD3AAALQYWAWVANCNFSM4J2MINTA>
.
|
|
A point I'd like to raise is This seems very long and not really reliable, as I'm skipping constants for example. I was trying to figure out a way to improve this kind of generation, but I couldn't come up with anything. |
I'll try to think about that for some time. Perhaps @EarthCitizen have some ideas in that regard |
|
You either need to specify a series and bounds and have it derive the
iteration count from that. Or specify the iteration count and the bounds
and have the gen work out if it can be exhaustive or not.
Pick your poison.
…On Sat, 14 Dec 2019, 20:32 Leonardo Colman Lopes, ***@***.***> wrote:
It might work but we need to see a concrete syntax suggestion.
I'll try to think about that for some time. Perhaps @EarthCitizen
<https://github.com/EarthCitizen> have some ideas in that regard
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#1121>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFVSGUXLB24JC53FAKYVKDQYWJMLANCNFSM4J2MINTA>
.
|
|
OK. So, here is a rough idea of how the sealed classes idea could work. This is not intended to be complete, but rather a POC on the sealed classes approach. Goals:
What is not included here (but not incompatible):
sealed class Gen<T>
abstract class Arbitrary<T> : Gen<T>() {
fun asIterable(random: Random = Random) = Iterable {
object : Iterator<T> {
override fun hasNext(): Boolean = true
override fun next(): T = generate(random)
}
}
fun asSequence(random: Random = Random) = Sequence { asIterable(random).iterator() }
abstract fun generate(r: Random): T
companion object {
fun ints() = object : Arbitrary<Int>() {
override fun generate(r: Random): Int = r.nextInt()
}
fun longs() = object : Arbitrary<Long>() {
override fun generate(r: Random): Long = r.nextLong()
}
}
}
abstract class Scaled<T> : Gen<T>() {
abstract fun scale(s: Int): T
companion object {
const val SCALE_MIN = 0
const val SCALE_MAX = 99
const val SCALE_STEPS = 100
fun positiveInt() = object : Scaled<Int>() {
override fun scale(s: Int): Int = when(s) {
SCALE_MIN -> 0
SCALE_MAX -> Int.MAX_VALUE
else -> (Int.MAX_VALUE / SCALE_STEPS) * s
}
}
}
}
// Constructor private in order to enforce finite iterable
class Series<T> private constructor(private val i: Iterable<T>) : Gen<T>(), Iterable<T> {
override fun iterator(): Iterator<T> = i.iterator()
companion object {
fun <T> of(s: Set<T>) = Series(s)
fun of(r: CharRange) = Series(r)
fun of(r: IntRange) = Series(r)
}
} |
|
I do like the nomenclature of Series and Arbitrary as both subclasses of Gen (even though I think Arbitrary means something different in Quickcheck style property test frameworks). I tend to drive my API design from a users perspective and not from an implementation one. So lets consider how this would work. If we have forAll(Series.int(100), Arbitrary.strings(20)) { ... }In this example, the first gen is an 1000 int series - every int from 1 to 100 inclusive. You don't need to specify iterations because its implicit in the fact it's a series. The second gen is 20 random strings from across the string continuum. If we say that iteration count is the multiple of gen counts, then this could work. We have 100 x 20 iterations (2000 iterations). This means it would work if we have say a series for a enum. forAll(Series.enum<Season>(), Arbitrary.strings(250)) { ... }In this example, the four seasons as the first param x 250 random strings = 1000 iterations. I'm not sure we need this Scaled thing at all. If you want to "scale" whatever that is, then it can be a parameter to the int gen. Gen.int(some bound). In this design, the exhaustive parameter goes away because it's now explicit in the generator type. We've taken care of the iteration count. Seed could still be passed in, but only handed off to generators and not series. If we still want to allow implied generators, we can add backwards compatible forAlls that derive gens from the type parameter using reflection like now. |
|
Yeah, In the case of |
|
I'm not clear on why you need Scaled as a type, when "scale" (or distribution) can be a parameter to those gen's that support the concept (like number based ones). |
|
Yeah, I don't disagree. I was just explaining what I was trying to explain originally. |
|
Ok, do you think it's worth pursuing that idea, or going down the line using distribution with the Arbitraries ? |
|
Just thinking out loud, defining things with my own words to ensure I am not confused:
Please correct me on anything that I appear to be confused on. |
|
So my take on it is slightly different. Series is for an exhaustive list of values. Maybe series is the wrong term. Arbitrary is for "random" values. I'm not sure scale is needed as you can pass scale directly into the constructors of the arbitaries that support it. |
|
Yeah, I think we are saying the same thing. Scale (as a parameter) would be equivalent to the But, I concede that this could be extra functionality that is not truly needed. |
|
I see. I think it's probably more complicated than useful. You probably know the ranges when you're specifying the generators. |
|
Here is another implementation proposal based on feedback received so far. tl;dr - there are two types of Gen. The arbitrary instances generate values that include a function for returning shrunk values. These shrunk values themselves then contain a function for further shrunk values and so on. This mechanism allows us to map and filter on gens and preserve the shrinking functionality. The forAll method provides for minSuccess, maxFailure, a seed for the random instance, and a shrinking mode which can be used to disable shrinking, or bound the number of shrinks before stopping. The iteration count for the property test is taken as the combination of the length of the sequences generated from the Gen instances). So for progressions, that's the number of values in the closed set. For arbitraries you must pass in how many you want. This is better I believe because if you want to test 1000 random numbers in a function that also accepts a boolean, you can now have 1000 x true and 1000 x false, rather than 1000 x (random int, random boolean). Code follows: /**
* A Generator, or [Gen] is responsible for generating data to be used in property testing.
* Each generator will generate data for a specific type <T>.
*
* There are two supported types of generators - [Arbitrary] and [Progression] - defined
* as sub interfaces of this interface.
*
* An arbitrary is used when you need random values across a large space.
* A progression is useful when you want all values in a smaller closed space.
*
* Both types of generators can be mixed and matched in property tests. For example,
* you could test a function with 1000 random positive integers (arbitrary) and every
* even number from 0 to 200 (progression).
*/
interface Gen<T> {
fun generate(random: Random): Sequence<PropertyInput<T>>
}/**
* An [Arbitrary] is a type of [Gen] which generates two types of values: edge cases and random values.
*
* Edge cases are values that should usually be included on every test run: those edge cases
* which are common sources of bugs. For example, a function using ints is more likely to fail
* for common edge cases like zero, minus 1, positive 1, [Int.MAX_VALUE] and [Int.MIN_VALUE]
* rather than random values like 965489.
*
* The random values are used to give us a greater breadth to the test cases. In the case of a
* function using ints, these random values could be from across the entire integer number line,
* or could be limited to a subset of the integer space.
*/
interface Arbitrary<T> : Gen<T> {
/**
* Returns the values that are considered common edge case for the type.
*
* For example, for [String] this may include the empty string, a string with white space,
* a string with unicode, and a string with non-printable characters.
*
* The result can be empty if for type T there are no common edge cases.
*
* @return the common edge cases for type T.
*/
fun edgecases(): Iterable<T>
/**
* Returns a sequence of random [PropertyInput] values to be used for testing.
*
* @param random the [Random] instance to be used for generating values. This random instance is
* seeded using the seed provided to the test framework so that tests can be deterministically re-run.
* Implementations should honour the random provider whenever possible.
*
* @return the random test values as instances of [PropertyInput].
*/
fun randoms(random: Random): Sequence<PropertyInput<T>>
override fun generate(random: Random): Sequence<PropertyInput<T>> =
edgecases().map { PropertyInput(it) }.asSequence() + randoms(random)
companion object
}fun <T, U> Arbitrary<T>.map(f: (T) -> U): Arbitrary<U> = object : Arbitrary<U> {
override fun edgecases(): Iterable<U> = this@map.edgecases().map(f)
override fun randoms(random: Random): Sequence<PropertyInput<U>> =
this@map.randoms(random).map { it.map(f) }
}fun <T> Arbitrary<T>.filter(predicate: (T) -> Boolean): Arbitrary<T> = object : Arbitrary<T> {
override fun edgecases(): Iterable<T> = this@filter.edgecases().filter(predicate)
override fun randoms(random: Random): Sequence<PropertyInput<T>> =
this@filter.randoms(random).filter { predicate(it.value) }
}/**
* A [Progression] is a type of [Gen] which generates a deterministic, repeatable, and finite set
* of values for a type T.
*
* An example of a progression is the range of integers from 0 to 100.
* Another example is all strings of two characters.
*
* A progression is useful when you want to generate an exhaustive set of values from a given
* sample space, rather than random values from that space. For example, if you were testing a
* function that used an enum, you might prefer to guarantee that every enum value is used, rather
* than selecting randomly from amongst the enum values (with possible duplicates and gaps).
*
* Progressions do not shrink their values. There is no need to find a smaller failing case, because
* the smaller values will themselves naturally be included in the tested values.
*/
interface Progression<T> : Gen<T> {
/**
* @return the values for this progression as a lazy list.
*/
fun values(): Sequence<T>
override fun generate(random: Random): Sequence<PropertyInput<T>> = values().map { PropertyInput(it) }
companion object
}Some sample progressions: fun Progression.Companion.int(range: IntRange) = object : Progression<Int> {
override fun values(): Sequence<Int> = range.asSequence()
}
fun Progression.Companion.azstring(range: IntRange) = object : Progression<String> {
private fun az() = ('a'..'z').asSequence().map { it.toString() }
override fun values(): Sequence<String> = range.asSequence().flatMap { size ->
List(size) { az() }.reduce { acc, seq -> acc.zip(seq).map { (a, b) -> a + b } }
}
}
/**
* Returns a [Progression] of the two possible boolean values - true and false.
*/
fun Progression.Companion.bools() = object : Progression<Boolean> {
override fun values(): Sequence<Boolean> = sequenceOf(true, false)
}Some sample arbitraries: fun Arbitrary.Companion.int(
iterations: Int,
range: IntRange = Int.MIN_VALUE..Int.MAX_VALUE,
distribution: IntDistribution = IntDistribution.Uniform
) = object : Arbitrary<Int> {
override fun edgecases(): Iterable<Int> = listOf(Int.MIN_VALUE, Int.MAX_VALUE, 0)
override fun randoms(random: Random): Sequence<PropertyInput<Int>> {
return sequence {
for (k in 0 until iterations) {
val block = distribution.get(k, iterations, range.first.toLong()..range.last.toLong())
val next = random.nextLong(block).toInt()
val input = PropertyInput(next, IntShrinker)
yield(input)
}
}
}
}
sealed class IntDistribution {
abstract fun get(k: Int, iterations: Int, range: LongRange): LongRange
/**
* Splits the range into discrete "blocks" to ensure that random values are distributed
* across the entire range in a uniform manner.
*/
object Uniform : IntDistribution() {
override fun get(k: Int, iterations: Int, range: LongRange): LongRange {
val step = (range.last - range.first) / iterations
return (step * k)..(step * (k + 1))
}
}
/**
* Values are distributed according to the Pareto distribution.
* See https://en.wikipedia.org/wiki/Pareto_distribution
* Sometimes referred to as the 80-20 rule
*
* tl;dr - more values are produced at the lower bound than the upper bound.
*/
object Pareto : IntDistribution() {
override fun get(k: Int, iterations: Int, range: LongRange): LongRange {
// this isn't really the pareto distribution so either implement it properly, or rename this implementation
val step = (range.last - range.first) / iterations
return 0..(step * k + 1)
}
}
}
fun Arbitrary.Companion.long(
iterations: Int,
range: LongRange = Long.MIN_VALUE..Long.MAX_VALUE
) = object : Arbitrary<Long> {
override fun edgecases(): Iterable<Long> = listOf(Long.MIN_VALUE, Long.MAX_VALUE, 0)
override fun randoms(random: Random): Sequence<PropertyInput<Long>> {
return sequence {
for (k in 0 until iterations) {
val next = random.nextLong(range)
val input = PropertyInput(next, LongShrinker)
yield(input)
}
}
}
}
/**
* Returns a stream of values where each value is a randomly
* chosen Double.
*/
fun Arbitrary.Companion.double(): Arbitrary<Double> = object : Arbitrary<Double> {
val literals = listOf(
0.0,
1.0,
-1.0,
1e300,
Double.MIN_VALUE,
Double.MAX_VALUE,
Double.NEGATIVE_INFINITY,
Double.NaN,
Double.POSITIVE_INFINITY
)
override fun edgecases(): Iterable<Double> = literals
override fun randoms(random: Random): Sequence<PropertyInput<Double>> {
return generateSequence {
val d = random.nextDouble()
PropertyInput(d, DoubleShrinker)
}
}
}
fun Arbitrary.Companion.positiveDoubles(): Arbitrary<Double> = double().filter { it > 0.0 }
fun Arbitrary.Companion.negativeDoubles(): Arbitrary<Double> = double().filter { it < 0.0 }
/**
* Returns an [Arbitrary] which is the same as [Arbitrary.double] but does not include +INFINITY, -INFINITY or NaN.
*
* This will only generate numbers ranging from [from] (inclusive) to [to] (inclusive)
*/
fun Arbitrary.Companion.numericDoubles(
from: Double = Double.MIN_VALUE,
to: Double = Double.MAX_VALUE
): Arbitrary<Double> = object : Arbitrary<Double> {
val literals = listOf(0.0, 1.0, -1.0, 1e300, Double.MIN_VALUE, Double.MAX_VALUE).filter { it in (from..to) }
override fun edgecases(): Iterable<Double> = literals
override fun randoms(random: Random): Sequence<PropertyInput<Double>> {
return generateSequence {
val d = random.nextDouble()
PropertyInput(d, DoubleShrinker)
}
}
}And some example tests you might write: fun main() {
// tests 1000 random ints with all integers 0 to 100
forAll(
Arbitrary.int(1000),
Progression.int(0..100)
) { a, b ->
a + b == b + a
}
// tests 10 random longs in the range 11 to MaxLong with all combinations of a-z strings from 0 to 10 characters
forAll(
Arbitrary.long(10, 11..Long.MAX_VALUE),
Progression.azstring(0..10)
) { a, b ->
b.length < a
}
// convenience functions which mimics the existing style
// tests random longs, using reflection to pick up the Arbitrary instances
// each arb will have the same number of iterations
forAll<Long, Long>(1000) { a, b -> a + b == b + a }
}You can see all this in the io.kotest.property package in module kotest-assertions. |
|
Thanks for the link @sksamuel! Skimming through the comments, just some initial thoughts. Not sure if I understand how #1135 addresses #472 yet. One way is to strengthen compositionality, but I'm not sure if you want to support fully monadic binding as things start to become complicated pretty quickly. For example, to support composition with generator tuplings like we implemented in #506, if two generators accept a common input, shrinking becomes a multi-objective optimization problem in which there is not only one, but a set of solutions along the Pareto frontier. In general, shrinking becomes a lot harder the more compositional things are. Letting the user manually implement the shrinker might be less elegant, but seems more practical and aligns with QuickCheck's implementation. I like the idea of encapsulating the value and shrinker in |
|
I think most cases have some standard default values. |
|
First of all, everything here is open to change. For #472 the idea is that the Of course, if you have bind and it has 8 inputs, you could be potentially shrinking 8 ways, so it's better to write something custom. We will support that (in the current rework, you would implement your own Arb and include the shrinker in that, but perhaps we do-couple them). Some parameters are passed to the forAll method (or the forNone, forSome, whatever). And some parameters are passed to the generators themselves.
A possible example of a prop test in the current design with all the knobs. forAll(
Arb.int(1000, Distribution.Pareto), // 1000 randoms from the entire int range, biased towards 0
Arb.double(10, 1..100.0), // 10 random doubles in the range 1..100.0
Prog.enum<CustomerType>, // every value from the enum CustomerType
args = PropTestArgs(
seed = 112549833451L, // setting the random seed
shrinking = ShrinkingMode.Bounded(1000) // no more than 1000 shrinks per arg
)
) { a,b,c -> ... }This would give you 1000 x 10 x |CustomerType| combinations. I propose to also keep convenience methods, like this: forAll<Int, Double, CustomerType>(1000) { a,b,c -> ... }Which will take 333 from each parameter to get you back to ~1000. |
|
One impact to the new API for us is that currently we are using the generators outside of property based testing, simply to generate random data for traditional TDD based tests. I don't know whether others are also doing this; but I find it to be a nice capability to be able to use the same generators outside of the property based testing? Because of the new API needing to be passed the random and because of the PropertyInput wrapper, it is more more involved to do this: Previously: I guess we can write our own extension functions to hide the details; but if others use the generators in this way, maybe this should be considered in the updated API design? |
|
I agree with @DannyRyman. I use I believe we intend to support this, and if we don't ATM, we surely will before a release candidate is ready. It also fits property testing concepts, as I'm escaping example-based testing by just testing that my function works for any given user (even when checking it only once) |
|
Yes we will definitely support this. If it's hard to do with the current
API then lets change the API.
…On Tue, 14 Jan 2020 at 09:51, Leonardo Colman Lopes < ***@***.***> wrote:
I agree with @DannyRyman <https://github.com/DannyRyman>.
I use
UserGen.next() a lot, for example, when I just need a random user.
I believe we intend to support this, and if we don't ATM, we surely will
before a release candidate is ready
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1121>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFVSGVZ7I3DECFX4SYP75TQ5XNPVANCNFSM4J2MINTA>
.
|
I believe we can probably add these extensions to Kotest itself :P |
|
It's actually what we do with |
|
We could have an extension method on Arbitrary called |
|
That's what we do with Gen, yes
…On Tue., Jan. 14, 2020, 15:07 Stephen Samuel, ***@***.***> wrote:
We could have an extension method on Arbitrary called next which does the
plumbing for you - putting in a default random instance and taking the
first value.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1121>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAMBCI4GHIV3FOERDQ6US7LQ5X5OVANCNFSM4J2MINTA>
.
|
|
Doesn't make sense for progression tho
On Tue., Jan. 14, 2020, 15:28 Leonardo Lopes, <leonardo.colman98@gmail.com>
wrote:
… That's what we do with Gen, yes
On Tue., Jan. 14, 2020, 15:07 Stephen Samuel, ***@***.***>
wrote:
> We could have an extension method on Arbitrary called next which does
> the plumbing for you - putting in a default random instance and taking the
> first value.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#1121>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAMBCI4GHIV3FOERDQ6US7LQ5X5OVANCNFSM4J2MINTA>
> .
>
|
|
Try writing next() for an Arb. You can't because Arbs are created with a size parameter, so there's nothing stopping you doing: Arb.int(0).next() // create an arb of size 0 and ask for nextwhich would throw an error. So for this to work the size parameter has to move out of Arb's constructors and into the interface method. So generate takes a size. This means that you would need to do forAll like this: forAll(1000, Arb.int(), Arb.double()) { a, b -> ... }Like we do now. But then, if you do forAll(2, Arb.int(), Prog.enum<Season>()) { a, b -> ... }There are 4 seasons and you've only asked for 2 iterations, so it's not exhaustive. So to enforce exhaustivity we need to introduce a parameter to choose whether we care about exhaustivity or not. And what happens if you ask for 10 iterations on a progression of size 3 - do you just loop back around again? We could throw an exception if you ask for iterations < |max_progression_size|. Or we could introduce a new interface, so you start off with an forAll(Arb.int().generate(100), Arb.double().generate(200), Prog.int(0, 10)) { a, b, c -> ... }Then you could do everything we need, but the usability is dropping each time we increase complexity. |
|
If the default size for |
|
Then we can throw an |
|
That would work.
…On Tue, 14 Jan 2020, 17:15 Leonardo Colman Lopes, ***@***.***> wrote:
Then we can throw an IllegalArgumentException if you try to do
Arbitrary.int(0).next()
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1121>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFVSGT4AOZQZ7HCRQZ3AUDQ5ZBSZANCNFSM4J2MINTA>
.
|
|
Just throwing another brain storm out here. Rename forAll(
1000,
Arbitrary.ints(),
Arbitrary.doubles(),
Exhausive.int(0, 100)
) { a, b, c -> .... }This generates 1000 tests pulling from the two Arbs. The Exhausive provides 100 values and then cycles. If the iteration count is less than the max of any exhaustives an exception is thrown. In other words, an exhaustive generator will throw an error if you try to pull less iterations than it generates. Pros - less setup for user, can use a gen to provide single values. |
It's not necessary to move the iterations to |
|
Ok everyone, another iteration to the prop test proposals and I think we're getting closer on a final implementation (if everyone agrees!). Code here: #1174 What's changed?
forAll(Gen.int().take(100, ShrinkingMode.Off)) { a -> .... }
The end result is code like this: forAll(
Gen.int(0..500).take(100),
Gen.positiveInts().take(10),
Exhaustive.enum<SomeEnum>()
) { a, b, c ->
// test
}And we'll support convenience functions like: Gen.int().forAll(1000) { a -> ... }and forAll<Int, Int>(1000) { a, b -> ... }Please feedback as quickly as you like. Once the property test changes are finalised (or thereabouts) we can move 4.0 to a beta release. |
The only issue I see with this is if people want to use the module outside of KotlinTest environment. However, Kotlin environment itself is very easy to integrate with coroutines, so this won't be much more than an annoyance.
I suppose these values will have defaults. I think this is great! However...
Are we going to have I think this will be a little bit confusing. At least it is confusing for me right now, maybe because I'm already in the mindset of |
|
A Gen becomes an Arb when you call .take()
Arb and Exhausitve are used in property testing. Gen's can be used anywhere
you want to produce a value.
…On Thu, 23 Jan 2020 at 09:41, Leonardo Colman Lopes < ***@***.***> wrote:
All prop test methods are now suspendable, rather than inline, so they
compile much quicker. The downside is you need to execute them in a
coroutine, but KT is fully enabled for coroutines in every test scope
transparently to the user, so I don't forsee this being an issue
The only issue I see with this is if people want to use the module outside
of KotlinTest environment. However, Kotlin environment itself is very easy
to integrate with coroutines, so this won't be much more than an annoyance.
And we'll support convenience functions like:
Gen.int().forAll(1000) { a -> ... }
I suppose these values will have defaults. I think this is great!
However...
Gen has been restored as a top level construct
Are we going to have Gen, Exhaustive AND Arbitrary?
Is Exhaustive a Gen?
I think this will be a little bit confusing. At least it is confusing for
me right now, maybe because I'm already in the mindset of Arb and Prog.
I'll wait for an API proposal to comment further
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1121>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFVSGQAROKEY6N4LVGVTWTQ7G3D5ANCNFSM4J2MINTA>
.
|
|
Happy to rename things.
…On Thu, 23 Jan 2020 at 10:08, Stephen Samuel (Sam) ***@***.***> wrote:
A Gen becomes an Arb when you call .take()
Arb and Exhausitve are used in property testing. Gen's can be used
anywhere you want to produce a value.
On Thu, 23 Jan 2020 at 09:41, Leonardo Colman Lopes <
***@***.***> wrote:
> All prop test methods are now suspendable, rather than inline, so they
> compile much quicker. The downside is you need to execute them in a
> coroutine, but KT is fully enabled for coroutines in every test scope
> transparently to the user, so I don't forsee this being an issue
>
> The only issue I see with this is if people want to use the module
> outside of KotlinTest environment. However, Kotlin environment itself is
> very easy to integrate with coroutines, so this won't be much more than an
> annoyance.
>
> And we'll support convenience functions like:
> Gen.int().forAll(1000) { a -> ... }
>
> I suppose these values will have defaults. I think this is great!
>
> However...
>
> Gen has been restored as a top level construct
>
> Are we going to have Gen, Exhaustive AND Arbitrary?
> Is Exhaustive a Gen?
>
> I think this will be a little bit confusing. At least it is confusing for
> me right now, maybe because I'm already in the mindset of Arb and Prog.
> I'll wait for an API proposal to comment further
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#1121>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAFVSGQAROKEY6N4LVGVTWTQ7G3D5ANCNFSM4J2MINTA>
> .
>
|
|
All the recent work has been merged to master. I think its stable enough to be used. |
I am working on overhauling our property testing as part of the upcoming 4.0 release. To this end, I have brought together the requirements (based off tickets existing in this tracker), and come up with a basic design. I would like feedback on this design before I fully implement it. There are also some questions that I don't have an answer to yet that I would like to discuss. At this stage everything is open to change.
Property Test Requirements
Deterministic Re-runs
If a test failed it is useful to be able to re-run the tests with the same values. Especially in cases where shrinking is not available. Therefore, the test functions accept a seed value which is used to create the Random instance used by the tests. This seed can then be programatically set to re-run the tests with the same random instance.
By default the seed is null, which means the seed changes each time the test is run.
Exhaustive
The generators are passed an Exhaustivity enum value which determines the behavior of generated values.
By default Auto mode is used.
Question - do we want to be able to specify exhaustivity per parameter?
In #1101 @EarthCitizen talks about
SeriesvsGen. I do like the nomenclature, but we would need another abstraction (Arbitrary?) on top which would then be able to provide a Gen or Series as required based on the exhaustive flag.Question - do we want to implement this way as opposed to the way I have outlined in the code below
Min and Max Successes
These values determine bounds on how many tests should pass. Typically min and max success would be equal to the iteration count, which gives the
forAllbehavior.For
forNonebehavior, min and max would both be zero. Other values can be used to mimic behavior likeforSome,forExactly(n)and so on.By default, min and max success are set to the iteration count.
Distribution
It is quite common to want to generate values across a large number space, but have a bias towards certain values. For example, when writing a function to test emails, it might be more useful to generate more strings with a smaller number of characters than larger amounts. Most emails are probably < 50 characters for example.
The distribution mode can be used to bias values by setting the bound from which each test value is generated.
By default the uniform distribution is used.
The distribution mode may be ignored by a generator if it has no meaning for the types produced by that generator.
The distribution mode has no effect if the generator is acting in exhaustive mode.
Question - it would be nice to be able to specify specific "biases" when using specific generators. For example, a generator of A-Z chars may choose to bias towards vowels. How to specify this when distribution is a sealed type? Use an interface and allow generators to create their own implementations?
Shrinking Mode
The ShrinkingMode determines how failing values are shrunk.
By default shrinking is set to Bounded(1000).
Question1 - do we want to be able to control shrinking per parameter? Turn it off for some parameters, and not others?
When mapping on a generator, shrinking becomes tricky.
If you have a mapper from GenT to GenU and a value u fails, you need to turn that u back into a t, so you can feed that t into the original shrinker. So you can either keep the association between the original value and mapped value, or precompute (lazily?) shinks along with the value.
Question2 - which is the best approach?
Gen design
The gens accept the Random instance used for this run.
They accept an iterations parameter so they know the sample space when calculating based on a distribution
They accept the exhausitivity mode and the distribution mode.
Question - move the iteration count into the distribution parameter itself?
Note that the gens no longer specify a shrinker, but should provide the shrinks along with the value (see shrinker section for discussion).
The text was updated successfully, but these errors were encountered: