Skip to content

Latest commit

 

History

History
2593 lines (1480 loc) · 31.4 KB

yownight2017-find-more-bugs.md

File metadata and controls

2593 lines (1480 loc) · 31.4 KB
Raw

class: center, bottom, heading-black background-image: url(images/example-based.jpeg)

Finding More Bugs With Less Effort

@charlesofarrell

???

  • We basically write property-based tests exclusively
  • Property testing is a powerful tool for testing

class: center, bottom, heading-black background-image: url("images/chopsticks.jpg")

@charlesofarrell

???

  • Property-based testing is our default
  • We have a majority of property-based tests
  • Changed my life and the way I test
  • I write better code now

class: bottom, left, heading-white background-image: url(images/goal.jpeg)

Motivation?

???

  • Why do we test code in the first place?
  • I believe we all want to write correct and reliable software
  • Safety
  • At Ambiata it costs $$$
  • But, it's very hard
  • Testing is one tool to achieve that

class: bottom, right, heading-black background-image: url(images/example-based.jpeg)

Testing

???

  • Testing is first line of defence

class: bottom, right, heading-black background-image: url(images/example-based.jpeg)

Example-based Testing

???

  • The hard part is coming up with the examples
  • I want to demonstrate now...

class: code

def substring(s: String, i: Int): String

class: code

def substring(s: String, i: Int): String

def testSubstring = {
  substring("abc", 1) == "bc"
}

class: code

def substring(s: String, i: Int): String

def testSubstring = {
  substring("abc", 1) == "bc"
  substring("abc", 3) == ""
}

class: code

def substring(s: String, i: Int): String

def testSubstring = {
  substring("abc", 1) == "bc"
  substring("abc", 3) == ""
  substring("돪돪", 1) == "돪"
}

class: code

def substring(s: String, i: Int): String

def testSubstring = {
  substring("abc", 1) == "bc"
  substring("abc", 3) == ""
  substring("돪돪", 1) == "돪"
  substring("abc", 4) == ""
}

class: code

def substring(s: String, i: Int): String

def testSubstring = {
  substring("abc", 1) == "bc"
  substring("abc", 3) == ""
  substring("돪돪", 1) == "돪"
  substring("abc", 4) == ""
  substring("abc", -1) == "c"
}

???

  • How many examples is enough?

class: middle, center

???

  • Writing good examples means knowing about the internals of the functions
  • Doesn't help with other systems or libraries, or new edge cases

background-image: url(images/computer.jpeg)

???

  • Let's get the computer to generate them instead!

class: middle, center

???

  • Year 2000!

class: middle

class: middle

class: middle

http://www.quviq.com/volvo-quickcheck/

http://www.quviq.com/volvo-quickcheck/

  • 3,000 pages of specifications
  • 1,000,000 LOC, 6 suppliers

http://www.quviq.com/volvo-quickcheck/

  • 3,000 pages of specifications
  • 1,000,000 LOC, 6 suppliers
  • 20,000 lines of QuickCheck
  • 200 problems
  • 100 problems in the standard

https://jepsen.io/

  • MongoDB
  • Cassandra
  • RabbitMQ
  • Redis
  • Riak
  • Zookeeper
  • VoltDB
  • RethinkDB
  • Elasticsearch

???

  • Kyle Kingsbury
  • "Call me maybe"
  • Distributed systems are hard
  • Documentation is usually wrong
  • If you're using one of these DBs go an read them
  • Has changed how people test DBs
  • Industry standard
  • Badge of honour

class: middle, center

https://en.wikipedia.org/wiki/QuickCheck

31 Languages

???

  • Using ScalaCheck, a derivitive of QuickCheck for Scala
  • I want to empaphasise, this is not about Scala
  • This talk is about property testing

class: top, left, heading-black background-image: url(images/fuzzy.jpg)

Fuzzing

???

  • Throw random values at code/systems
  • Usually for security purposes

class: code

def substring(s: String, i: Int): String

def testSubstring =



    substring("abc", 1) == "bc"

class: code

def substring(s: String, i: Int): String

def testSubstring =



    substring("abc", 1)







    substring("abc", 1) == "bc"

class: code

def substring(s: String, i: Int): String

def testSubstring =



    substring("abc", 1)

class: code




  generateString      s


              s

def substring(s: String, i: Int): String

def testSubstring =
  generateString    { s =>


    substring(s, 1)
  }

class: code

 


  forAll(         )





def forAll[A](Gen[A], A => Prop): Prop

def substring(s: String, i: Int): String

def testSubstring =
  forAll(genString) { s =>


    substring(s, 1)
  }

def forAll[A](Gen[A], A => Prop): Prop

class: code

 


         genString







def genString: Gen[String]

def substring(s: String, i: Int): String

def testSubstring =
  forAll(genString) { s =>


    substring(s, 1)
  }

def forAll[A](Gen[A], A => Prop): Prop

def genString: Gen[String]

???

PAUSE

class: code





         genInt    i

                 i






def genInt: Gen[Int]

def substring(s: String, i: Int): String

def testSubstring =
  forAll(genString) { s =>
  forAll(genInt) { i =>

    substring(s, i)
  }}

def forAll[A](Gen[A], A => Prop): Prop

def genString: Gen[String]

def genInt: Gen[Int]

class: code

 

def testSubstring =
  forAll(genString) { s =>
  forAll(genInt) { i =>

    substring(s, i)
  }}

def substring(s: String, i: Int): String

def testSubstring =
  forAll(genString) { s =>
  forAll(genInt) { i =>

    substring(s, i)
  }}

???

PAUSE

class: code

def substring(s: String, i: Int): String

def testSubstring =
  forAll(genString) { s =>
  forAll(genInt) { i =>

    substring(s, i)
  }}

! testSubstring: StringIndexOutOfBoundsException
    String index out of range: 0
> ARG_0: ""
> ARG_1: 0

class: code

 




  i < s.length ==>



def ==>(p: Boolean, Prop): Prop

def substring(s: String, i: Int): String

def testSubstring =
  forAll(genString) { s =>
  forAll(genInt) { i =>
  i < s.length ==>
    substring(s, i)
  }}

def ==>(p: Boolean, Prop): Prop

class: code

def substring(s: String, i: Int): String

def testSubstring =
  forAll(genString) { s =>
  forAll(genInt) { i =>
  i < s.length ==>
    substring(s, i)
  }}

! testSubstring: StringIndexOutOfBoundsException
    String index out of range: -1
> ARG_0: ""
> ARG_1: -1

class: code

def substring(s: String, i: Int): String

def testSubstring =
  forAll(genString) { s =>
  forAll(genInt) { i =>
  i >= 0 && i < s.length ==>
    substring(s, i)
  }}

class: code

def substring(s: String, i: Int): String

def testSubstring =
  forAll(genString) { s =>
  forAll(genInt) { i =>
  i >= 0 && i < s.length ==>
    substring(s, i)
  }}
 


+ HelloWorld.substring: OK, passed 100 tests.

???

  • Same test found different edge cases (bugs?)
  • Learned about substring

class: code

 








withMinSuccessfulTests(1000)

def substring(s: String, i: Int): String

def testSubstring =
  forAll(genString) { s =>
  forAll(genInt) { i =>
  i >= 0 && i < s.length ==>
    substring(s, i)
  }}

withMinSuccessfulTests(1000)


+ HelloWorld.substring: OK, passed 1000 tests.

???

  • We might lower the number of tests for databases
  • Test doesn't change
  • Could increase for a different build

class: code

 





    substring(s, i) == ???

def substring(s: String, i: Int): String

def testSubstring =
  forAll(genString) { s =>
  forAll(genInt) { i =>
  i >= 0 && i < s.length ==>
    substring(s, i) == ???
  }}

???

  • This is where things get tricky
  • Know from personal experience
  • End up rewriting

class: code

 





    substring(s, i) == substringAgain(s, i)

def substring(s: String, i: Int): String

def testSubstring =
  forAll(genString) { s =>
  forAll(genInt) { i =>
  i >= 0 && i < s.length ==>
    substring(s, i) == substringAgain(s, i)
  }}

???

  • Frustrating
  • Actually not the idea!

class: center

???

  • Patterns
  • Not just writing list sorting functions
  • Can be hard to get started

class: middle, center

Round-trip

class: code

def toBytes(s: String): Array[Byte]

def fromBytes(b: Array[Bytes]): String

class: code

 




forAll(genString) { s =>
  val b = toBytes(s)
  fromBytes(b) == s
}

def toBytes(s: String): Array[Byte]

def fromBytes(b: Array[Bytes]): String


forAll(genString) { s =>
  val b = toBytes(s)
  fromBytes(b) == s
}

class: code

 




forAll(genString) { s =>
  val b = toBytes(s)
  fromBytes(b) == s
}

def toBytes(s: String): Array[Byte]

def fromBytes(b: Array[Bytes]): String


forAll(genString) { s =>
  val b = toBytes(s)
  fromBytes(b) == s
}

"돪" != "?"

> ARG_0: "돪"

class: code

                       c: Charset


                    c: Charset


       genCharset

def toBytes(s: String, c: Charset): Array[Byte]

def fromBytes
  (b: Array[Bytes], c: Charset): String

forAll(genString) { s =>
forAll(genCharset) { c =>
  val b = toBytes(s, c)
  fromBytes(b, c) == s
}}

class: code

                       c: Charset


                    c: Charset


       genCharset

def toBytes(s: String, c: Charset): Array[Byte]

def fromBytes
  (b: Array[Bytes], c: Charset): String

forAll(genString) { s =>
forAll(genCharset) { c =>
  val b = toBytes(s, c)
  fromBytes(b, c) == s
}}

"돪" != "?"

> ARG_0: "돪"
> ARG_1: windows-1252

class: code

import org.joda.time._

forAll(genDateTime) { dt =>

  val formatter = DateTimeFormat.fullDateTime()

  val s = formatter.print(dt)
  formatter.parseDateTime(s) == dt
}

???

  • Symmetrical

class: code

import org.joda.time._

forAll(genDateTime) { dt =>

  val formatter = DateTimeFormat.fullDateTime()

  val s = formatter.print(dt)
  formatter.parseDateTime(s) == dt
}
Invalid format:
"Sunday, September 22, 2148 9:08:08 PM ART"
is malformed at "ART"

class: code

def toJson(user: User): Json

def fromJson(json: Json): Option[User]

class: code

def toJson(user: User): Json

def fromJson(json: Json): Option[User]

forAll(genUser) { user =>
  val json = toJson(user)
  fromJson(json) == Some(user)
}

class: code

def toJson(user: User): Json

def fromJson(json: Json): Option[User]

forAll(genUser) { user =>
  val json = toJson(user)
  fromJson(json) == Some(user)
}

2015-01-01T12:57:39.123456
  != 2015-01-01T12:57:39.123

???

class: code

def insertUser(u: User): UserId

def getUser(u: UserId): Option[User]

class: code

def insertUser(u: User): UserId

def getUser(u: UserId): Option[User]


forAll(genUser) { user =>

  val id = insertUser(user)
  getUser(id) == Some(user)
}

class: code

def insertUser(u: User): UserId

def getUser(u: UserId): Option[User]


forAll(genUser) { user =>

  val id = insertUser(user)
  getUser(id) == Some(user)
}
Some(User(\NULL)) != Some(User())

???

  • Date/Double precision

class: bottom, right, heading-white background-image: url(images/edges.jpeg)

Edge Cases

???

  • Discovered edge cases in our libraries/systems
  • Only wrote one test

class: middle, center

Test Oracle

class: code

def timSort(l: List[Int]): List[Int]

class: code

def timSort(l: List[Int]): List[Int]


forAll(genList(genInt)) { l =>

  timSort(l) == bubbleSort(l)
}

class: code

def timSort(l: List[Int]): List[Int]


forAll(genList(genInt)) { l =>

  timSort(l) == bubbleSort(l)
}

https://bugs.openjdk.java.net/browse/JDK-8072909

TimSort fails with ArrayIndexOutOfBounds
on worst case long arrays

class: code

def listUsersSortByName: List[User] =
  "SELECT * FROM user ORDER BY name ASC"

class: code

 


forAll(genList(genUser)) { users =>

  users.foreach(u => insertUser(u))

  listUsersSortByName ==
    users.sortBy(_.name)
}

def listUsersSortByName: List[User] =
  "SELECT * FROM user ORDER BY name ASC"

forAll(genList(genUser)) { users =>

  users.foreach(u => insertUser(u))

  listUsersSortByName ==
    users.sortBy(_.name)
}

class: code

 


forAll(genList(genUser)) { users =>

  users.foreach(u => insertUser(u))

  listUsersSortByName ==
    users.sortBy(_.name)
}

def listUsersSortByName: List[User] =
  "SELECT * FROM user ORDER BY name ASC"

forAll(genList(genUser)) { users =>

  users.foreach(u => insertUser(u))

  listUsersSortByName ==
    users.sortBy(_.name)
}

List("a", "b", "A") != List("a", "A", "b")

class: code

case class Date(value: Int)

class: code

case class Date(value: Int)


def toJoda(date: Date): JodaDate

def fromJoda(date: JodaDate): Date

class: code

case class Date(value: Int)


def toJoda(date: Date): JodaDate

def fromJoda(date: JodaDate): Date


forAll { d: Date =>

  d.toJoda.fromJoda == d
}

class: code

def dayPlus(d: Date, i: Int): Date = {
  ...
}

class: code

def dayPlus(d: Date, i: Int): Date = {
  ...
}

forAll { (d: Date, i: Int) =>

  dayPlus(d, i) ==
    d.toJoda.plusDays(i).fromJoda
}

class: code

def dayPlus(d: Date, i: Int): Date = {
  ...
}

forAll { (d: Date, i: Int) =>

  dayPlus(d, i) ==
    d.toJoda.plusDays(i).fromJoda
}
Date(2004,2,29) != Date(2004,3,1)

ARG_0: Date(2004,2,28)
ARG_1: 1

class: bottom, left, heading-white background-image: url(images/property-based.jpeg)

Find More Bugs

class: center, middle

Idempotence

class: code

forAll(genList(genInt)) { s =>

  l.distinct.distinct == l.distinct
}

class: center

"not idempotent"

 fn foo() {
      #[cfg(target_os = "freertos")]
           match port_id {

 fn foo() {
      #[cfg(target_os = "freertos")]
      #[cfg(target_os = "freertos")]
           match port_id {

We do test for idempotency by running the 'target' of every test through rustfmt as a 'source'. Obviously, this only catches bugs if there is a relevant test.

???

rust-lang/rustfmt#1668

We do test for idempotency by running the 'target' of every test through rustfmt as a 'source'. Obviously, this only catches bugs if there is a relevant test.

class: top, center, heading-black background-image: url(images/rebuild.jpg)

Rebuild

???

Hard to prove, easy to verify

class: code

 

def testSubstring =
  forAll(genString) { s =>
  forAll(genInt) { i =>

    substring(s, i) == ???
  }}

def substring(s: String, i: Int): String

class: code

 

def testSubstring =
  forAll(genString) { s =>
  forAll(genString) { t =>

    substring(s + t, s.length) == t
  }}

def substring(s: String, i: Int): String

class: code

 

def testSubstring =
  forAll(genString) { s =>
  forAll(genString) { t =>

    substring(s + t, s.length) == t
  }}

def substring(s: String, i: Int): String

def testSubstring =
  forAll(genString) { s =>
  forAll(genString) { t =>

    substring(s + t, s.length) == t
  }}


substring("a" + "bc", "a".length) == "bc"

substring("xy" + "z", "xy".length) == "z"

???

PAUSE

class: center, middle, heading-black

Invariant

???

  • Need to be combined with multiple properties to test the entire function

class: code

forAll(genString) { s =>
  s.toLowerCase.length == s.length
}

class: code

forAll(genString) { s =>
  s.toLowerCase.length == s.length
}

Expected 2 but got 1
ARG_0: "İ"

Class: right, top, heading-white background-image: url(images/shrink.jpeg)

Shrinking

class: code

forAll(genString) { s =>
  s.toLowerCase.length == s.length
}

class: code

forAll(genString) { s =>
  s.toLowerCase.length == s.length
}

! testLowerCase: Falsified after 89 passed tests.

Expected 15 but got 14

ARG_0:          "()*@#%KFPSDlİDcx;lk1&(#"

class: code

forAll(genString) { s =>
  s.toLowerCase.length == s.length
}

! testLowerCase: Falsified after 89 passed tests.

Expected 15 but got 14
ARG_0: "İ"
ARG_0_ORIGINAL: "()*@#%KFPSDlİDcx;lk1&(#"

class: code

forAll(genInt) { i =>

  i % 2 == 0
}

class: code

forAll(genInt) { i =>

  i % 2 == 0
}

ARG_0: 1
ARG_0_ORIGINAL: 481771087

class: code

forAll(genInt) { i =>
  println(i)
  i % 2 == 0
}

class: code

forAll(genInt) { i =>
  println(i)
  i % 2 == 0
}

15936 -15936 23904 -23904 27888 -27888 29880
-29880 30876 -30876 31374 -31374 31623 0 15812
-15812 23718 -23718 27671 0 13836 -13836 20754
-20754 24213 0 12107 0 6054 -6054 9081 0 4541
0 2271 0 1136 -1136 1704 -1704 1988 -1988 2130
-2130 2201 0 1101 0 551 0 276 -276 414 -414 483
0 242 -242 363 0 182 -182 273 0 137 0 69 0 35
0 18 -18 27 0 14 -14 21 0 11 0 6 -6 9 0 5 0 1

class: center, middle, heading-white background-image: url(images/generators.jpeg)

Generators

???

  • Investment!

class: code

def testInsertUser =


    insertUser("bob")


def insertUser(user: String)

class: code

def testInsertUser =
  forAll(genString) { user =>

    insertUser(user)
  }

class: code

def testInsertUser =
  forAll(genString) { user =>

    insertUser(user)
  }

! testInsertUser:
    Name cannot be blank
> ARG_0: ""

class: code

 

  user != "" ==>

def testInsertUser =
  forAll(genString) { user =>
  user != "" ==>
    insertUser(user)
  }

class: code

 

  user != "" ==>

def testInsertUser =
  forAll(genString) { user =>
  user != "" ==>
    insertUser(user)
  }


! testInsertUser:
    Name cannot contain spaces
> ARG_0: "a b"

class: code

 

  user != "" && !user.contains(" ") ==>

def testInsertUser =
  forAll(genString) { user =>
  user != "" && !user.contains(" ") ==>
    insertUser(user)
  }

class: code

 

  user != "" && !user.contains(" ") && ... ==>

def testInsertUser =
  forAll(genString) { user =>
  user != "" && !user.contains(" ") && ... ==>
    insertUser(user)
  }

class: code

 

  user != "" && !user.contains(" ") && ... ==>

def testInsertUser =
  forAll(genString) { user =>
  user != "" && !user.contains(" ") && ... ==>
    insertUser(user)
  }


Gave up after only 15 passed tests.
85 tests were discarded.

class: code

 
         genList1(choose('a', 'z')




def choose(a: Char, b: Char): Gen[Char]

def genList1[A](g: Gen[A]): Gen[List[A]]

def testInsertUser =
  forAll(genList1(choose('a', 'z'))) { user =>

    insertUser(user)
  }

def choose(a: Char, b: Char): Gen[Char]

def genList1[A](g: Gen[A]): Gen[List[A]]

class: code

 
         genUser




def genUser: Gen[String] =
  genList1(choose('a', 'z'))

def testInsertUser =
  forAll(genUser) { user =>

    insertUser(user)
  }

def genUser: Gen[String] =
  genList1(choose('a', 'z'))

class: code

 
         genUser




def genUser: Gen[String] =
  genList1(choose('a', 'z'))


         genUser

def testInsertUser =
  forAll(genUser) { user =>

    insertUser(user)
  }

def genUser: Gen[String] =
  genList1(choose('a', 'z'))

def testDeleteUser =
  forAll(genUser) { user =>
    deleteUser(user)
  }

class: code

 
         genUser




def genUser: Gen[User] =
  for {
    name <- genList1(choose('a', 'z'))
    age  <- choose(0, 100)
  } yield User(name, age)

def testInsertUser =
  forAll(genUser) { user =>

    insertUser(user)
  }

def genUser: Gen[User] =
  for {
    name <- genList1(choose('a', 'z'))
    age  <- choose(0, 100)
  } yield User(name, age)

class: middle, center

???

True Story - "null" username

class: code



                "null"

class: code

def testMigration =


    migrateUser("bob")


def testMigrationNull =
    migrateUser("null")

???

  • Only tests one code-path

class: code

def testMigration =
  forAll(genUser) { user =>

    migrateUser(user)
  }

def genUser: Gen[String] =

          genList1(choose('a', 'z'))

???

  • Ideally you don't have to add any more tests

class: code

 






  genFrequency(

    1  -> genConst("null")
  )

def testMigration =
  forAll(genUser) { user =>

    migrateUser(user)
  }

def genUser: Gen[String] =
  genFrequency(
    19 -> genList1(choose('a', 'z'))
  , 1  -> genConst("null")
  )

class: bottom, right, heading-white background-image: url(images/state-based.jpeg)

State-based Testing

???

  • "State of the art"
  • Unit test vs integration testing
  • Generate the whole test

class: code

 




forAll(genUser) { user =>

  val id = insertUser(user)
  getUser(id) == Some(user)
}

def insertUser(u: User): UserId

def getUser(u: UserId): Option[User]


forAll(genUser) { user =>

  val id = insertUser(user)
  getUser(id) == Some(user)
}

class: code

 




forAll(genUser) { user =>
forAll(genUser) { user2 =>

  val id = insertUser(user)
  val id2 = insertUser(user2)
  getUser(id) == Some(user)
}

def insertUser(u: User): UserId

def getUser(u: UserId): Option[User]


forAll(genUser) { user =>
forAll(genUser) { user2 =>

  val id = insertUser(user)
  val id2 = insertUser(user2)
  getUser(id) == Some(user)
}

class: code

 




forAll(genUser) { user =>
forAll(genUser) { user2 =>

  val id = insertUser(user)
  val id2 = insertUser(user2)
  getUser(id) == Some(user)
  deleteUser(id)
  getUser(id) == None
}}

def insertUser(u: User): UserId

def getUser(u: UserId): Option[User]


forAll(genUser) { user =>
forAll(genUser) { user2 =>

  val id = insertUser(user)
  val id2 = insertUser(user2)
  getUser(id) == Some(user)
  deleteUser(id)
  getUser(id) == None
}}

class: middle, center

class: code

case class State(users: Map[UserId, User])

class: code

case class State(users: Map[UserId, User])

case class Insert(user: User) { .. }

class: code

case class State(users: Map[UserId, User])

case class Insert(user: User) extend Commands {

  def run: Result =
    ...






}

class: code

case class State(users: Map[UserId, User])

case class Insert(user: User) extend Commands {

  def run: Result =
    ...

  def nextState(state: State): State =
    ...



}

class: code

case class State(users: Map[UserId, User])

case class Insert(user: User) extend Commands {

  def run: Result =
    ...

  def nextState(state: State): State =
    ...

  def postCondition(s: State, r: Result): Prop =
    ...
}

class: code

case class State(users: Map[UserId, User])

case class Insert(user: User) { .. }

case class Get(id: UserId) { .. }

class: code

case class State(users: Map[UserId, User])

case class Insert(user: User) { .. }

case class Get(id: UserId) { .. }

def genCommand: Gen[Command] =
  oneOf(Insert(..), Get(..))

class: code

case class State(users: Map[UserId, User])

case class Insert(user: User) { .. }

case class Get(id: UserId) { .. }
! Falsified after 11 passed tests.

Expected User("a") but got User("b")

Steps:
  Insert(User("a"))
  Insert(User("b"))
  Get(1)

class: middle, center

http://www.quviq.com/google-leveldb/

???

  • LevelDB is a fast key-value storage C++ library written at Google that provides an ordered mapping from string keys to string values.

class: code, thinner

1. open new database
2. put key1 and val1
3. close database
4. open database
5. delete key2
6. delete key1
7. close database
4. open database
5. delete key2
6. delete key1
7. close database
11. open database
12. put key3 and val1
13. close database
14. open database
15. close database
16. open database
17. seek first

???

  • 17 steps.
  • THEN 33 steps

https://groups.google.com/forum/#!topic/leveldb/gnQEgMhxZAs

  • Was in 2013

class: middle, center

???

  • Concurrency is hard!
  • Testint concurrent programmers is really hard!!!

class: code

def property(threadCount: Int): Prop

class: code

def property(threadCount: Int): Prop
! transfer: Falsified after 9 passed tests.

Expected User("a") but got User("b")

Sequential:
  1. Insert(User("a")) => 1
  2. Insert(User("b")) => 2

Parallel:
  1. Get(1) => User("a")
  2. Get(1) => User("b")

class: middle

???

2009

class: bottom, left, heading-white background-image: url(images/goal.jpeg)

Motivation

class: bottom, right, heading-black background-image: url(images/example-based.jpeg)

Example-based Testing

background-image: url(images/computer.jpeg)

class: top, left, heading-black background-image: url(images/fuzzy.jpg)

Fuzzing

class: center, middle, heading-white

class: center, middle

class: center, middle

class: top, center, heading-black background-image: url(images/rebuild.jpg)

Rebuild

class: right, top, heading-white background-image: url(images/shrink.jpeg)

Shrinking

class: center, middle, heading-white background-image: url(images/generators.jpeg)

Generators

class: bottom, right, heading-white background-image: url(images/state-based.jpeg)

State-based Test

class: middle, center

https://en.wikipedia.org/wiki/QuickCheck

31 Languages

???

  • Call to arms
  • Can introduce gradually
    • Fuzzing
  • Not a framework
  • One test at a time
  • Property testing is not specific to one language
  • It's an idea
  • I would write my own if I had to

Java

https://github.com/pholser/junit-quickcheck

@RunWith(JUnitQuickcheck.class)
public class StringProperties {

  @Property
  public void testSubstring(String s1, String s2) {
    assertEquals((s1 + s2).substring(s1.length), s2);
  }
}

Javascript

https://github.com/jsverify/jsverify

jsc.property("substring", jsc.string, jsc.string,
  function (s1, s2) {
    return (s1 + s2).substring(s1.length) === s2;
  }
);

Python

https://hypothesis.readthedocs.io/

@given(text(), text())
def test_substring(s1, s2):
  assert substring(s1 + s2, len(s1)) == s2

???

  • Hypothesis is particularly good
    • Good community

class: bottom, right, heading-white background-image: url(images/edges.jpeg)

Edge Cases

???

  • Basically the same across different langauges
  • No longer feels like you're writing tests for test sake
  • Enjoyable tot think about your system invariants
  • Live longer than other tests
  • File multiple bugs with one test
  • Less code

class: bottom, left, heading-white background-image: url(images/property-based.jpeg)

Find More Bugs

class: bottom, left, heading-white background-image: url(images/property-based.jpeg)

Find More Bugs with Less Effort

???

  • Find better bugs

Links