What is the best Kotlin assertion library?
The purpose of this repository is to serve as a case study for the various Kotlin assertion libraries we have available today. A comparison was conducted based on the following criteria:
- Popularity: Based on GitHub stars and time of creation.
- Documentation: Quality of material found on how to use and how to customize the library.
- Performance: Time added to execute an unit-test assertion compared to other libraries.
- Fluency / Kotlin idiomatic: Readability and Kotlin-like style.
- Customizability: If it could be customized easily.
- Kotlin Multi Platform: If it could be used in KMM projects.
Some of the criteria in this report can be biased on my personal subjectiveness and understanding. The results reflect my own opinion and analysis.
- Match
assertion(expected, actual)
- Wrap
assert(actual).assertion(expected)
- Extend
actual.assert(expected)
oractual assert expected
- JUnit
- Kotlin Junit
- Hamcrest
- Hamkrest
- Google Truth
- AssertJ
- AssertK
- Atrium
- Expekt
- Kluent
- Kotest assertion
- Strikt
- Truthish
All libraries were tested under a set of scenarios. You can find all of them in this codebase:
- Scenario 1: Perform assertion on a simple integer sum.
- Scenario 2: Perform a complex list assertion validating instance type, value and order.
- Scenario 3: Perform assertion on a ViewState element simulating an Android ViewModel with coroutines.
If you wish to run the microbenchmark, with an emulator running use the following command:
./gradlew assertionbenchmark:connectedCheck --info
If you wish just to run the unit tests, use the following command:
./gradlew benchmarkable:testDebugUnitTest
Although you can run the tests on the terminal, the results are best seen when running in the IDE.
Performance was measured using the Android Micro Benchmark library. For each scenario, the test configuration and action are the same, the only thing that changes is the assertion.
Lib | test 1 (sum) |
---|---|
JUnit | 2.9 ns |
Kotlin JUnit | 2.9 ns |
Truthish | 14.8 ns |
Hamcrest | 15.5 ns |
Hamkrest | 18.5 ns |
Kotest | 22.6 ns |
AssertJ | 96.2 ns |
Expekt | 100 ns |
AssertK | 115 ns |
Google Truth | 124 ns |
Kluent | 150 ns |
Atrium | 390 ns |
Strikt | 1,250 ns |
Lib | Test 1 (remove) | Test 2 (favorite only) | test 3 (no favorite) | test 4 (favorite and non-favorite) |
---|---|---|---|---|
JUnit | 988 ns | 852 ns | 770 ns | 1,113 ns |
Kotlin JUnit | 1,142 ns | 1,011 ns | 933 ns | 1,348 ns |
Truthish | 1,462 ns | 1,486 ns | 1,331 ns | 2,482 ns |
AssertK | 1,658 ns | 1,533 ns | 1,050 ns | 1,919 ns |
Google Truth | 2,382 ns | 2,371 ns | 1,081 ns | 2,428 ns |
Expekt | 3,507 ns | 3,361 ns | 1,342 ns | 4,725 ns |
Atrium | 1,393 ns | 24,086 ns | 7,712 ns | 34,373 ns |
AssertJ | 11,545 ns | 11,273 ns | 6,333 ns | 15,429 |
Kluent | 18,990 ns | 17,689 ns | 1,551 ns | 27,798 ns |
Kotest | 37,103 ns | 32,497 ns | 36,240 ns | 49,885 ns |
Strikt | 47,405 ns | 47,209 ns | 45,645 ns | 49,009 ns |
Hamkrest | 3,783,240 ns | 2,385,701 ns | 2,297,980 ns | 3,294,005 ns |
Hamcrest | N/A | N/A | N/A | N/A |
Note: Had troubles setting up latest hamcrest version on instrumented tests as it has conflicts with some androidx test dependencies. Using the contains() method didn't work and trying to resolve the conflicts didn't work either
Lib | Test 1 (success) | Test 2 (error) |
---|---|---|
JUnit | 157 ns | 23 ns |
Hamcrest | 179 ns | 61.8 ns |
Hamkrest | 191 ns | 79.6 ns |
AssertK | 224 ns | 98.3 ns |
Kotlin JUnit | 235 ns | 108 ns |
Google Truth | 430 ns | 247 ns |
Kotest | 479 ns | 326 ns |
Truthish | 551 ns | 214 ns |
Strikt | 551 ns | 452 ns |
Expekt | 973 ns | 685 ns |
AssertJ | 271 ns | 2,903 ns |
Kluent | 1,555 ns | 1,188 ns |
Atrium | 4,432 ns | 1,846 ns |
Note: Had trouble setting a benchmark that launches a coroutine (runTest). Did a slightly different setup here just benchmarking the assertion and not the whole test
Lib | Popularity | Documentation | Performance | Fluency | Customizability | KMM |
---|---|---|---|---|---|---|
#1 Kotlin JUnit 🏆 | * |
✅ | ✅ | ❌ | ❌ | ✅ |
#2 JUnit 🏆 | * |
✅ | ✅ | ❌ | ❌ | ❌ |
#3 Hamcrest 🏅 | ✅ | ✅️ | ❌ | ✅ | ❌ | |
#4 Hamkrest 🏅 | ❌ | ✅ | ✅ | ❌ |
Lib | Popularity | Documentation | Performance | Fluency | Customizability | KMM |
---|---|---|---|---|---|---|
#1 Kotest 🏆 | * |
✅ | ✅ | ✅ | ✅ | |
#2 Kluent 🏆 | ❌ | ✅ | ✅ | ✅ | ||
#3 Expekt 🏅 | ❌ | ❌ | ❌ | ❌ |
*
This lib is part of a bigger library
✅ Consistent positive result.
❌ Consistent negative result (in my opinion and analysis).
🏆 Great Lib.
🏅 Good but use it with caution.