In 2020, I will be participating in the Advent of Code using Kotlin multi-platform code only. This means I can run my solutions on the JVM, on Node JS and natively on Windows. I did the same in 2019 and 2018. Each year the Kotlin platform has evolved, so it's interesting to see how the relative performance of these build targets evolve. Just to spice things up, this year I'll be using GraalVM to run the JVM versions, and perhaps also the Node implementation of GraalVM if I find the time.
I will be measuring the performance of each platform by measuring the elapsed time calculating the solution N
times:
val times = (1..repeat).map {
measureNanos {
Puzzles.run(day, part)
}
}
where repeat
is the number of desired repetitions, and day
and part
are used to select the proper challenge.
Note that the measureNanos
function is the first example of functionality that isn't available in common code, since measuring time is platform-specific. The function is implemented using Kotlin's expext/actual
mechanism.
Runner.kt
defines its expectation of a function measureNanos
for each supported platform with the signature:
expect inline fun measureNanos(block: () -> Unit): Long
Each platform provides an actual implementation in files called Actuals.kt
. On the JVM and native, we can use measureNanoTime
as provided by the Kotlin standard library on these platforms:
JVM and Native:
actual inline fun measureNanos(block: () -> Unit) = measureNanoTime(block)
On Node, we need to write our own implementation:
actual inline fun measureNanos(block: () -> Unit): Long {
val start = process.hrtime()
block()
val end = process.hrtime(start)
val nanos = (end[0] * 1e9 + end[1]) as Double
return nanos.roundToLong()
}
Usually, I try to finish a puzzle as quickly as possible, then spend some time to cleanup and optimise the code. If and when these optimizations alter the performance, I'll update the measurements below to reflect the new solution.
There is a very explicit warning in the FAQ of Kotlin/Native that says the current version of Kotlin/Native is not suited for performance analysis, as it has not been optimised in any way for performance and benchmarking.
Apart from that, all measurements were taken on my laptop, under non-controlled circumstances using non-optimized code and tools. So, if you use the results below for anything important, you're insane...
Today I had a lot of trouble getting everything compiling and running. First, somehow my IntelliJ run configuration for the JVM had somehow been corrupted which took me ages to discover, and then there was an issue with the native runner.
Platform | Average (ms) | Measurements (ms) |
---|---|---|
GraalVM | 21 ± 20 | 99, 30, 25, 41, 20, 31, 16, 10, 15, 10, 15, 11, 11, 11, 12, 12, 12, 11, 11, 13 |
Node JS | 14.5 ± 14.8 | 70, 35, 31, 12, 9, 8, 9, 8, 7, 11, 9, 9, 8, 8, 8, 9, 8, 7, 7, 8 |
Native | 147 ± 9.7 | 158, 143, 132, 141, 143, 142, 141, 152, 171, 143, 142, 146, 144, 144, 137, 149, 150, 137, 150, 170 |
Platform | Average (ms) | Measurements (ms) |
---|---|---|
GraalVM | 9.1 ± 14.7 | 71, 16, 11, 7, 7, 6, 5, 5, 5, 8, 4, 5, 2, 3, 3, 4, 2, 2, 4, 4 |
Node JS | 17.4 ± 18.6 | 90, 43, 25, 20, 12, 10, 8, 8, 9, 8, 9, 8, 8, 8, 19, 11, 11, 11, 9, 11 |
Native | 39.1 ± 7.5 | 36, 36, 37, 34, 34, 36, 34, 45, 41, 68, 34, 42, 36, 35, 42, 34, 38, 35, 41, 34 |
Platform | Average (ms) | Measurements (ms) |
---|---|---|
GraalVM | 5.7 ± 11.1 | 53, 7, 4, 4, 2, 5, 5, 4, 3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1 |
Node JS | 13.1 ± 17.0 | 73, 41, 34, 11, 7, 9, 6, 10, 6, 7, 6, 5, 6, 8, 5, 3, 3, 4, 4, 4 |
Native | 25.2 ± 6.4 | 20, 20, 29, 31, 22, 17, 36, 17, 30, 26, 20, 18, 31, 22, 33, 27, 25, 18, 36, 17 |
Platform | Average (ms) | Measurements (ms) |
---|---|---|
GraalVM | 11.3 ± 14,5 | 70, 21, 16, 17, 10, 10, 8, 11, 6, 6, 5, 5, 4, 4, 6, 5, 3, 4, 3, 2 |
Node JS | 14.8 ± 13.5 | 65, 26, 35, 17, 8, 8, 9, 7, 7, 8, 8, 8, 12, 8, 8, 12, 10, 12, 10, 8 |
Native | 56.2 ± 6.0 | 53, 69, 51, 51, 47, 60, 59, 47, 56, 49, 52, 56, 53, 57, 62, 65, 56, 48, 64, 57 |
Platform | Average (ms) | Measurements (ms) |
---|---|---|
GraalVM | 3.9 ± 6.0 | 29, 5, 2, 2, 2, 2, 4, 3, 2, 2, 2, 1, 1, 1, 2, 1, 1, 1, 5, 1 |
Node JS | 8.8 ± 8.1 | 35, 24, 13, 17, 8, 9, 4, 4, 4, 5, 3, 5, 4, 5, 5, 3, 4, 3, 4, 5 |
Native | 18.9 ± 2.6 | 18, 18, 16, 19, 18, 18, 27, 18, 18, 16, 18, 16, 19, 17, 22, 16, 18, 17, 22, 17 |
Platform | Average (ms) | Measurements (ms) |
---|---|---|
GraalVM | 7.4 ± 10.7 | 52, 10, 11, 12, 7, 5, 8, 6, 5, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2 |
Node JS | 19.4 ± 21. 0 | 108, 29, 20, 22, 13, 11, 11, 11, 14, 13, 15, 16, 13, 12, 11, 15, 9, 12, 10, 14 |
Native | 58.5 ± 5.0 | 64, 61, 63, 55, 58, 56, 65, 54, 56, 58, 72, 62, 54, 52, 54, 55, 59, 52, 52, 59 |
Platform | Average (ms) | Measurements (ms) |
---|---|---|
GraalVM | 84 ± 71 | 390, 83, 68, 66, 62, 63, 61, 60, 73, 85, 71, 77, 62, 63, 73, 65, 62, 61, 60, 60 |
Node JS | 1679 ± 142 | 1723, 1726, 1628, 1657, 1517, 1843, 1748, 1988, 2061, 1620, 1614, 1730, 1646, 1606, 1686, 1577, 1567, 1555, 1506, 1568 |
Native | 4721 ± 383 | 4417, 4235, 4648, 4426, 4581, 5086, 4806, 5037, 4304, 5537, 4693, 5095, 4679, 5624, 4955, 4457, 4256, 4552, 4506, 4508 |
Platform | Average (ms) | Measurements (ms) |
---|---|---|
GraalVM | 8.7 ± 9.5 | 47, 15, 16, 7, 5, 5, 3, 4, 5, 6, 5, 5, 7, 5, 5, 5, 5, 6, 6, 4 |
Node JS | 23.3 ± 14.1 | 81, 37, 20, 22, 24, 17, 20, 17, 18, 23, 15, 18, 17, 18, 15, 16, 16, 20, 19, 23 |
Native | 124 ± 17 | 106, 134, 141, 112, 136, 105, 139, 106, 134, 107, 153, 108, 128, 135, 117, 119, 154, 90, 118, 124 |
Platform | Average (ms) | Measurements (ms) |
---|---|---|
GraalVM | 32.6 ± 36.3 | 183, 68, 41, 25, 18, 19, 21, 19, 26, 20, 24, 18, 24, 20, 18, 20, 19, 18, 26, 17 |
Node JS | 32.5 ± 22.5 | 118, 57, 59, 33, 32, 22, 24, 29, 21, 20, 20, 21, 21, 20, 24, 24, 21, 25, 21, 26 |
Native | 105 ± 3.7 | 103, 106, 106, 102, 109, 108, 100, 100, 105, 108, 100, 106, 114, 109, 107, 105, 106, 100, 107, 101 |
Platform | Average (µs) | Measurements (µs) |
---|---|---|
GraalVM | 2483 ± 5719 | 27370, 1606, 1629, 1394, 1368, 1281, 1475, 1500, 1343, 1276, 1303, 1243, 1274, 1177, 708, 489, 588, 1031, 831, 760 |
Node JS | 6837 ± 8230 | 35319, 12176, 6247, 5928, 6658, 6698, 2986, 3525, 24430, 2477, 3497, 3641, 3060, 2472, 2099, 1761, 1379, 5845, 3452, 3080 |
Native | 2432 ± 669 | 2581, 2194, 2174, 3230, 2548, 2318, 2854, 1781, 1830, 1770, 3586, 1858, 2007, 3387, 2382, 2218, 4189, 1826, 1841, 2067 |