Timings comparaison with Play json

[crakjie]

I had made few test to compare performances of play json and circe.
First I was chock by a huge differences between thems 8ms for circe and just 0.1ms for play.
( theses test are made in a scala console. This may have an incidence )

Here a gist of the test.
you can copy past this in console (with the right dependency ) and run Test.run

The first time it's run I see a huge advantage for play but the second run is less significant because circe won.

when I past this code in the console every time I run it I see theses results

val v = IP(Some("1"),"2")

for(i <- 1 to 3) { Test.time(v.asJson) }
 elapsed time :8.349507ms
 elapsed time :0.037627ms
 elapsed time :0.034833ms

but with play I have this

for (i <- 1 to 3 ) { Test.time(play.api.libs.json.Json.toJson(v)) }
 elapsed time :0.104637ms
 elapsed time :0.073606ms
 elapsed time :0.071487ms

Because there is a long call every time the for is invoked I wonder if there is not a performances issues.

ps : the time become significant if you have a lot of field

scala> case class Loonng(un: Int, deux: Int, trois : Int, quatre : Int, cinq: Int, six: Int, sept: Int, huit: Int, neuf: Int, dix :Int, onze : Int, douze: Int, triez :Int, quatroze: Int, quinze: Int, saize : Int, dixsept: Int, dixhuit: Int, dixneuf: Int, vinght: Int, vinghtetun : Int, vinghtdeux: Int, vinghttrois: Int)
defined class Loonng

scala> Loonng(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23)
res11: Loonng = Loonng(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23)

scala> for(i <- 1 to 3) { Test.time(res11.asJson) }
 elapsed time :57.142316ms
 elapsed time :0.311121ms
 elapsed time :0.285936ms

[ktoso]

For writing benchmarks please consider using sbt-jmh (it's wrapping OpenJDK's JMH) as it will give more accurate results than running something in a loop or a few times in the repl – there's all kinds of warm-up method JIT-ing and other effects in play here (no pun intended 😉) so it's hard to measure properly without the right infrastructure (i.e. what JMH does around benchmark code).

I'd be very interested in comparing the various libs, but with the right tools :-)
I'm expecting circe to be faster due to it's nature, still curious how much faster.

[travisbrown]

Thanks, @ktoso. We're very happily using your sbt-jmh for circe and Finch's benchmarks, and I'll close this issue when I've added Play JSON to the circe benchmarks (which currently only compare against Argonaut).

(And right, with a little warmup circe looks faster with @crakjie's code.)

[crakjie]

Why play json don't have this "problem" ?

[ktoso]

Great, thanks @travisbrown :-)

(I've got this knee-jerk reaction when I see hand-rolled benchmarks to recommend proper tools, sorry for the noise 😉)

[travisbrown]

@crakjie There's so much stuff happening in any JVM implementation that you just can't meaningfully compare the performance of operations like this without warmup, etc.

For example, even just running the Play version first pretty much evens things out:

scala> for (i <- 1 to 3 ) { Test.time(play.api.libs.json.Json.toJson(v)) }
 elapsed time :57.044ms
 elapsed time :0.113ms
 elapsed time :0.089ms

scala> for(i <- 1 to 3) { Test.time(v.asJson) }
 elapsed time :61.344ms
 elapsed time :0.067ms
 elapsed time :0.039ms

But this still doesn't really tell you anything.