Skip to content

In memory compilation "benchmark"

Tamás Koczka edited this page Jan 30, 2018 · 1 revision

This is not really a good benchmark of the in-memory compilers, but at least it gives a glance into where it helps.

Testing method: I used Ubuntu 16.04 WSL on my Windows 10 machine.

  • for in-memory compilation measurements I ran ./compiler_backend.py, opened http://127.0.0.1:8000/ with the HelloWorld input and modified one of the constant values 7 times. I threw out the first two results (warmup) and took the average of the other 5 measurements
  • for normal compilation measurements I ran ./compiler_backend.py --noInMemoryCompilation instead, everything else was the same as previously

There are a lot of problems with this of course, for example the browser only opens 6 TCP connections at once, but this does not count into the compilation time actually, and of course running 6 compilers at the same time wants to use more than 100% CPU, so they will slow down each other.

Also good to know that starting a new process in WSL is really slow. So the speed up would be much less in a native environment...

language normal
compilation
in-memory
compilation
speed up
Perl 749 ms (*) 610 ms 1.22x
Swift 4799 ms (*) 3857 ms 1.25x
C++ 7793 ms (*) 4669 ms 1.7x
Go 4929 ms (*) 1957 ms 2.5x
C# 6002 ms 273 ms 22x
Ruby 769 ms 33 ms 23x
Java 7342 ms 284 ms 26x
JavaScript 1535 ms 38 ms 40x
TypeScript 7843 ms 176 ms 44x
Python 844 ms 15 ms 56x
PHP 1991 ms 19 ms 105x

(*) Does not support in-memory compilation actually, but still runs faster because other compilers require less resources (eg. CPU), so it has more.