Home
Welcome to the wiki page for this benchmark test, designed to evaluate and compare the performance of various scripting languages in a simulated game environment. This project is not a conventional video game but mirrors its fundamental structure, complete with an update loop, game logic, and rendering processes.
At the heart of this experiment is a 2D top-down simulation that features the savannah - an ecosystem comprising multiple giraffes, a lion, obstacles like trees and a lake, and elements of survival and predator-prey interactions. The animals in this virtual world exhibit steering behaviors governed by a simple physics solver.
Giraffes will attempt to move towards the food. The lion starts the simulation asleep, but will soon wake up - chase the closest giraffe until it either catches it, or he runs out of energy, after which it will go back to sleep for a bit. Giraffes will try to flee for the lion, if it comes too close.
Animals attempt to move around obstacles - there are trees, and a big lake. Sometimes, their steering behaviors take them across the lake - but that slows them down significantly.
The core of this benchmark test lies in its implementation across various scripting languages, chosen for their similarities in features and design principles. These languages include:
- Lua 5.1
- LuaJIT
- LuaU
- Angelscript
Each of these languages share a lot of common features - garbage-collection, the easy of embedding and interacting safely with the host application, emphasizing data security over direct memory access.
A reference implementation in C++ was the start - it uses a simple OpenGL based "game framework". Each sprite is a quad - rendering isn't the slow part. The benchmark focuses on a deliberately inefficient algorithm to magnify performance differences between the scripting languages. This is exemplified by the quadratic complexity in the separation behavior, where each giraffe calculates its distance from every other giraffe, significantly slowing down the simulation. The implementation is mostly in one file: game_state_playing.cpp
I wrote Lua interface functions to expose the same function used by the C++ implementation to a Lua environment: if_game.cpp
Lua 5.1, LuaU, and LuaJIT all use mostly the same functions here - there are some #if defined(HAS_LUAX) branches for specific features and functions I want to test in the different flavors of Lua.
I noticed that for LuaJIT, enabling just-in-time compilation really didn't have a large performance impact in this implementation. If anything, it made performance slightly worse. I created a variant of main.lua where I wrote a short vector library in Lua instead. Without JIT it was far worse - but wit it, it was much more performant. But more on the results and findings later.
I wrote a similar interface for Angelscript: angelscript_game.cpp
The performance of each script was rigorously tested in a controlled environment (I just ran them all on my local machine and took notes).
I compiled the application in a Release configuration - full optimizations and inlining settings, spawned 1,000 giraffes in the simulation. I ran these tests through the Superluminal profiler and noted down how long the update() function ran for. Due to the quadratic complexity of the agent behaviors - it got quite slow with 1,000 giraffes.
These are the results!
Now keep in mind that this example is bad on purpose - there are many, many ways to improve performance here, like space partitioning, clustering, multithreading, reusing pools of vector instances, etc. It can be difficult to benchmark "real" code without making up a contrived example. I didn't want to benchmark an algorithm that runs in 2,4 ms in Lua 5.1 and 2,02 ms in LuaJIT, that doesn't magnify the scale enough to draw any conclusions from.
The idea with this example is to benchmark something that's fairly representative in how we program games using Stingray and Lua. We build engine-side functions, exposed to a Lua VM environment, there's a lot of chatter across the Lua stack, which leads to poor performance.
| Language | Update time in ms. |
|---|---|
| Lua 5.1 | 572 ms. |
| Angelscript | 419 ms. |
| LuaJIT | 328 ms. |
| LuaU | 306 ms. |
LuaJIT with jit.on() using Lua vector math |
26 ms. |
| C++ | 1,5 ms. |
I knew that C++ would be fastest, of course. I knew that Lua 5.1 would be slowest. I - however, wasn't prepared for how much slower Lua runs in this example, however contrived it may be. Lua doesn't have a marketing department - but it's still marketed as "the fasted embedded, interpreted script language." It's certainly a phrase I've retold. And it's easy to add "And LuaJIT, in particular, is on par with native C!" I mean why wouldn't it be? It just-in-time compiles Lua to native bytecode and if you benchmark a LuaJIT Fibonacci algorithm, you'll end up in the same ball park as C. But that's wholly divested from real, actual code.
I got some help from Henrik to check out the Angelscript version of the code - he replaced the vector math functions there with a native implementation, but since Angelscript doesn't come with JIT out of the box - it ran slower.
The biggest times spent on this application was writing the Lua interfaces, I kid you not. Let's compare lines of code.
The C++ implementation in game_state_playing.cpp weighs in at 476 lines of code.
The file that exposes all of the native rendering and vector math functions to Lua is if_game.cpp. It sets up all of the binding and types as well - it's at 1,546 lines of code.
The Lua implementation in main.lua is 582 lines of code. But - obviously, it relies on the much larger interface functions to talk to the game engine.
As for Angelscript, the angelscript_game.cpp file is 526 lines long. And the script.as file which contains the implementation of the game is 379 lines long, but I'm cheating a little bit there. I didn't finish the implementation - it doesn't handle inputs like keyboard commands, nor does it render all of the ImGui overlays that the Lua and C++ implementation does.
I initially used a library called sol2 to dynamically generate Lua bindings for structs and functions. This was pretty straightforward but since it uses a lot of generics, compile time really shot through the roof. if_game.cpp took as long to compile as the entire rest of the project, including all of Lua. And in the runtime, it also added a lot of function call and reflection overhead. Slowly implementing your interfaces by hand is the way to go...
This is the implementation using sol2: if_game.cpp
I attempted to optimize the LuaJIT implementation by using its FFI features. Instead of using the Lua stack for the vector library, I defined the GLM vector library functions and structs using LuaJIT's cdef annotation. This didn't really improve performance - in fact, it was slightly worse, and decidedly less safe.
Allocations and reallocations are slow, when talking back and forth using the stack, there's a lot of that. I created an allocation function that pre-allocated a 1 GB chunk of memory, that we just let grow until it crashed. It didn't really improve performance significantly. The stack is still a very flow - yet safe, way of interfacing with the game.
There's one avenue of experimentation I have only touched on - script languages that either transpile to native C, or compiles to statically linked libraries that you link with your program. You're losing the ability to runtime-reload script, but you gain the advantage of a more "user friendly" programming language, but you don't lose performance to foreign function invocation which - in my conclusion, always adds overhead.
There are quite a few different languages that match this description: D, Zig, Rust, Embeddable Common Lisp, for instance.
The main branch has a sloppy Zig implementation. Any language that interfaces with a C ABI is tricky to get working in a C++ application. I had to copy-paste a bunch of struct definitions, and implement a few glue functions that memcpy vectors in a very unsafe manner. But it's working at least - and it's very fast.
The slowest Lua implementation too 572 ms for one update, whereas the fastest one (albeit kind of cheating) took 26 ms. The references implementation in C++ was 1.5 ms - and Zig clocked in at 3.6 ms. I think that's a winner, in terms of speed. And the reason for this, of course, is that we compile Zig to a static library, and it calls extern functions using C ABI calling conventions.
The line of code count for the Zig implementation is 399 lines of code - but it's kind of cheating. It doesn't handle input or drawing using ImGui. The glue-code includes the C header that re-defines all structs and functions. They also include the functions that map vector functions between vector structs used by Zig and the glm vector library. These files weigh in at 262 lines of code. This is a big improvement over the Lua glue code.
Another added benefit is that you get symbols that your debugger and profiling tools understand out of the box - no further configuration was required to step through Zig code using my standard C/C++ debugger.
