Test framework
Botcraft has two types of tests: simple and "online" tests. Simple tests are very similar to what you can find on other C++ projects using Catch2. It's however not really possible to check whether an entity is correctly loaded or a block is correctly mined with these kinds of tests. Hence the development of a custom test framework extending Catch2 to be able to easily perform tests on a Minecraft server.
The online test framework is highly inspired by the one used internally at Mojang (see this video presentation by Henrik Kniberg for more info on it). The goals behind its design were:
- easy to setup new tests
- working for all supported Minecraft versions
- fully automatic process
- controlled and reproducible test environment
- easy to spot/diagnose/iterate on failed tests
The result implementation is built around Catch2 and subprocess.h to have a Minecraft vanilla server that can be used as a test environment. The server is automatically started on default port (25565) on startup and stopped at the end (⚠ killing the test process before it ends will likely result in a ghost minecraft server java process running in the background, make sure you kill it too). It means that while the tests are running, you can connect to the server with your vanilla Minecraft client and see the tests being performed, which is pretty cool.
Every time the test executable is called, the server is initialized from scratch. It means that previous test runs won't interfere with current ones. It also means that we can't run multiple instances of the test executable to run the tests in parallel (as it would spawn multiple servers). This is why all the tests are registered as only one in CTest. Otherwise CTest would launch each test on different processes, requiring to reinitialize the server everytime, wasting a lot of time between tests.
Server setup includes jar libraries extraction (for recent minecraft versions), world generation and server.properties and gamerules setup. Server cleanup makes sure all files are properly saved on disk before shutting down (which means you can restart the server after the tests if you want to check the state of a specific test).
For each test and subtest, a teleport sign is created near spawn. Right clicking the sign will teleport you directly to the test. Sign text is set to filename on first line, then test/section names.
Test teleport signs
A test is composed of two things:
- C++ code with the logic of the test
- a .nbt structure file that is loaded into the world for this test (they can be found here in the repo)
The nbt filename must match the name given in the C++ code (spaces are replaced by underscores in the filename) for a structure to be properly loaded during test run. Tests with no matching file will use a default 3x3x3 empty structure (this can be useful when testing behaviour like chat that doesn't need any specific block or entities to interact with).
Before each test and subtest structure, a beacon is added, indicating the status of the test: green for passing, red for failing, orange for expected failing (test used to track a known bug for example) and yellow for currently running.
The four possible test status indicator
When a test fails, a book is created with a summary of what went wrong (as Minecraft book pages are pretty small, it might be truncated though, see test console output for full details). For versions <1.14, books are in an item frame instead of a lectern. As you are in creative mode, you can't pop it out of the item frame. You have to "Pick Block click" (mouse wheel usually) it to copy it to your inventory.
Example failure book content
As the test framework is built on top of Catch2, it is possible to use many command line arguments to alter the behaviour of the test executable when launched. Here are the most useful ones for Botcraft. They can be combined.
botcraft_online_tests --wait-for-keypress exitbotcraft_online_tests -v highbotcraft_online_tests "test name"botcraft_online_tests -# [#filename]botcraft_online_tests [.physics]