diff --git a/docs/Splashkit/Unit Testing/Index.md b/docs/Splashkit/Unit Testing/Index.md new file mode 100644 index 000000000..036917d6a --- /dev/null +++ b/docs/Splashkit/Unit Testing/Index.md @@ -0,0 +1,17 @@ +# Overview + +It's important that all aspects of SplashKit are thoroughly tested. This is especially important +because SplashKit is designed to be used by beginners - they should be able to focus on learning to +develop software and fix bugs in their own code rather than wondering why some aspect of the +framework isn't functioning as expected. Unit tests allow us to efficiently test and validate the +core functionality of SplashKit. + +# Goals + +Currently, unit testing isn't given much of a spotlight in documentation, and developing and running +tests is more cumbersome than it could be. So, goals for unit testing are: + +- Research ways of improving unit testing workflow, including integration into Visual Studio Code, + which is the IDE recommended in SplashKit documentation +- Create documentation for unit testing, including an onboarding guide for those wanting to write + and run unit tests diff --git a/docs/Splashkit/Unit Testing/research/build-run-workflow.md b/docs/Splashkit/Unit Testing/research/build-run-workflow.md new file mode 100644 index 000000000..1650980b5 --- /dev/null +++ b/docs/Splashkit/Unit Testing/research/build-run-workflow.md @@ -0,0 +1,173 @@ +# Current workflow + +As can be seen, the current workflow doesn't require a lot of set up but involves using the command +line (including debugging). + +## Linux + +1. Install prerequisites + + - Update + + ```shell + sudo apt-get update + sudo apt-get upgrade -y + ``` + + - Install + + ```shell + sudo apt-get install -y \ + git build-essential cmake g++ libpng-dev libcurl4-openssl-dev libsdl2-dev \ + libsdl2-mixer-dev libsdl2-gfx-dev libsdl2-image-dev libsdl2-net-dev libsdl2-ttf-dev \ + libmikmod-dev libbz2-dev libflac-dev libvorbis-dev libwebp-dev libfreetype6-dev + ``` + +2. Build test project + + ```shell + cd projects/cmake + cmake . + make + ``` + +3. Run unit tests + + - Running all tests + + ```shell + cd ../../bin + ./skunit_tests + ``` + + Example output: + + ``` + (14/04/2025) ERROR -> Invalid binary string passed to bin_to_oct: 2, returning empty string [raised in basics.cpp:340] + (14/04/2025) ERROR -> Invalid hexadecimal string passed to hex_to_oct: G, returning empty string [raised in basics.cpp:371] + (14/04/2025) ERROR -> Invalid octal string passed to hex_to_dec: G, returning 0 [raised in basics.cpp:305] + (14/04/2025) ERROR -> Invalid octal string passed to oct_to_hex: 8, returning empty string [raised in basics.cpp:383] + =============================================================================== + All tests passed (1258 assertions in 74 test cases) + ``` + + - Running a specific test + + ```shell + ./skunit_tests [test name or tag] + ``` + + Example: + + ``` + ./skunit_tests "[least_common_multiple]" + Filters: [least_common_multiple] + =============================================================================== + All tests passed (7 assertions in 1 test case) + ``` + +# New workflow + +The details depend on which VS Code extension is used, but this example using the CMake Tools +extension shows that the process can be much smoother. This also offers features like test listing +and debugging in VS Code, albeit with extra setup. + +## Linux + +1. Install prerequisites + + - Update + + ```shell + sudo apt-get update + sudo apt-get upgrade -y + ``` + + - Install packages + + ```shell + sudo apt-get install -y \ + git build-essential cmake g++ libpng-dev libcurl4-openssl-dev libsdl2-dev \ + libsdl2-mixer-dev libsdl2-gfx-dev libsdl2-image-dev libsdl2-net-dev libsdl2-ttf-dev \ + libmikmod-dev libbz2-dev libflac-dev libvorbis-dev libwebp-dev libfreetype6-dev + ``` + + - Install CMake Tools from VS Code's extension browser or from + https://marketplace.visualstudio.com/items?itemName=ms-vscode.cmake-tools + +2. Configure extension + + - Select ${workspaceFolder}/projects/cmake/CMakeLists.txt + + - Select the Linux configure preset + + - On the CMake Tools extension tab set the Build target to skunit_tests + + - Set the Debug and Launch target to skunit_tests + +3. Configure VS Code debugging + + - Go to the Run and Debug tab of VS Code and create a launch.json file + + ```json + { + "version": "0.2.0", + "configurations": [ + { + "name": "(gdb) Launch", + "type": "cppdbg", + "request": "launch", + "program": "${command:cmake.launchTargetPath}", + "args": [], + "stopAtEntry": false, + "cwd": "${workspaceFolder}", + "environment": [ + { + "name": "PATH", + "value": "${env:PATH}:${command:cmake.getLaunchTargetDirectory}" + } + ], + "externalConsole": false, + "MIMode": "gdb", + "setupCommands": [ + { + "description": "Enable pretty-printing for gdb", + "text": "-enable-pretty-printing", + "ignoreFailures": true + }, + { + "description": "Set Disassembly Flavor to Intel", + "text": "-gdb-set disassembly-flavor intel", + "ignoreFailures": true + } + ] + } + ] + } + ``` + +4. Build test project + + Building can be done from the terminal, as before, or within VS Code in one of the following ways + + - In the CMake Tools extension, select Build + + - On the Testing tab, select Refresh Tests + +5. Run unit tests + + In addition to running tests from the terminal, tests will show up on the Testing tab of VS Code. + + - Running all tests + + Select Run Tests, each test will be run and the status of each can be seen in the test list. + + - Running a specific test + + Select Run Test next to any test on the test list to run it + +6. Run unit tests with debugging + + Break points can be set in VS Code like in normal programs, by selecting the red next to a line + in a test or pressing F9. Then, select Debug Tests to run all tests with debugging or Debug Test + next to the test name to run a specific test with debugging. diff --git a/docs/Splashkit/Unit Testing/research/code-coverage.md b/docs/Splashkit/Unit Testing/research/code-coverage.md new file mode 100644 index 000000000..9f0a89b9e --- /dev/null +++ b/docs/Splashkit/Unit Testing/research/code-coverage.md @@ -0,0 +1,143 @@ +# Code Coverage + +Collecting code coverage would help us check whether the codebase is adequately tested by unit +tests. Ideally, coverage would be integrated into the unit testing workflow and able to be done in +Visual Studio Code. Some research into code coverage has been done and so far it seems feasible to +do on Linux and Windows with WSL, but problems were encountered doing so on Windows with MSYS2. +Collecting coverage on Mac hasn't been tried. + +## Collecting coverage on Windows + +- Coverage collection when using WSL works fine. +- The coverage collection process failed on MSYS2 when using Mingw64, which is recommended by the + SplashKit installation guide. +- Using other MSYS2 environments, like UCRT64 or MSYS wasn't tried. + +## Code coverage on Linux/WSL + +Code coverage on Linux involves setting options at compile time to enable coverage generation, +running the unit tests to generate the data, then displaying it (either as a summary of coverage +over the source files, or highlighting lines that have been executed in the IDE). + +1. Setting compiler and linker options Can be done as part of the build process with CMake by adding + the options to CMakeLists.txt Assuming the use of g++: + + ``` + add_compile_options(--coverage) + add_link_options(--coverage) + ``` + + This can be integrated with CMake configure presets to provide an easy way to switch between + building with and without coverage: + + ``` + option(USE_COVERAGE "Enable GCOV during the build" OFF) + if(USE_COVERAGE) + add_compile_options(--coverage) + add_link_options(--coverage) + endif() + ``` + + See more on CMake presets: https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html See + more on gcc options, including --coverage: + https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html + +2. Collect coverage Coverage on Linux can be collected with LCOV or gcovr + +- https://github.com/linux-test-project/lcov +- https://github.com/gcovr/gcovr + +Example of integrating LCOV into the build process: + +``` +# Coverage collection with LCOV +# Get absolute path to SK_SRC so we can pass it to LCOV +# and collect coverage only for src files +get_filename_component(SRC_ABS ${SK_SRC} ABSOLUTE) +message(SRC_ABS="${SRC_ABS}") + +find_program(LCOV lcov REQUIRED) +set(LCOV_BASE lcov-base.info) +set(LCOV_TEST lcov-test.info) +set(LCOV_TOTAL lcov.info) +set(LCOV_LOG lcov.log) +set(LCOV_ERR lcov.err) +add_custom_target(init-coverage + COMMENT "Collecting initial coverage" + COMMAND lcov -c -i -d ${CMAKE_CURRENT_BINARY_DIR} --include "'${SRC_ABS}*'" + -o ${LCOV_BASE} 2>${LCOV_ERR} >${LCOV_LOG}) +add_dependencies(init-coverage reset-coverage) + +add_custom_target(reset-coverage + COMMENT "Reset all coverage counters to zero" + COMMAND lcov -q -z -d ${CMAKE_CURRENT_BINARY_DIR} + -o ${LCOV_BASE} + COMMAND lcov -q -z -d ${CMAKE_CURRENT_BINARY_DIR} + -o ${LCOV_TEST} + COMMAND lcov -q -z -d ${CMAKE_CURRENT_BINARY_DIR} + -o ${LCOV_TOTAL}) + +add_custom_target(capture-coverage + COMMENT "Capture coverage data" + DEPENDS ${LCOV_BASE} + COMMAND lcov -c -d ${CMAKE_CURRENT_BINARY_DIR} -o ${LCOV_TEST} --include "'${SRC_ABS}*'" + 2>${LCOV_ERR} >${LCOV_LOG} + COMMAND lcov -a ${LCOV_BASE} -a ${LCOV_TEST} -o ${LCOV_TOTAL} + >>${LCOV_LOG}) +``` + +Breaking it down: + +``` +COMMAND lcov -c -i -d ${CMAKE_CURRENT_BINARY_DIR} --include "'${SRC_ABS}*'" + -o ${LCOV_BASE} 2>${LCOV_ERR} >${LCOV_LOG}) +``` + +Collects initial (zero) coverage for files in coresdk/, so we aren't wasting time getting coverage +on external libraries. As the lcov man page states for -i/--initial: + +> Run lcov with -c and this option on the directories containing .bb, .bbg or .gcno files before +> running any test case. The result is a "baseline" coverage data file that contains zero coverage +> for every instrumented line. Combine this data file (using lcov -a) with coverage data files +> captured after a test run to ensure that the percentage of total lines covered is correct even +> when not all source code files were loaded during the test. Refer to: +> https://linux.die.net/man/1/lcov + +``` +COMMAND lcov -q -z -d ${CMAKE_CURRENT_BINARY_DIR} + -o ${LCOV_BASE} +COMMAND lcov -q -z -d ${CMAKE_CURRENT_BINARY_DIR} + -o ${LCOV_TEST} +COMMAND lcov -q -z -d ${CMAKE_CURRENT_BINARY_DIR} + -o ${LCOV_TOTAL}) +``` + +Reset execution counts to zero so that successive runs don't influence the count. + +``` +COMMAND lcov -c -d ${CMAKE_CURRENT_BINARY_DIR} -o ${LCOV_TEST} --include "'${SRC_ABS}*'" + 2>${LCOV_ERR} >${LCOV_LOG} +COMMAND lcov -a ${LCOV_BASE} -a ${LCOV_TEST} -o ${LCOV_TOTAL} + >>${LCOV_LOG}) +``` + +Combine the baseline and test coverage data. + +## Viewing coverage + +The Visual Studio Code extension CMake Tools can be configured to show code coverage when tests are +run from the Testing tab. It offers line highlighting for executed lines as well as a summary of +coverage over all source files. + +- https://marketplace.visualstudio.com/items?itemName=ms-vscode.cmake-tools +- Setup requires having custom targets in CMakeLists.txt which you can set in the extension options + +Alternatively, the Gcov Viewer can highlight executed lines but doesn't appear to offer a summary +view + +- https://marketplace.visualstudio.com/items?itemName=JacquesLucke.gcov-viewer + +## Concerns + +- What about platforms other than Linux/WSL? +- What if someone is using clang and not gcc? diff --git a/docs/Splashkit/Unit Testing/research/extension-comparison.md b/docs/Splashkit/Unit Testing/research/extension-comparison.md new file mode 100644 index 000000000..cac552450 --- /dev/null +++ b/docs/Splashkit/Unit Testing/research/extension-comparison.md @@ -0,0 +1,50 @@ +# Visual Studio Code extensions for unit testing + +Here's a brief comparison of two extensions to integrate our Catch2 unit testing into VS Code. It's +also worth noting that both of these can be installed at the same time, if the user doesn't mind +having two sets of tests in VS Code, each under their own tree. + +## CMake Tools + +- Author: Microsoft +- https://marketplace.visualstudio.com/items?itemName=ms-vscode.cmake-tools +- https://github.com/microsoft/vscode-cmake-tools + +This extension requires some changes to the project's CMakeLists.txt for the CTest integration to +work. The developer Catch2 recommends CMake integration anyway, and it's a fairly simple change, +just a few lines: + +``` +include(CTest) +include("${CMAKE_CURRENT_SOURCE_DIR}/catch/Catch.cmake") +catch_discover_tests(skunit_tests) +``` + +CMake Tools also has integration with VS Code's coverage feature, giving us another feature for +future consideration. This highlights lines that have run in testing and also shows a summary of +code coverage for the whole project. + +Doesn't allow you to select a test to go straight to its line in code, and the tests are structured +in a flat list. A tree view requires renaming the tests themselves with delimiters that the +extension can read, for example "utilities/is_prime". + +## C++ Test Mate + +- Author: Mate Pek +- https://marketplace.visualstudio.com/items?itemName=matepek.vscode-catch2-test-adapter +- https://github.com/matepek/vscode-catch2-test-adapter + +Can pick up tests without them being registered with CTest, meaning no changes to CMakeLists.txt is +necessary. There's the ability to select a test to jump to it in code. The extension can be +configured to parse tests so that the list is nicely organised (for example, grouping by source code +file), without requiring changes to test names. + +Doesn't have a coverage feature, so that would have to be done via a separate extension. See issue: +https://github.com/matepek/vscode-catch2-test-adapter/issues/433 Gcov viewer is an option for this, +but isn't as nicely integrated with VS Code, and doesn't have a nice coverage summary. See +extension: https://marketplace.visualstudio.com/items?itemName=JacquesLucke.gcov-viewer + +Has a problem with Catch2 2.x (which we use), where test names greater than 80 characters fail to +get run. See issue: https://github.com/matepek/vscode-catch2-test-adapter/issues/21 This seems to +have been fixed in Catch2 3.x but that would require addressing some breaking changes in the upgrade +from 2.x. diff --git a/docs/Splashkit/Unit Testing/research/isolation.md b/docs/Splashkit/Unit Testing/research/isolation.md new file mode 100644 index 000000000..8abad4154 --- /dev/null +++ b/docs/Splashkit/Unit Testing/research/isolation.md @@ -0,0 +1,74 @@ +# What is a unit? +Conceptually, the definition of a unit is fuzzy. It could be a single function, a method of a class, or a single behaviour. +More broadly, a unit is the smallest testable piece of code. + +For SplashKit we use the [Catch2](https://github.com/catchorg/Catch2) testing framework, which allows for writing tests in C++ with useful macros for test cases and assertions. + +# Isolation +As the name suggests, unit tests test a single unit in isolation. We check the behaviour of a unit by making assertions as to how it should behave. +For example, to test a function that converts a hexadecimal string to an integer we might check that its output for a given input matches the expect output: +```cpp +TEST_CASE("convert a hex string to an integer", "[hex_to_dec]") +{ + SECTION("hex string is converted to integer") + { + REQUIRE(hex_to_dec("A") == 10); + } + SECTION("input uses the invalid symbol G") + { + REQUIRE(hex_to_dec("G") == 0); + } +} +``` +Test isolation is desirable because it makes bugs easier to trace. If a unit test fails, we know where it failed and what behaviour it failed to produce (given by the test case). In an integration test, where two or more units are being tested together, the source of the problem is less obvious. In a system or end-to-end test, even less so. Nevertheless, integration and wider scoped tests are also important, to test behaviour in an environment closer to production. + +It gets harder to test a unit in isolation with more complex code. We might want to test the behaviour of a function that calls another function, that in turn calls another function, and some of these function calls might be external (e.g. provided by a library). If we simply test such a function as it would be used in production code, we're effectively performing an integration test - testing how multiple units behave together. + +As an example, consider the following SplashKit function that returns the screen center: +```cpp +point_2d screen_center() +{ + return point_at(_camera_x + screen_width() / 2.0f, _camera_y + screen_height() / 2.0f); +} +``` +At first glance, this seems like a simple enough unit to test: check that the value returned is of an expected value. However, this implementation calls functions `screen_width` and `screen_height`, which in turn call functions that return the width/height of the current window: +```cpp +int screen_width() +{ + return window_width(current_window()); +} +``` +This implies the presence of a window object to begin with, so we'd not only be testing the logic of computing the screen's center point, but also the behaviours of creating a new window and being able to retrieve the window dimensions of it. This also means using an external dependency, SDL, to draw the window. +```cpp +TEST_CASE_METHOD(CameraTest, "Get screen center in world space", "[camera][graphics][window][integration]") +{ + open_window("Get screen center in world space", 100, 100); + point_2d center = screen_center(); + REQUIRE_THAT(center.x, WithinRel(50.0)); + REQUIRE_THAT(center.y, WithinRel(50.0)); + close_current_window(); +} +``` +If we test this without a window, we can't get the dimensions to compute the screen center to begin with. + +## Reducing dependencies +To address this, we can use the concept of test doubles. These are functions or objects that we write to satisfy a dependency without using the dependency itself. +For example, we could create a window with specific dimensions and the minimum functionality to allow the `screen_center` function to work, without everything that a real window requires (like drawing the window on screen, handling on close events, and so on). + +There are different types of test doubles, including stubs, mocks, fakes, and dummies. Some provide static input or output, or serve as a stand-in for an object. The overall concept of test doubles is what is relevant in this document. + +## Mocking frameworks +To make the process of creating test doubles simpler, we can use a framework. The two frameworks investigated for use in SplashKit were: +[FakeIt](https://github.com/eranpeer/FakeIt) +[Trompeloeil](https://github.com/rollbear/trompeloeil) + +Given similar feature sets, Trompeloeil was selected, because it's simple to integrate with the Catch2 testing framework and examples are provided for that usage. + +# Test doubles and SplashKit +Test doubles generally rely on a degree of abstraction to implement. Often this is done by implementing an interface, so that instead of the production implementation, a double is used with the minimum requirements for a test to run. + +However, SplashKit has a lot of free functions, which makes it difficult to do. With Trompeleoil, they cannot be mocked, and need to be wrapped in an [API structure](https://github.com/rollbear/trompeloeil/blob/main/docs/CookBook.md#mocking_free_functions). + +Refactoring SplashKit to increase abstraction in this way to accommodate test doubles would be a great effort that is likely beyond a single trimester's worth of work. It would likely result in all sorts of breaking changes. An alternative would be to link the test executable with only the code being tested, and create test doubles for any functions that cause the test in question to not compile. This would require compilation/linking for each individual source file. + +It is for this reason that integration tests, which are also capable of being written in C++ with Catch2, will be written where unit tests are currently not able to be written. This will still improve the code coverage and testing quality of SplashKit without being too much of a burden. \ No newline at end of file