Unit testing research

docs/Splashkit/Unit Testing/Index.md

@@ -0,0 +1,17 @@
+# Overview
+
+It's important that all aspects of SplashKit are thoroughly tested. This is especially important
+because SplashKit is designed to be used by beginners - they should be able to focus on learning to
+develop software and fix bugs in their own code rather than wondering why some aspect of the
+framework isn't functioning as expected. Unit tests allow us to efficiently test and validate the
+core functionality of SplashKit.
+
+# Goals
+
+Currently, unit testing isn't given much of a spotlight in documentation, and developing and running
+tests is more cumbersome than it could be. So, goals for unit testing are:
+
+- Research ways of improving unit testing workflow, including integration into Visual Studio Code,
+  which is the IDE recommended in SplashKit documentation
+- Create documentation for unit testing, including an onboarding guide for those wanting to write
+  and run unit tests

docs/Splashkit/Unit Testing/research/build-run-workflow.md

@@ -0,0 +1,173 @@
+# Current workflow
+
+As can be seen, the current workflow doesn't require a lot of set up but involves using the command
+line (including debugging).
+
+## Linux
+
+1. Install prerequisites
+
+   - Update
+
+     ```shell
+     sudo apt-get update
+     sudo apt-get upgrade -y
+     ```
+
+   - Install
+
+     ```shell
+     sudo apt-get install -y \
+     git build-essential cmake g++ libpng-dev libcurl4-openssl-dev libsdl2-dev \
+     libsdl2-mixer-dev libsdl2-gfx-dev libsdl2-image-dev libsdl2-net-dev libsdl2-ttf-dev \
+     libmikmod-dev libbz2-dev libflac-dev libvorbis-dev libwebp-dev libfreetype6-dev
+     ```
+
+2. Build test project
+
+   ```shell
+   cd projects/cmake
+   cmake .
+   make
+   ```
+
+3. Run unit tests
+
+   - Running all tests
+
+     ```shell
+     cd ../../bin
+     ./skunit_tests
+     ```
+
+     Example output:
+
+     ```
+     (14/04/2025) ERROR -> Invalid binary string passed to bin_to_oct: 2, returning empty string [raised in basics.cpp:340]
+     (14/04/2025) ERROR -> Invalid hexadecimal string passed to hex_to_oct: G, returning empty string [raised in basics.cpp:371]
+     (14/04/2025) ERROR -> Invalid octal string passed to hex_to_dec: G, returning 0 [raised in basics.cpp:305]
+     (14/04/2025) ERROR -> Invalid octal string passed to oct_to_hex: 8, returning empty string [raised in basics.cpp:383]
+     ===============================================================================
+     All tests passed (1258 assertions in 74 test cases)
+     ```
+
+   - Running a specific test
+
+     ```shell
+     ./skunit_tests [test name or tag]
+     ```
+
+     Example:
+
+     ```
+     ./skunit_tests "[least_common_multiple]"
+     Filters: [least_common_multiple]
+     ===============================================================================
+     All tests passed (7 assertions in 1 test case)
+     ```
+
+# New workflow
+
+The details depend on which VS Code extension is used, but this example using the CMake Tools
+extension shows that the process can be much smoother. This also offers features like test listing
+and debugging in VS Code, albeit with extra setup.
+
+## Linux
+
+1. Install prerequisites
+
+   - Update
+
+     ```shell
+     sudo apt-get update
+     sudo apt-get upgrade -y
+     ```
+
+   - Install packages
+
+     ```shell
+     sudo apt-get install -y \
+     git build-essential cmake g++ libpng-dev libcurl4-openssl-dev libsdl2-dev \
+     libsdl2-mixer-dev libsdl2-gfx-dev libsdl2-image-dev libsdl2-net-dev libsdl2-ttf-dev \
+     libmikmod-dev libbz2-dev libflac-dev libvorbis-dev libwebp-dev libfreetype6-dev
+     ```
+
+   - Install CMake Tools from VS Code's extension browser or from
+     https://marketplace.visualstudio.com/items?itemName=ms-vscode.cmake-tools
+
+2. Configure extension
+
+   - Select ${workspaceFolder}/projects/cmake/CMakeLists.txt
+
+   - Select the Linux configure preset
+
+   - On the CMake Tools extension tab set the Build target to skunit_tests
+
+   - Set the Debug and Launch target to skunit_tests
+
+3. Configure VS Code debugging
+
+   - Go to the Run and Debug tab of VS Code and create a launch.json file
+
+     ```json
+     {
+       "version": "0.2.0",
+       "configurations": [
+         {
+           "name": "(gdb) Launch",
+           "type": "cppdbg",
+           "request": "launch",
+           "program": "${command:cmake.launchTargetPath}",
+           "args": [],
+           "stopAtEntry": false,
+           "cwd": "${workspaceFolder}",
+           "environment": [
+             {
+               "name": "PATH",
+               "value": "${env:PATH}:${command:cmake.getLaunchTargetDirectory}"
+             }
+           ],
+           "externalConsole": false,
+           "MIMode": "gdb",
+           "setupCommands": [
+             {
+               "description": "Enable pretty-printing for gdb",
+               "text": "-enable-pretty-printing",
+               "ignoreFailures": true
+             },
+             {
+               "description": "Set Disassembly Flavor to Intel",
+               "text": "-gdb-set disassembly-flavor intel",
+               "ignoreFailures": true
+             }
+           ]
+         }
+       ]
+     }
+     ```
+
+4. Build test project
+
+   Building can be done from the terminal, as before, or within VS Code in one of the following ways
+
+   - In the CMake Tools extension, select Build
+
+   - On the Testing tab, select Refresh Tests
+
+5. Run unit tests
+
+   In addition to running tests from the terminal, tests will show up on the Testing tab of VS Code.
+
+   - Running all tests
+
+     Select Run Tests, each test will be run and the status of each can be seen in the test list.
+
+   - Running a specific test
+
+     Select Run Test next to any test on the test list to run it
+
+6. Run unit tests with debugging
+
+   Break points can be set in VS Code like in normal programs, by selecting the red next to a line
+   in a test or pressing F9. Then, select Debug Tests to run all tests with debugging or Debug Test
+   next to the test name to run a specific test with debugging.

docs/Splashkit/Unit Testing/research/code-coverage.md

@@ -0,0 +1,143 @@
+# Code Coverage
+
+Collecting code coverage would help us check whether the codebase is adequately tested by unit
+tests. Ideally, coverage would be integrated into the unit testing workflow and able to be done in
+Visual Studio Code. Some research into code coverage has been done and so far it seems feasible to
+do on Linux and Windows with WSL, but problems were encountered doing so on Windows with MSYS2.
+Collecting coverage on Mac hasn't been tried.
+
+## Collecting coverage on Windows
+
+- Coverage collection when using WSL works fine.
+- The coverage collection process failed on MSYS2 when using Mingw64, which is recommended by the
+  SplashKit installation guide.
+- Using other MSYS2 environments, like UCRT64 or MSYS wasn't tried.
+
+## Code coverage on Linux/WSL
+
+Code coverage on Linux involves setting options at compile time to enable coverage generation,
+running the unit tests to generate the data, then displaying it (either as a summary of coverage
+over the source files, or highlighting lines that have been executed in the IDE).
+
+1. Setting compiler and linker options Can be done as part of the build process with CMake by adding
+   the options to CMakeLists.txt Assuming the use of g++:
+
+   ```
+   add_compile_options(--coverage)
+   add_link_options(--coverage)
+   ```
+
+   This can be integrated with CMake configure presets to provide an easy way to switch between
+   building with and without coverage:
+
+   ```
+   option(USE_COVERAGE "Enable GCOV during the build" OFF)
+   if(USE_COVERAGE)
+       add_compile_options(--coverage)
+       add_link_options(--coverage)
+   endif()
+   ```
+
+   See more on CMake presets: https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html See
+   more on gcc options, including --coverage:
+   https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html
+
+2. Collect coverage Coverage on Linux can be collected with LCOV or gcovr
+
+- https://github.com/linux-test-project/lcov
+- https://github.com/gcovr/gcovr
+
+Example of integrating LCOV into the build process:
+
+```
+# Coverage collection with LCOV
+# Get absolute path to SK_SRC so we can pass it to LCOV
+# and collect coverage only for src files
+get_filename_component(SRC_ABS ${SK_SRC} ABSOLUTE)
+message(SRC_ABS="${SRC_ABS}")
+
+find_program(LCOV lcov REQUIRED)
+set(LCOV_BASE lcov-base.info)
+set(LCOV_TEST lcov-test.info)
+set(LCOV_TOTAL lcov.info)
+set(LCOV_LOG lcov.log)
+set(LCOV_ERR lcov.err)
+add_custom_target(init-coverage
+    COMMENT "Collecting initial coverage"
+    COMMAND lcov -c -i -d ${CMAKE_CURRENT_BINARY_DIR} --include "'${SRC_ABS}*'"
+        -o ${LCOV_BASE} 2>${LCOV_ERR} >${LCOV_LOG})
+add_dependencies(init-coverage reset-coverage)
+
+add_custom_target(reset-coverage
+    COMMENT "Reset all coverage counters to zero"
+    COMMAND lcov -q -z -d ${CMAKE_CURRENT_BINARY_DIR}
+        -o ${LCOV_BASE}
+    COMMAND lcov -q -z -d ${CMAKE_CURRENT_BINARY_DIR}
+        -o ${LCOV_TEST}
+    COMMAND lcov -q -z -d ${CMAKE_CURRENT_BINARY_DIR}
+        -o ${LCOV_TOTAL})
+
+add_custom_target(capture-coverage
+    COMMENT "Capture coverage data"
+    DEPENDS ${LCOV_BASE}
+    COMMAND lcov -c -d ${CMAKE_CURRENT_BINARY_DIR} -o ${LCOV_TEST} --include "'${SRC_ABS}*'"
+        2>${LCOV_ERR} >${LCOV_LOG}
+    COMMAND lcov -a ${LCOV_BASE} -a ${LCOV_TEST} -o ${LCOV_TOTAL}
+        >>${LCOV_LOG})
+```
+
+Breaking it down:
+
+```
+COMMAND lcov -c -i -d ${CMAKE_CURRENT_BINARY_DIR} --include "'${SRC_ABS}*'"
+    -o ${LCOV_BASE} 2>${LCOV_ERR} >${LCOV_LOG})
+```
+
+Collects initial (zero) coverage for files in coresdk/, so we aren't wasting time getting coverage
+on external libraries. As the lcov man page states for -i/--initial:
+
+> Run lcov with -c and this option on the directories containing .bb, .bbg or .gcno files before
+> running any test case. The result is a "baseline" coverage data file that contains zero coverage
+> for every instrumented line. Combine this data file (using lcov -a) with coverage data files
+> captured after a test run to ensure that the percentage of total lines covered is correct even
+> when not all source code files were loaded during the test. Refer to:
+> https://linux.die.net/man/1/lcov
+
+```
+COMMAND lcov -q -z -d ${CMAKE_CURRENT_BINARY_DIR}
+    -o ${LCOV_BASE}
+COMMAND lcov -q -z -d ${CMAKE_CURRENT_BINARY_DIR}
+    -o ${LCOV_TEST}
+COMMAND lcov -q -z -d ${CMAKE_CURRENT_BINARY_DIR}
+    -o ${LCOV_TOTAL})
+```
+
+Reset execution counts to zero so that successive runs don't influence the count.
+
+```
+COMMAND lcov -c -d ${CMAKE_CURRENT_BINARY_DIR} -o ${LCOV_TEST} --include "'${SRC_ABS}*'"
+    2>${LCOV_ERR} >${LCOV_LOG}
+COMMAND lcov -a ${LCOV_BASE} -a ${LCOV_TEST} -o ${LCOV_TOTAL}
+    >>${LCOV_LOG})
+```
+
+Combine the baseline and test coverage data.
+
+## Viewing coverage
+
+The Visual Studio Code extension CMake Tools can be configured to show code coverage when tests are
+run from the Testing tab. It offers line highlighting for executed lines as well as a summary of
+coverage over all source files.
+
+- https://marketplace.visualstudio.com/items?itemName=ms-vscode.cmake-tools
+- Setup requires having custom targets in CMakeLists.txt which you can set in the extension options
+
+Alternatively, the Gcov Viewer can highlight executed lines but doesn't appear to offer a summary
+view
+
+- https://marketplace.visualstudio.com/items?itemName=JacquesLucke.gcov-viewer
+
+## Concerns
+
+- What about platforms other than Linux/WSL?
+- What if someone is using clang and not gcc?

docs/Splashkit/Unit Testing/research/extension-comparison.md

@@ -0,0 +1,50 @@
+# Visual Studio Code extensions for unit testing
+
+Here's a brief comparison of two extensions to integrate our Catch2 unit testing into VS Code. It's
+also worth noting that both of these can be installed at the same time, if the user doesn't mind
+having two sets of tests in VS Code, each under their own tree.
+
+## CMake Tools
+
+- Author: Microsoft
+- https://marketplace.visualstudio.com/items?itemName=ms-vscode.cmake-tools
+- https://github.com/microsoft/vscode-cmake-tools
+
+This extension requires some changes to the project's CMakeLists.txt for the CTest integration to
+work. The developer Catch2 recommends CMake integration anyway, and it's a fairly simple change,
+just a few lines:
+
+```
+include(CTest)
+include("${CMAKE_CURRENT_SOURCE_DIR}/catch/Catch.cmake")
+catch_discover_tests(skunit_tests)
+```
+
+CMake Tools also has integration with VS Code's coverage feature, giving us another feature for
+future consideration. This highlights lines that have run in testing and also shows a summary of
+code coverage for the whole project.
+
+Doesn't allow you to select a test to go straight to its line in code, and the tests are structured
+in a flat list. A tree view requires renaming the tests themselves with delimiters that the
+extension can read, for example "utilities/is_prime".
+
+## C++ Test Mate
+
+- Author: Mate Pek
+- https://marketplace.visualstudio.com/items?itemName=matepek.vscode-catch2-test-adapter
+- https://github.com/matepek/vscode-catch2-test-adapter
+
+Can pick up tests without them being registered with CTest, meaning no changes to CMakeLists.txt is
+necessary. There's the ability to select a test to jump to it in code. The extension can be
+configured to parse tests so that the list is nicely organised (for example, grouping by source code
+file), without requiring changes to test names.
+
+Doesn't have a coverage feature, so that would have to be done via a separate extension. See issue:
+https://github.com/matepek/vscode-catch2-test-adapter/issues/433 Gcov viewer is an option for this,
+but isn't as nicely integrated with VS Code, and doesn't have a nice coverage summary. See
+extension: https://marketplace.visualstudio.com/items?itemName=JacquesLucke.gcov-viewer
+
+Has a problem with Catch2 2.x (which we use), where test names greater than 80 characters fail to
+get run. See issue: https://github.com/matepek/vscode-catch2-test-adapter/issues/21 This seems to
+have been fixed in Catch2 3.x but that would require addressing some breaking changes in the upgrade
+from 2.x.