Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(testing): add a unit testing framework (#5509)
* feat(tests): commit Catch2 v2.13.2 to the repo
* feat(tests): add test_main.cpp
* chore(tests): update gitignore for multiple build locations
* build(tests): copy & update the .winmake file
- still overcomplicated for what it needs to do (e.g. no build_dir indirection is needed)
- will need modification as we experiment with mocking of dependencies
* ci(tests): Add unit tests to AppVeyor build process
- rename .test-winmake to makefile, fix path separator
- add test artifact uploading
* ci(tests): add windows-test workflow
- maybe: Add test binary caching even when tests fail
- add comment-on-pr reporting
- catch2 's junit reporter isn't actually junit, just junit-esque. So don't bother with it for GitHub Actions
- disabled, as it requires linux
* doc(tests): add a template test file, to be cp'd and then edited
provide some bootstrapping for following work
* build: remove unnecessary main.cpp rule
* test(Set): add unit tests for the Set template class
* test(Point): add generic Point tests
TODO: redo makefile strategy
- The original strategy works only for header-only classes, as including the header is sufficient for the implementation to be generated.
- Any non-header class has an implementation binary that needs to be linked against
- Simply chucking all implementation files at the end of the linker includes will make it too easy to write tests that use actual implementations of other files, rather than requiring us to stub the behavior of classes not being immediately tested.
- can possibly write a specific make rule for translating test_A.cpp into test_A.dll <- test_A.o + A.o
- Probably it makes sense to define the test target in the root .winmake, to avoid duplicating the rules for building the actual source files as well, & to make sure the same flags are used for test and implementation
* ci(appveyor): use existing .obj files
* build(cbp): update path separators from "\" to "/"
* build(codeblocks): Create Endless Sky workspace, containing game and library projects
- source files are included in the respective static libraries
- executables produced by linking the game's static library + exe-specific source files (e.g. main.cpp), and the third-party source files (SDL, OpenAl, etc)
TODO: test project with similar dependency on endless-sky-lib
* build(codeblocks): add test project with configured dependencies
- simplify the required local changes to just the compiler + linker search directories
- Remove the path component of linked libraries
- Add library search directories
- Update makefile approach to use the library
- for now, remove link-against-dev64 things as the tested classes don't need them
* build(winmake): Add static archive generation step
TODO: add a "test" target that
1) ensures the correct archive exists
2) passes the compiler flags down
3) to compile the test files and
4) create the test binary
* build(linux): create intermediate archive file
* build(linux): update SCons target list
- default to building the game
- 2 new test targets
`scons build-tests` -> will compile game sources & test sources, and link them into "endless-sky-tests"
`scons test` -> will first run `scons build-tests` and then will execute the test binary
- print scons version when building the game binary via GitHub Actions
* ci(appveyor): guard test report upload on report existence
- remove unnecessary .winmake generation
* build(test): allow override of library dir
- use override in AppVeyor
- TODO: refactor windows CLI builds to avoid this hack
* ci(travis): Update Travis CI config to run unit tests
* ci: fix appveyor, maybe add ccache to travis
- ci(appveyor): update test CLI flags
- Remove double-builds in AppVeyor, Travis
- only build for pushes to
- branch master
- branches that have a PR pointed to master
- branches like v1 or v0.9.13
* ci(actions): require passing unit tests to set continuous tag
* ci(actions): limit dependency downloads
- minor step renames
- define specific libglew runtime dep per os node
* ci(appveyor): run unit tests before parse/integration tests
* build(actions): remove deprecated set-env, set-path syntax
* ci(linux): add cache/artifact steps for libendless-sky.a file
may not need to both cache and upload
* test(linux): add unit test CI job for linux
- update ubuntu tests, rename jobs
- fix xvfb job declaration
- add missing cache key
* ci(ui-tests): remove dev-dep installs
- probably need additional dependencies
- add more debugging to ubuntu test stage
- more debugging, try install-recommends for xvfb
- download devdeps for debugging unit test compile reasons
* ci: remove caching of dev lib
- rely on ccache, since timestamps aren't preserved and we would need to cache the whole build directory
* ci(ui-tests): Investigate updates to XVFB configuration
- Need to install deps, even if not compiling test binary, to execute test binaries
- OpenGL support is equivalent for 20.04 and 18.04
- should investigate differences in e.g. SDL, OS env, etc.
* ci(xvfb): refactor headless test script
- add test for xvfb process, i.e. exit if the process couldn't be created
- move glxinfo query to function
- conditionally print OpenGL info (e.g. only in CI runs)
* test: swap some orthogonal asserts from REQUIRE to CHECK
test all the parts of the leaf section
Keep REQUIRE for "showstopper" asserts
* docs(testing): update notes on currnet unit test limitations
- need to further investigate link-time seam injections, to replace the tested code's dependencies that it links against
e.g., for something that links against GameData methods, we would like to provide a mock GameData object whose behavior we can control, rather than the real GameData implementation being called
Without this injection support, to unit test e.g. PlayerInfo or NPC or StellarObject, we would need to link against all dependencies, including audio & SDL support.
* test(DataNode): add basic DataNode tests
- much more to do, e.g. test ::Value, etc.
* build(linux): enable link-time-optimization
- allow the linker step to omit including known-dead code, which limits the required link-time dependencies
- e.g. since Files::Init is not called by the DataNode test file, we can link against the actual Files implementation without including SDL headers or implementations
- also enables a smarter executable, since optimizations can be done across object files rather than just inside each TU.
TODO: enable LTO for CodeBlocks & makefile builds
* build(linux): attempt invoking g++7 on 16.04
* build(linux): allow passing flags to AR
e.g. to specify where the LTO plugin can be found
* refactor: forbid C-style casts
* ci(macos): re-enable macOS builds
* refactor(font): use compiler-generated default constructor
* docs(testing): create a test readme
slim down the test template and provide some generalized instructions
* build(codeblocks): enable link-time optimization for Code::Blocks builds
- Users will likely need to change their linker from "ar.exe" to "gcc-ar.exe" in the Code::Blocks toolchain settings page
* build(winmake): Replace Windows makefile with a wrapper to SCons
- reduce the number of build pipelines we need to support
* build(winmake): enable deleting the scons-local tarball
* build(winmake): use makefile automatic variables to limit repetition
* build(winmake): disable implicit rules to speed up make layer
* build(winmake): read PYTHON_BINARY from the environment
- in case user's default python is v2
* build(tests): remove windows test makefile
- deferring to scons build definition for proper dependency management
* build(winmake): don't show curl progress bar
* build(actions): update windows workflows
- remove ES_SRCLIB_DIR hack and update make invocations
* ci(travis): pass LTO-ready AR / RANLIB to xenial image
* ci(appveyor): use `before_test` stage to build test binary
- rather than run it via scons in build_script, and then run it manually later
* ci(actions): possibly fix macos path for parse test
* build(Scons): configure libraries and search paths for Windows
* build: disable old-style-cast on calls from included deps
* build(windows): avoid parallel invocation of scons by makefile
* build(scons): avoid double update of symbol table
- SCons StaticLibrary builder invokes ar and ranlib, so pass S to ar to skip the symbol table step. (This also means we don't need both gcc-ar and gcc-ranlib to be passed.)
- Enable thin archive support, if we're using gcc ar/ranlib, to speed up archive creation step
* build(scons): use os.path.join instead of assuming "/" separator
* ci(actions): update windows binary name EndlessSky -> endless-sky
* ci(appveyor): remove copy-to-root, update binary names/paths
* ci(actions): debug macos parse
* build(winmake): ensure we forward search dirs, even if using defaults
* build: update paths
- macos parse trial ? of ???
- remove relative path in build_windows
- print args passed to scons env
- print scons include dir
* build(winmake): wrap in quotes
- may need to escape "\", switch to "/"
* build(appveyor): enable multithreaded builds
* build(winmake): forward -j argument to scons
* build(winmake): update comments, don't pass DIR_MINGW64 to scons
(since scons tooling is finding the MinGW binaries and libraries without issue)
* build(winmake): re-add "dist" target
- restore -march=native for windows release builds
* build(winmake): simplify recipe declarations
* ci(actions): update windows ci & cd workflows
* build: allow specifying output binary directory
- default SCons build behavior (placing game in root directory) is preserved, but can be customized with parameter BIN_DIR
- restore default "dist" target behavior of winmake
- Windows workflows keep the old binary name, "EndlessSky.exe", to preserve compatibility with external tools like ESLauncher2
* build(winmake): don't pass raw -j to SCons
- SCons doesn't support auto-max-parallelism like make does
* ci: general fixups
* build(appveyor): Improve caching & artifact name
- strip out invalid filename chars, too, in case the branch name has any
* ci(actions): copy runtime deps for windows job
- remove unused redirect to file
* build(appveyor): parameterize over MinGW versions (6.3, 7.3, 8.1)
* ci(macos): persist only the macOS binary in ci job
* test(parse): pass explicit resource directory to tested binary
- redirect stderr to file & cat it if the game doesn't start properly
* ci(macos): only prepare macOS binary if newly built
* test: enable benchmarking tests
Will not run by default, but can be run manually:
./tests/endless-sky-tests [!benchmark]
* ci: run benchmarks via GitHub CI workflows
* test(ConditionSet): Add basic create & use tests
- beginning with empty ConditionSets
* test(ConditionSet): add basic tests for non-empty ConditionSets
- use istream interface of DataFile to create DataNodes with content from test strings
- MAYBE: could extract tokenizing logic from DataFile into something usable by DataFile and DataNode, and add a DataNode(std::string) constructor
- MAYBE: prevent errors.txt from being written by test code (or clean it up/configure location more appropriately, e.g. tests/errors.txt)
* test: disable WinCRT memory debugger- Loading branch information
