Start using a C++ unit testing framework #643
Comments
jsiwek
added
the
Type: Project
|
After looking some more at doctest, I also think that that might be good choice for us. I like how the tests can be put right into the source code, and compiling along with the main binary solves the problem of having to redeclare all those globals in a separate test executable. |
|
Another nice-to-have: Supporting benchmarks as well as tests. As we refactor and modernize more core bits, would be great to have individual benchmarks for things like 'Dict'. |
|
Is doctest consensus at this point? I still have the unit tests for |
|
@Neverlord doesn't feel like strong consensus, but yeah, seems a couple people are curious if doctest will work well. If you did find time to try setting up a few tests with it, will be nice to hear your opinion of that experience. Especially since CAF's test framework was a viable alternative, interested to know if the reasons for preferring doctest were legit (e.g. workflow benefits). |
Well, the initial scaffolding is easy enough:
I've pushed my initial scaffolding if anyone wants to take a look (see 0955947). Unfortunately, linking is where trouble starts. Doctest bundles the unit tests with production code, but this does not play nicely with the way we currently build the I figured the latter should be simple, because zeek/zeekdiff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 564fd7acb..bf55a332a 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -15,7 +15,7 @@ set(bro_PLUGIN_BIF_SCRIPTS CACHE INTERNAL "Zeek script stubs for BIFs in Zeek pl
# If TRUE, use CMake's object libraries for sub-directories instead of
# static libraries. This requires CMake >= 2.8.8.
-set(bro_HAVE_OBJECT_LIBRARIES FALSE)
+set(bro_HAVE_OBJECT_LIBRARIES YES)
configure_file(version.c.in ${CMAKE_CURRENT_BINARY_DIR}/version.c)
configure_file(util-config.h.in ${CMAKE_CURRENT_BINARY_DIR}/util-config.h)
@@ -139,7 +139,9 @@ list(APPEND BINPAC_OUTPUTS "${BINPAC_OUTPUT_CC}")
########################################################################
set(bro_SUBDIR_LIBS CACHE INTERNAL "subdir libraries" FORCE)
+set(bro_SUBDIR_DEPS CACHE INTERNAL "subdir dependencies" FORCE)
set(bro_PLUGIN_LIBS CACHE INTERNAL "plugin libraries" FORCE)
+set(bro_PLUGIN_DEPS CACHE INTERNAL "plugin dependencies" FORCE)
add_subdirectory(analyzer)
add_subdirectory(broker)
@@ -396,12 +398,12 @@ add_dependencies(generate_outputs generate_outputs_stage2a generate_outputs_stag
# Build __load__.zeek files for standard *.bif.zeek.
bro_bif_create_loader(bif_loader "${bro_BASE_BIF_SCRIPTS}")
-add_dependencies(bif_loader ${bro_SUBDIRS})
+add_dependencies(bif_loader ${bro_PLUGIN_DEPS} ${bro_SUBDIR_DEPS})
add_dependencies(zeek bif_loader)
# Build __load__.zeek files for plugins/*.bif.zeek.
bro_bif_create_loader(bif_loader_plugins "${bro_PLUGIN_BIF_SCRIPTS}")
-add_dependencies(bif_loader_plugins ${bro_SUBDIRS})
+add_dependencies(bif_loader_plugins ${bro_PLUGIN_DEPS} ${bro_SUBDIR_DEPS})
add_dependencies(zeek bif_loader_plugins)
# Install *.bif.zeek.
@@ -452,6 +454,9 @@ install(FILES
DESTINATION include/zeek/3rdparty/tsl-ordered-map
)
+########################################################################
+## Unit test setup
+
########################################################################
## Clang-tidy target now that we have all of the sources
zeek/cmakediff --git a/ZeekPluginStatic.cmake b/ZeekPluginStatic.cmake
index 339b0c8..2883a30 100644
--- a/ZeekPluginStatic.cmake
+++ b/ZeekPluginStatic.cmake
@@ -35,6 +35,7 @@ function(bro_plugin_end_static)
add_dependencies(${_plugin_lib} generate_outputs)
set(bro_PLUGIN_LIBS ${bro_PLUGIN_LIBS} "${_target}" CACHE INTERNAL "plugin libraries")
+ set(bro_PLUGIN_DEPS ${bro_PLUGIN_DEPS} "${_plugin_lib}" CACHE INTERNAL "plugin dependencies")
endfunction()
macro(_plugin_target_name_static target ns name)
diff --git a/ZeekSubdir.cmake b/ZeekSubdir.cmake
index 8ae987b..8c74a36 100644
--- a/ZeekSubdir.cmake
+++ b/ZeekSubdir.cmake
@@ -11,5 +11,7 @@ function(bro_add_subdir_library name)
endif ()
set(bro_SUBDIR_LIBS "${_target}" ${bro_SUBDIR_LIBS} CACHE INTERNAL "subdir libraries")
+ set(bro_SUBDIR_DEPS "bro_${name}" ${bro_SUBDIR_DEPS} CACHE INTERNAL "subdir dependencies")
add_clang_tidy_files(${ARGN})
endfunction()
+
After applying the patches, I get tons of duplicate symbols errors: It's probably not hard to fix, but I didn't want to invest too much time since I don't know how deep the rabbit hole goes and what surprises are coming up after fixing that. If object libraries (and doctest) are the way we want to go, I can continue from here later - or someone else takes over. 🙂 |
|
Switching to use object libraries (and then also removing the alternate, now-unused CMake code paths) generally sounds nice. A guess at what your patch is missing is just removing the Lines 7 to 9 in 7b9a27c A related commit: |
The file Commit 8e7ef00 also removed a static in |
|
No immediate thought on what's broken here with the disabled cmake path, but generally going with object libraries sounds good; that was the plan already several years ago until it became clear we couldn't because of old cmakes ... Now that we don't have that cmake restriction anymore, it'll be good to get this cleaned up and remove the work-arounds. |
|
Initial doctest integration merged via #682 |
This follows up from part of #633. We want to decide on a testing framework/setup to use and then add the basic infrastructure to use it in Zeek. E.g. pick a few util functions and add tests for them to serve as examples. Catch2, googletest, Boost.Test, doctest, and CAF were mentioned frameworks:
Agree that's a nice bonus, except actor-ifying Zeek isn't something that's really made it into the near-term roadmap and Broker is niche enough that's it's not too terrible that it uses a separate testing framework.
I'd prioritize low-barrier/low-impact unit testing -- make it easy for people (core devs or external contributors) to write/run tests. My wishlist in order of what I'd like most in a unit test framework:
Other nice-to-haves (IMO) to consider:
Any thoughts on requirements/features or initial opinions on frameworks ?
doctest seems to align with the low-barrier/low-impact goal so I'm interested to try it.
The text was updated successfully, but these errors were encountered: