add ctest as test driver

[felixschurk]

• add enable_testing() in top level CMakeLists to discover tests
• add add_test for both unit tests as well as integration tests with prefixing the tests with the corresponding test type
• remove custom targets for tests
• remove unneeded warning from https://gitlab.kitware.com/cmake/cmake/-/issues/16062
• adds basic setup for resolving Change test runner to ctest #3445

Basically one can now invoke the whole test suite with running

cmake -S . -B build
cmake --build build --target build_tests
ctest --test-dir build/test 

and one does has a quite decent output of the actual test runs.

Parallel running -j 8, running specific test cases -R cpp or -R variant, having minimal progress bar --progress as well as repeating tests until failure --repeat-until-failure 100 is also possible with CTest.

[ryneeverett]

By switching from CMD to RUN for the test invocation in the docker images, the tests are now run when the images are built and nothing happens when they are run.

[felixschurk]

Yes that is true, I realized that yesterday but could not fix it on the spot. My knowledge with Docker is limited.

Would it maybe make sense to move the test invocation out into the test.yaml file?
I mean updating this line

taskwarrior/.github/workflows/tests.yaml

Line 105 in 60575a1

run: docker-compose run test-$CONTAINER

to

run: docker-compose run test-$CONTAINER ctest --test-dir build/test -j 8 --rerun-failed --output-on-failure

This way we would not have duplicated code in all the docker files, however it maybe would be a bit less/or maybe even more clear how we do run the tests.

[ryneeverett]

In my opinion reducing the duplication is beneficial and passing the command into the docker run invocation is equally clear to having a CMD in the image.

[felixschurk]

I am a bit sorry for all the failing CI jobs and commit updates, however locally testing it with one image/container it does run. So I am guessing there is a problem with escaping etc. in the tests.yaml file.

I had a similar issue already when setting up another CI. (locally working but in GH Actions not).

[ryneeverett]

I don't think --entrypoint is what you want. It looks like you were close with 1888816. CI failed with "No such service: test-fedora40 ctest". Maybe you just need some quotes so it splits the arguments correctly?

        run: docker-compose run 'test-${{ env.CONTAINER }}' 'ctest --test-dir build/test -j 8 --output-on-failure --rerun-failed'

(Not 100% sure if it should be single or double quotes but I assume the template variables are substituted prior to evaluation and thus don't care.)

[felixschurk]

Hm, yes definetely the splitting fixed one problem.
I then detected some problem with a whitespace (which as somehow different encoded).

Now it does not find some file/directory. Let's see what the problem ist.

[djmitche]

This is great! I love the more succinct output. A few notes from my experience that don't need to be fixed before this is landed:

• The ctest invocation, and similarly make -Cbuild test, don't seem to rebuild tests if their source files have changed, which can lead to surprises when a test run keeps failing after the test has been fixed. Maybe cmake --build-and-test helps with this?
• The tests take a loong time to run. ctest takes a -jN option just like make, which might be a useful thing to recommend (although it did cause ids.test.py to fail for me!)
• Having a documented way to run a single test will be useful, too.

One thing that does need to be addressed here, is updating the docs on how to run the tests (doc/devel/contrib/development.md).

[djmitche]

Thanks for getting rid of this!

[felixschurk]

In my opinion reducing the duplication is beneficial and passing the command into the docker run invocation is equally clear to having a CMD in the image.

I am sorry, I was not able to work that out. Maybe it makes sense to have this as a new issue (should be an easy one) when someone knows GitHub Actions and Docker.

[felixschurk]

Currently, missing is the update of the test/README.

I also realized now that basically the --repeat or respectively --repeat-until-failure options from ctest do work as a stress test, which we removed in #3447, therefore the functionality is still available, however now a bit more unified.

[djmitche]

Looks good - let's merge it!

[djmitche]

Nice!