[libc++][test] Adds more generic test macros.

These macros are intended to replace the macros in rapid-cxx-test.h.

Reviewed By: #libc, ldionne

Differential Revision: https://reviews.llvm.org/D142808

libcxx/docs/Contributing.rst

@@ -31,7 +31,7 @@ Pre-commit check list
 Before committing or creating a review, please go through this check-list to make
 sure you don't forget anything:
 
-- Do you have tests for every public class and/or function you're adding or modifying?
+- Do you have :ref:`tests <testing>` for every public class and/or function you're adding or modifying?
 - Did you update the synopsis of the relevant headers?
 - Did you update the relevant files to track implementation status (in ``docs/Status/``)?
 - Did you mark all functions and type declarations with the :ref:`proper visibility macro <visibility-macros>`?

libcxx/docs/TestingLibcxx.rst

@@ -5,6 +5,8 @@ Testing libc++
 .. contents::
   :local:
 
+.. _testing:
+
 Getting Started
 ===============
 
@@ -121,7 +123,7 @@ modifying files on your local machine will also modify what the Docker container
 This is useful for editing source files as you're testing your code in the Docker container.
 
 Writing Tests
--------------
+=============
 
 When writing tests for the libc++ test suite, you should follow a few guidelines.
 This will ensure that your tests can run on a wide variety of hardware and under
@@ -143,6 +145,189 @@ few requirements to the test suite. Here's some stuff you should know:
   necessarily available on all devices we may want to run the tests on (even
   though supporting Python is probably trivial for the build-host).
 
+Structure of the testing related directories
+--------------------------------------------
+
+The tests of libc++ are stored in libc++'s testing related subdirectories:
+
+- ``libcxx/test/support`` This directory contains several helper headers with
+  generic parts for the tests. The most important header is ``test_macros.h``.
+  This file contains configuration information regarding the platform used.
+  This is similar to the ``__config`` file in libc++'s ``include`` directory.
+  Since libc++'s tests are used by other Standard libraries, tests should use
+  the ``TEST_FOO`` macros instead of the ``_LIBCPP_FOO`` macros, which are
+  specific to libc++.
+- ``libcxx/test/std`` This directory contains the tests that validate the library under
+  test conforms to the C++ Standard. The paths and the names of the test match
+  the section names in the C++ Standard. Note that the C++ Standard sometimes
+  reorganises its structure, therefore some tests are at a location based on
+  where they appeared historically in the standard. We try to strike a balance
+  between keeping things at up-to-date locations and unnecessary churn.
+- ``libcxx/test/libcxx`` This directory contains the tests that validate libc++
+  specific behavior and implementation details. For example, libc++ has
+  "wrapped iterators" that perform bounds checks. Since those are specific to
+  libc++ and not mandated by the Standard, tests for those are located under
+  ``libcxx/test/libcxx``. The structure of this directories follows the
+  structure of ``libcxx/test/std``.
+
+Structure of a test
+-------------------
+
+Some platforms where libc++ is tested have requirement on the signature of
+``main`` and require ``main`` to explicitly return a value. Therefore the
+typical ``main`` function should look like:
+
+.. code-block:: cpp
+
+  int main(int, char**) {
+    ...
+    return 0;
+  }
+
+
+The C++ Standard has ``constexpr`` requirements. The typical way to test that,
+is to create a helper ``test`` function that returns a ``bool`` and use the
+following ``main`` function:
+
+.. code-block:: cpp
+
+  constexpr bool test() {
+    ...
+    return true;
+  }
+
+  int main(int, char**) {
+    test()
+    static_assert(test());
+
+    return 0;
+  }
+
+Tests in libc++ mainly use ``assert`` and ``static_assert`` for testing. There
+are a few helper macros and function that can be used to make it easier to
+write common tests.
+
+libcxx/test/support/assert_macros.h
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The header contains several macros with user specified log messages. This is
+useful when a normal assertion failure lacks the information to easily
+understand why the test has failed. This usually happens when the test is in a
+helper function. For example the ``std::format`` tests use a helper function
+for its validation. When the test fails it will give the line in the helper
+function with the condition ``out == expected`` failed. Without knowing what
+the value of ``format string``, ``out`` and ``expected`` are it is not easy to
+understand why the test has failed. By logging these three values the point of
+failure can be found without resorting to a debugger.
+
+Several of these macros are documented to take an ``ARG``. This ``ARG``:
+
+ - if it is a ``const char*`` or ``std::string`` its contents are written to
+   the ``stderr``,
+ - otherwise it must be a callable that is invoked without any additional
+   arguments and is expected to produce useful output to e.g. ``stderr``.
+
+This makes it possible to write additional information when a test fails,
+either by supplying a hard-coded string or generate it at runtime.
+
+TEST_FAIL(ARG)
+^^^^^^^^^^^^^^
+
+This macro is an unconditional failure with a log message ``ARG``. The main
+use-case is to fail when code is reached that should be unreachable.
+
+
+TEST_REQUIRE(CONDITION, ARG)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This macro requires its ``CONDITION`` to evaluate to ``true``. If that fails it
+will fail the test with a log message ``ARG``.
+
+
+TEST_LIBCPP_REQUIRE((CONDITION, ARG)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If the library under test is libc++ it behaves like ``TEST_REQUIRE``, else it
+is a no-op. This makes it possible to test libc++ specific behaviour. For
+example testing whether the ``what()`` of an exception thrown matches libc++'s
+expectations. (Usually the Standard requires certain exceptions to be thrown,
+but not the contents of its ``what()`` message.)
+
+
+TEST_DOES_NOT_THROW(EXPR)
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Validates execution of ``EXPR`` does not throw an exception.
+
+TEST_THROWS_TYPE(TYPE, EXPR)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Validates the execution of ``EXPR`` throws an exception of the type ``TYPE``.
+
+
+TEST_VALIDATE_EXCEPTION(TYPE, PRED, EXPR)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Validates the execution of ``EXPR`` throws an exception of the type ``TYPE``
+which passes validation of ``PRED``. Using this macro makes it easier to write
+tests using exceptions. The code to write a test manually would be:
+
+
+.. code-block:: cpp
+
+  void test_excption([[maybe_unused]] int arg) {
+  #ifndef TEST_HAS_NO_EXCEPTIONS // do nothing when tests are disabled
+    try {
+      foo(arg);
+      assert(false); // validates foo really throws
+    } catch ([[maybe_unused]] const bar& e) {
+      LIBCPP_ASSERT(e.what() == what);
+      return;
+    }
+    assert(false); // validates bar was thrown
+  #endif
+    }
+
+The same test using a macro:
+
+.. code-block:: cpp
+
+  void test_excption([[maybe_unused]] int arg) {
+    TEST_VALIDATE_EXCEPTION(bar,
+                            [](const bar& e) {
+                              LIBCPP_ASSERT(e.what() == what);
+                            },
+                            foo(arg));
+    }
+
+
+libcxx/test/support/concat_macros.h
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This file contains a helper macro ``TEST_WRITE_CONCATENATED`` to lazily
+concatenate its arguments to a ``std::string`` and write it to ``stderr``. When
+the output can't be concatenated a default message will be written to
+``stderr``. This is useful for tests where the arguments use different
+character types like ``char`` and ``wchar_t``, the latter can't simply be
+written to ``stderrr``.
+
+This macro is in a different header as ``assert_macros.h`` since it pulls in
+additional headers.
+
+ .. note: This macro can only be used in test using C++20 or newer. The macro
+          was added at a time where most of lib++'s C++17 support was complete.
+          Since it is not expected to add this to existing tests no effort was
+          taken to make it work in earlier language versions.
+
+
+Additional reading
+------------------
+
+The function ``CxxStandardLibraryTest`` in the file
+``libcxx/utils/libcxx/test/format.py`` has documentation about writing test. It
+explains the difference between the test named  ``foo.pass.cpp`` and named
+``foo.verify.cpp`` are.
+
 Benchmarks
 ==========
 

libcxx/test/std/containers/container.adaptors/container.adaptors.format/format.functions.format.pass.cpp

@@ -35,13 +35,14 @@
 #include "test_format_string.h"
 #include "test_macros.h"
 #include "assert_macros.h"
+#include "concat_macros.h"
 
 auto test = []<class CharT, class... Args>(
                 std::basic_string_view<CharT> expected, test_format_string<CharT, Args...> fmt, Args&&... args) {
   std::basic_string<CharT> out = std::format(fmt, std::forward<Args>(args)...);
-  TEST_REQUIRE(
-      out == expected,
-      test_concat_message("\nFormat string   ", fmt, "\nExpected output ", expected, "\nActual output   ", out, '\n'));
+  TEST_REQUIRE(out == expected,
+               TEST_WRITE_CONCATENATED(
+                   "\nFormat string   ", fmt, "\nExpected output ", expected, "\nActual output   ", out, '\n'));
 };
 
 auto test_exception = []<class CharT, class... Args>(std::string_view, std::basic_string_view<CharT>, Args&&...) {

libcxx/test/std/containers/container.adaptors/container.adaptors.format/format.functions.vformat.pass.cpp

@@ -32,13 +32,14 @@
 #include "format.functions.tests.h"
 #include "test_macros.h"
 #include "assert_macros.h"
+#include "concat_macros.h"
 
 auto test = []<class CharT, class... Args>(
                 std::basic_string_view<CharT> expected, std::basic_string_view<CharT> fmt, Args&&... args) {
   std::basic_string<CharT> out = std::vformat(fmt, std::make_format_args<context_t<CharT>>(args...));
-  TEST_REQUIRE(
-      out == expected,
-      test_concat_message("\nFormat string   ", fmt, "\nExpected output ", expected, "\nActual output   ", out, '\n'));
+  TEST_REQUIRE(out == expected,
+               TEST_WRITE_CONCATENATED(
+                   "\nFormat string   ", fmt, "\nExpected output ", expected, "\nActual output   ", out, '\n'));
 };
 
 auto test_exception =
@@ -49,11 +50,11 @@ auto test_exception =
 #ifndef TEST_HAS_NO_EXCEPTIONS
       try {
         TEST_IGNORE_NODISCARD std::vformat(fmt, std::make_format_args<context_t<CharT>>(args...));
-        TEST_FAIL(test_concat_message("\nFormat string   ", fmt, "\nDidn't throw an exception.\n"));
+        TEST_FAIL(TEST_WRITE_CONCATENATED("\nFormat string   ", fmt, "\nDidn't throw an exception.\n"));
       } catch (const std::format_error& e) {
         TEST_LIBCPP_REQUIRE(
             e.what() == what,
-            test_concat_message(
+            TEST_WRITE_CONCATENATED(
                 "\nFormat string   ", fmt, "\nExpected exception ", what, "\nActual exception   ", e.what(), '\n'));
 
         return;

libcxx/test/std/containers/sequences/vector.bool/vector.bool.fmt/format.functions.format.pass.cpp

@@ -29,13 +29,14 @@
 #include "test_format_string.h"
 #include "test_macros.h"
 #include "assert_macros.h"
+#include "concat_macros.h"
 
 auto test = []<class CharT, class... Args>(
                 std::basic_string_view<CharT> expected, test_format_string<CharT, Args...> fmt, Args&&... args) {
   std::basic_string<CharT> out = std::format(fmt, std::forward<Args>(args)...);
-  TEST_REQUIRE(
-      out == expected,
-      test_concat_message("\nFormat string   ", fmt, "\nExpected output ", expected, "\nActual output   ", out, '\n'));
+  TEST_REQUIRE(out == expected,
+               TEST_WRITE_CONCATENATED(
+                   "\nFormat string   ", fmt, "\nExpected output ", expected, "\nActual output   ", out, '\n'));
 };
 
 auto test_exception = []<class CharT, class... Args>(std::string_view, std::basic_string_view<CharT>, Args&&...) {

libcxx/test/std/containers/sequences/vector.bool/vector.bool.fmt/format.functions.vformat.pass.cpp

@@ -30,13 +30,14 @@
 #include "format.functions.tests.h"
 #include "test_macros.h"
 #include "assert_macros.h"
+#include "concat_macros.h"
 
 auto test = []<class CharT, class... Args>(
                 std::basic_string_view<CharT> expected, std::basic_string_view<CharT> fmt, Args&&... args) {
   std::basic_string<CharT> out = std::vformat(fmt, std::make_format_args<context_t<CharT>>(args...));
-  TEST_REQUIRE(
-      out == expected,
-      test_concat_message("\nFormat string   ", fmt, "\nExpected output ", expected, "\nActual output   ", out, '\n'));
+  TEST_REQUIRE(out == expected,
+               TEST_WRITE_CONCATENATED(
+                   "\nFormat string   ", fmt, "\nExpected output ", expected, "\nActual output   ", out, '\n'));
 };
 
 auto test_exception =
@@ -47,11 +48,11 @@ auto test_exception =
 #ifndef TEST_HAS_NO_EXCEPTIONS
       try {
         TEST_IGNORE_NODISCARD std::vformat(fmt, std::make_format_args<context_t<CharT>>(args...));
-        TEST_FAIL(test_concat_message("\nFormat string   ", fmt, "\nDidn't throw an exception.\n"));
+        TEST_FAIL(TEST_WRITE_CONCATENATED("\nFormat string   ", fmt, "\nDidn't throw an exception.\n"));
       } catch (const std::format_error& e) {
         TEST_LIBCPP_REQUIRE(
             e.what() == what,
-            test_concat_message(
+            TEST_WRITE_CONCATENATED(
                 "\nFormat string   ", fmt, "\nExpected exception ", what, "\nActual exception   ", e.what(), '\n'));
 
         return;

libcxx/test/std/utilities/format/format.functions/escaped_output.ascii.pass.cpp

@@ -26,6 +26,7 @@
 #include "make_string.h"
 #include "test_format_string.h"
 #include "assert_macros.h"
+#include "concat_macros.h"
 
 #ifndef TEST_HAS_NO_LOCALIZATION
 #  include <iostream>
@@ -38,7 +39,7 @@ auto test_format = []<class CharT, class... Args>(
   {
     std::basic_string<CharT> out = std::format(fmt, std::forward<Args>(args)...);
     TEST_REQUIRE(out == expected,
-                 test_concat_message(
+                 TEST_WRITE_CONCATENATED(
                      "\nFormat string   ", fmt.get(), "\nExpected output ", expected, "\nActual output   ", out, '\n'));
   }
 #ifndef TEST_HAS_NO_LOCALIZATION

libcxx/test/std/utilities/format/format.functions/escaped_output.unicode.pass.cpp

@@ -32,6 +32,7 @@
 #include "make_string.h"
 #include "test_format_string.h"
 #include "assert_macros.h"
+#include "concat_macros.h"
 
 #ifndef TEST_HAS_NO_LOCALIZATION
 #  include <iostream>
@@ -44,7 +45,7 @@ auto test_format = []<class CharT, class... Args>(
   {
     std::basic_string<CharT> out = std::format(fmt, std::forward<Args>(args)...);
     TEST_REQUIRE(out == expected,
-                 test_concat_message(
+                 TEST_WRITE_CONCATENATED(
                      "\nFormat string   ", fmt.get(), "\nExpected output ", expected, "\nActual output   ", out, '\n'));
   }
 #ifndef TEST_HAS_NO_LOCALIZATION

libcxx/test/std/utilities/format/format.functions/format.locale.pass.cpp

@@ -28,14 +28,15 @@
 #include "string_literal.h"
 #include "test_format_string.h"
 #include "assert_macros.h"
+#include "concat_macros.h"
 
 auto test =
     []<class CharT, class... Args>(
         std::basic_string_view<CharT> expected, test_format_string<CharT, Args...> fmt, Args&&... args) constexpr {
       std::basic_string<CharT> out = std::format(std::locale(), fmt, std::forward<Args>(args)...);
       TEST_REQUIRE(
           out == expected,
-          test_concat_message(
+          TEST_WRITE_CONCATENATED(
               "\nFormat string   ", fmt.get(), "\nExpected output ", expected, "\nActual output   ", out, '\n'));
     };
 

libcxx/test/std/utilities/format/format.functions/format.pass.cpp

@@ -30,14 +30,15 @@
 #include "string_literal.h"
 #include "test_format_string.h"
 #include "assert_macros.h"
+#include "concat_macros.h"
 
 auto test =
     []<class CharT, class... Args>(
         std::basic_string_view<CharT> expected, test_format_string<CharT, Args...> fmt, Args&&... args) constexpr {
       std::basic_string<CharT> out = std::format(fmt, std::forward<Args>(args)...);
       TEST_REQUIRE(
           out == expected,
-          test_concat_message(
+          TEST_WRITE_CONCATENATED(
               "\nFormat string   ", fmt.get(), "\nExpected output ", expected, "\nActual output   ", out, '\n'));
     };
 

libcxx/test/std/utilities/format/format.functions/format_tests.h

@@ -2673,7 +2673,6 @@ void format_tests(TestFunction check, ExceptionTest check_exception) {
   check_exception("The format string contains an invalid escape sequence", SV("{:}-}"), 42);
 
   check_exception("The format string contains an invalid escape sequence", SV("} "));
-
   check_exception("The arg-id of the format-spec starts with an invalid character", SV("{-"), 42);
   check_exception("Argument index out of bounds", SV("hello {}"));
   check_exception("Argument index out of bounds", SV("hello {0}"));