Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[libc] remove MPFR and related tests in full build #87693

Merged
merged 3 commits into from
Apr 8, 2024

Conversation

SchrodingerZhu
Copy link
Contributor

In full build mode, the fuzzing tests fail to build. This PR works around the annoying behavior that MPFR pulls in system C++ headers blindly (see errors below). The fix is rather ugly but it seems to work. Let me know if you have better ideas.

[2/4] Building CXX object projects/libc/fuzzing/stdio/CMakeFiles/libc.fuzzing.stdio.printf_float_conv_fuzz.dir/printf_float_conv_fuzz.cpp.o
FAILED: projects/libc/fuzzing/stdio/CMakeFiles/libc.fuzzing.stdio.printf_float_conv_fuzz.dir/printf_float_conv_fuzz.cpp.o 
/usr/bin/clang++ -DLIBC_NAMESPACE=__llvm_libc_19_0_0_git -I/home/schrodingerzy/Documents/llvm/llvm-project/build/projects/libc/fuzzing/stdio -I/home/schrodingerzy/Documents/llvm/llvm-project/libc/fuzzing/stdio -I/home/schrodingerzy/Documents/llvm/llvm-project/libc -isystem /home/schrodingerzy/Documents/llvm/llvm-project/build/projects/libc/include -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -fsanitize=fuzzer -O2 -g -DNDEBUG -std=c++17 -MD -MT projects/libc/fuzzing/stdio/CMakeFiles/libc.fuzzing.stdio.printf_float_conv_fuzz.dir/printf_float_conv_fuzz.cpp.o -MF projects/libc/fuzzing/stdio/CMakeFiles/libc.fuzzing.stdio.printf_float_conv_fuzz.dir/printf_float_conv_fuzz.cpp.o.d -o projects/libc/fuzzing/stdio/CMakeFiles/libc.fuzzing.stdio.printf_float_conv_fuzz.dir/printf_float_conv_fuzz.cpp.o -c /home/schrodingerzy/Documents/llvm/llvm-project/libc/fuzzing/stdio/printf_float_conv_fuzz.cpp
In file included from /home/schrodingerzy/Documents/llvm/llvm-project/libc/fuzzing/stdio/printf_float_conv_fuzz.cpp:19:
In file included from /home/schrodingerzy/Documents/llvm/llvm-project/libc/utils/MPFRWrapper/mpfr_inc.h:21:
In file included from /usr/include/mpfr.h:53:
In file included from /usr/include/gmp.h:35:
In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/iosfwd:38:
In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/bits/requires_hosted.h:31:
In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/x86_64-pc-linux-gnu/bits/c++config.h:679:
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/x86_64-pc-linux-gnu/bits/os_defines.h:44:5: error: function-like macro '__GLIBC_PREREQ' is not defined
   44 | #if __GLIBC_PREREQ(2,15) && defined(_GNU_SOURCE)
      |     ^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/x86_64-pc-linux-gnu/bits/os_defines.h:55:5: error: function-like macro '__GLIBC_PREREQ' is not defined
   55 | #if __GLIBC_PREREQ(2, 26) \
      |     ^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/x86_64-pc-linux-gnu/bits/os_defines.h:66:6: error: function-like macro '__GLIBC_PREREQ' is not defined
   66 | # if __GLIBC_PREREQ(2, 27)
      |      ^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/x86_64-pc-linux-gnu/bits/os_defines.h:78:6: error: function-like macro '__GLIBC_PREREQ' is not defined
   78 | # if __GLIBC_PREREQ(2, 34)
      |      ^
In file included from /home/schrodingerzy/Documents/llvm/llvm-project/libc/fuzzing/stdio/printf_float_conv_fuzz.cpp:19:
In file included from /home/schrodingerzy/Documents/llvm/llvm-project/libc/utils/MPFRWrapper/mpfr_inc.h:21:
In file included from /usr/include/mpfr.h:53:
In file included from /usr/include/gmp.h:35:
In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/iosfwd:42:
In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/bits/postypes.h:40:
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:64:11: error: no member named 'mbstate_t' in the global namespace
   64 |   using ::mbstate_t;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:143:11: error: no member named 'btowc' in the global namespace
  143 |   using ::btowc;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:144:11: error: no member named 'fgetwc' in the global namespace
  144 |   using ::fgetwc;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:145:11: error: no member named 'fgetws' in the global namespace
  145 |   using ::fgetws;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:146:11: error: no member named 'fputwc' in the global namespace
  146 |   using ::fputwc;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:147:11: error: no member named 'fputws' in the global namespace
  147 |   using ::fputws;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:148:11: error: no member named 'fwide' in the global namespace
  148 |   using ::fwide;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:149:11: error: no member named 'fwprintf' in the global namespace
  149 |   using ::fwprintf;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:150:11: error: no member named 'fwscanf' in the global namespace
  150 |   using ::fwscanf;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:151:11: error: no member named 'getwc' in the global namespace
  151 |   using ::getwc;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:152:11: error: no member named 'getwchar' in the global namespace
  152 |   using ::getwchar;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:153:11: error: no member named 'mbrlen' in the global namespace
  153 |   using ::mbrlen;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:154:11: error: no member named 'mbrtowc' in the global namespace
  154 |   using ::mbrtowc;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:155:11: error: no member named 'mbsinit' in the global namespace
  155 |   using ::mbsinit;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:156:11: error: no member named 'mbsrtowcs' in the global namespace
  156 |   using ::mbsrtowcs;
      |         ~~^
fatal error: too many errors emitted, stopping now [-ferror-limit=]

@llvmbot
Copy link
Collaborator

llvmbot commented Apr 4, 2024

@llvm/pr-subscribers-libc

Author: Schrodinger ZHU Yifan (SchrodingerZhu)

Changes

In full build mode, the fuzzing tests fail to build. This PR works around the annoying behavior that MPFR pulls in system C++ headers blindly (see errors below). The fix is rather ugly but it seems to work. Let me know if you have better ideas.

[2/4] Building CXX object projects/libc/fuzzing/stdio/CMakeFiles/libc.fuzzing.stdio.printf_float_conv_fuzz.dir/printf_float_conv_fuzz.cpp.o
FAILED: projects/libc/fuzzing/stdio/CMakeFiles/libc.fuzzing.stdio.printf_float_conv_fuzz.dir/printf_float_conv_fuzz.cpp.o 
/usr/bin/clang++ -DLIBC_NAMESPACE=__llvm_libc_19_0_0_git -I/home/schrodingerzy/Documents/llvm/llvm-project/build/projects/libc/fuzzing/stdio -I/home/schrodingerzy/Documents/llvm/llvm-project/libc/fuzzing/stdio -I/home/schrodingerzy/Documents/llvm/llvm-project/libc -isystem /home/schrodingerzy/Documents/llvm/llvm-project/build/projects/libc/include -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -fsanitize=fuzzer -O2 -g -DNDEBUG -std=c++17 -MD -MT projects/libc/fuzzing/stdio/CMakeFiles/libc.fuzzing.stdio.printf_float_conv_fuzz.dir/printf_float_conv_fuzz.cpp.o -MF projects/libc/fuzzing/stdio/CMakeFiles/libc.fuzzing.stdio.printf_float_conv_fuzz.dir/printf_float_conv_fuzz.cpp.o.d -o projects/libc/fuzzing/stdio/CMakeFiles/libc.fuzzing.stdio.printf_float_conv_fuzz.dir/printf_float_conv_fuzz.cpp.o -c /home/schrodingerzy/Documents/llvm/llvm-project/libc/fuzzing/stdio/printf_float_conv_fuzz.cpp
In file included from /home/schrodingerzy/Documents/llvm/llvm-project/libc/fuzzing/stdio/printf_float_conv_fuzz.cpp:19:
In file included from /home/schrodingerzy/Documents/llvm/llvm-project/libc/utils/MPFRWrapper/mpfr_inc.h:21:
In file included from /usr/include/mpfr.h:53:
In file included from /usr/include/gmp.h:35:
In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/iosfwd:38:
In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/bits/requires_hosted.h:31:
In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/x86_64-pc-linux-gnu/bits/c++config.h:679:
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/x86_64-pc-linux-gnu/bits/os_defines.h:44:5: error: function-like macro '__GLIBC_PREREQ' is not defined
   44 | #if __GLIBC_PREREQ(2,15) && defined(_GNU_SOURCE)
      |     ^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/x86_64-pc-linux-gnu/bits/os_defines.h:55:5: error: function-like macro '__GLIBC_PREREQ' is not defined
   55 | #if __GLIBC_PREREQ(2, 26) \
      |     ^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/x86_64-pc-linux-gnu/bits/os_defines.h:66:6: error: function-like macro '__GLIBC_PREREQ' is not defined
   66 | # if __GLIBC_PREREQ(2, 27)
      |      ^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/x86_64-pc-linux-gnu/bits/os_defines.h:78:6: error: function-like macro '__GLIBC_PREREQ' is not defined
   78 | # if __GLIBC_PREREQ(2, 34)
      |      ^
In file included from /home/schrodingerzy/Documents/llvm/llvm-project/libc/fuzzing/stdio/printf_float_conv_fuzz.cpp:19:
In file included from /home/schrodingerzy/Documents/llvm/llvm-project/libc/utils/MPFRWrapper/mpfr_inc.h:21:
In file included from /usr/include/mpfr.h:53:
In file included from /usr/include/gmp.h:35:
In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/iosfwd:42:
In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/bits/postypes.h:40:
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:64:11: error: no member named 'mbstate_t' in the global namespace
   64 |   using ::mbstate_t;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:143:11: error: no member named 'btowc' in the global namespace
  143 |   using ::btowc;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:144:11: error: no member named 'fgetwc' in the global namespace
  144 |   using ::fgetwc;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:145:11: error: no member named 'fgetws' in the global namespace
  145 |   using ::fgetws;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:146:11: error: no member named 'fputwc' in the global namespace
  146 |   using ::fputwc;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:147:11: error: no member named 'fputws' in the global namespace
  147 |   using ::fputws;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:148:11: error: no member named 'fwide' in the global namespace
  148 |   using ::fwide;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:149:11: error: no member named 'fwprintf' in the global namespace
  149 |   using ::fwprintf;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:150:11: error: no member named 'fwscanf' in the global namespace
  150 |   using ::fwscanf;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:151:11: error: no member named 'getwc' in the global namespace
  151 |   using ::getwc;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:152:11: error: no member named 'getwchar' in the global namespace
  152 |   using ::getwchar;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:153:11: error: no member named 'mbrlen' in the global namespace
  153 |   using ::mbrlen;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:154:11: error: no member named 'mbrtowc' in the global namespace
  154 |   using ::mbrtowc;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:155:11: error: no member named 'mbsinit' in the global namespace
  155 |   using ::mbsinit;
      |         ~~^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/../../../../include/c++/13.2.1/cwchar:156:11: error: no member named 'mbsrtowcs' in the global namespace
  156 |   using ::mbsrtowcs;
      |         ~~^
fatal error: too many errors emitted, stopping now [-ferror-limit=]

Full diff: https://github.com/llvm/llvm-project/pull/87693.diff

1 Files Affected:

  • (modified) libc/utils/MPFRWrapper/mpfr_inc.h (+14)
diff --git a/libc/utils/MPFRWrapper/mpfr_inc.h b/libc/utils/MPFRWrapper/mpfr_inc.h
index 58fa7b25a9f210..0d8afbed39859f 100644
--- a/libc/utils/MPFRWrapper/mpfr_inc.h
+++ b/libc/utils/MPFRWrapper/mpfr_inc.h
@@ -17,7 +17,21 @@
 // MPFR header can be included in manner allowed in that repo.
 #include "CustomMPFRIncluder.h"
 #else
+
+extern "C" {
+#pragma push_macro("__cplusplus")
+#if defined(__clang__)
+#pragma clang diagnostic push
+#pragma clang diagnostic ignored "-Wbuiltin-macro-redefined"
+#endif
+#undef __cplusplus
 #include <mpfr.h>
+#if defined(__clang__)
+#pragma clang diagnostic pop
+#endif
+#pragma pop_macro("__cplusplus")
+}
+
 #endif
 
 #endif // LLVM_LIBC_UTILS_MPFRWRAPPER_MPFR_INC_H

@SchrodingerZhu
Copy link
Contributor Author

SchrodingerZhu commented Apr 4, 2024

Should we have a "LEAN_AND_MEAN" version of MPFR headers that define only the things we needed?

@nickdesaulniers
Copy link
Member

In full build mode, the fuzzing tests fail to build.

In full build mode, if anything EVER pulls in system headers other than strictly kernel headers:

  1. stop
  2. do not pass go
  3. do not collect $200

I'm beginning to think that, slightly orthogonal to the refactoring of headers (#87598, #87017), we would be better served by REQUIRING a --sysroot for full build mode (especially for the tests). Maybe there's a way to simplify construction of the sysroot to minimize changes to buildbots (or have a default implicit one, but use --sysroot no matter what), but if we plan to include things like the compiler resource headers, which frequently use #include_next, we should really be using a sysroot in order to have strong guarantees that system headers (other than kernel headers) don't get included. A sysroot is a hermetic way to do so that further simplifies cross compiling.

But if fuzzing NEEDS MPFR to compare results, then perhaps we don't allow fuzzing in full build mode. Or in full build mode, we don't invoke/use MPFR?

Copy link
Contributor

@lntue lntue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't think of a better way at the moment.

@michaelrj-google
Copy link
Contributor

the fuzzing tests aren't supported in fullbuild mode right now. If we really want to support fuzzing in fullbuild then this seems like the best way to do it.

@nickdesaulniers
Copy link
Member

If we want to support fuzzing against full build mode, then we should rebuild MPFR against our libc rather than the system libc. Otherwise libc/fuzzing/ is in some twighlight zone half way between full build and overlay mode. I don't want to support that.

@lntue
Copy link
Contributor

lntue commented Apr 4, 2024

If we want to support fuzzing against full build mode, then we should rebuild MPFR against our libc rather than the system libc. Otherwise libc/fuzzing/ is in some twighlight zone half way between full build and overlay mode. I don't want to support that.

Now that I think about it, probably we should disable building MPFR in full build mode until MPFR could be built with our libc, especially because of this: https://github.com/llvm/llvm-project/blob/main/libc/utils/MPFRWrapper/CMakeLists.txt#L12

@SchrodingerZhu SchrodingerZhu changed the title [libc] avoid c++ includes from MPFR [libc] remove MPFR and related tests in full build Apr 7, 2024
@SchrodingerZhu
Copy link
Contributor Author

I have updated this disable MPFR tests in full build.

@SchrodingerZhu SchrodingerZhu merged commit 54c24ec into llvm:main Apr 8, 2024
3 of 4 checks passed
@SchrodingerZhu SchrodingerZhu deleted the libc/fuzzing branch April 8, 2024 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants