CMake 3.20 breaks 32-bit building on GA CI: finds wrong libraries

[derekbruening]

The clang-format and both Linux x86 jobs suddenly started failing about an hour ago.

Xref #4780 where 32-bit apt packages were messed up.

Clang-format says:

Reading Script: /home/runner/work/dynamorio/dynamorio/suite/runsuite.cmake,automated_ci;64_only;require_format
Detected a CI clang suite: disabling running of tests
CMake Error at /home/runner/work/dynamorio/dynamorio/suite/runsuite.cmake:254 (message):
  FATAL ERROR: clang-format is required but not found!

That may be related to this:
https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu1604-README.md

[Ubuntu] Clang, GCC and Gfortran versions less than 9.x will be removed from images on March, 29

Though it still lists clang 9 as being installed in the base.

For the 32-bit the failures look like a messed-up toolchain finding the wrong libraries:

https://github.com/DynamoRIO/dynamorio/pull/4829/checks?check_run_id=2247626216

2021-04-01T16:07:58.4961762Z [  2%] Linking C executable bin/linux.infinite
2021-04-01T16:07:58.4962874Z /usr/lib/x86_64-linux-gnu/libdl.so: error adding symbols: File in wrong format
2021-04-01T16:07:58.4964490Z suite/tests/CMakeFiles/linux.infinite.dir/build.make:98: recipe for target 'suite/tests/bin/linux.infinite' failed
2021-04-01T16:07:58.4966284Z CMakeFiles/Makefile2:7501: recipe for target 'suite/tests/CMakeFiles/linux.infinite.dir/all' failed
2021-04-01T16:07:58.4967132Z collect2: error: ld returned 1 exit status

2021-04-01T16:07:58.4971965Z [  2%] Linking C executable bin/client.mbr_instrumentation_segment
2021-04-01T16:07:58.4973167Z /usr/lib/x86_64-linux-gnu/libdl.so: error adding symbols: File in wrong format
2021-04-01T16:07:58.4974711Z suite/tests/CMakeFiles/client.mbr_instrumentation_segment.dir/build.make:98: recipe for target 'suite/tests/bin/client.mbr_instrumentation_segment' failed

2021-04-01T16:09:28.5436729Z [ 58%] Linking CXX executable ../bin32/prefetch_analyzer_launcher
2021-04-01T16:09:28.5438665Z /usr/lib/x86_64-linux-gnu/libz.so: error adding symbols: File in wrong format
2021-04-01T16:09:28.5440235Z clients/drcachesim/CMakeFiles/prefetch_analyzer_launcher.dir/build.make:120: recipe for target 'clients/bin32/prefetch_analyzer_launcher' failed
2021-04-01T16:09:28.5441277Z collect2: error: ld returned 1 exit status

This breaks 64-bit as well b/c it has one 32-bit build for cross-arch testing:

2021-04-01T16:11:18.1127287Z 8: [100%] Linking C executable bin/linux.execve32
2021-04-01T16:11:18.1128546Z 8: /usr/lib/x86_64-linux-gnu/libdl.so: error adding symbols: File in wrong format
2021-04-01T16:11:18.1129378Z 8: collect2: error: ld returned 1 exit status

[derekbruening]

It does seem likely that these problems are coming from the announcement cited above as the VM's were deployed 1 hour ago: actions/runner-images#2950 (comment)

[derekbruening]

Explicitly installing clang-format does fix the missing clang-format error (though it's supposed to already be there in the base VM). Have not yet found a workaround for the messed-up 32-bit toolchain; installing g++-8 g++-8-multilib fails with its own errors (PR #4831).

[derekbruening]

Based on actions/runner-images#2950 (comment) and the results in the PR, the 32-bit-toolchain problem is from a cmake upgrade to 3.20. For now we will downgrade to 3.19 to get things green, while we try to figure out what the underlying issue is.

[derekbruening]

Renamed the title to focus on the part that's left: figuring out why CMake 3.20 results in trying to use 64-bit libraries for 32-bit targets. Is it purely a CMake problem, or does it involve GA CI VM setup? First step would be to try to reproduce outside of GA of course. Now that we have the CI green by using 3.19 this is less urgent but we will all start hitting it as our own machines move to 3.20 at some point.

[derekbruening]

Not seeing anything in the changelog that seems relevant: https://cmake.org/cmake/help/latest/release/3.20.html

[derekbruening]

I downloaded https://github.com/Kitware/CMake/releases/download/v3.20.0/cmake-3.20.0-linux-x86_64.tar.gz and ran:

$ CFLAGS=-m32 CXXFLAGS=-m32 ~/extsw/cmake/cmake-3.20.0-linux-x86_64/bin/cmake -GNinja -DBUILD_TESTS=ON -DDEBUG=ON ../src && ninja

And it worked fine. This is a Debian-ish distro with g++-multilib installed.