Failure on cmake test "TestEndianess"

[cmundi]

I've built llvm (including clang) and have run the emscripten tutorials but this is my first attempt to use emscripten on an existing project.

My project uses CMake and tests for endianess using /usr/share/cmake-2.8/Modules/TestEndianess.c.in

Natively, cmake works when I'm using the gcc toolchain, but when I switch to

emconfigure.py cmake

then I get a failure on the endianess test. I provide some context leading up to it in case it is important.

Performing Test SEMUN_DEFINED
Performing Test SEMUN_DEFINED - Failed
Performing Test SEMOP_NO_REFERENCE
Performing Test SEMOP_NO_REFERENCE - Failed
Performing Test SEMCTL_NO_REFERENCE
Performing Test SEMCTL_NO_REFERENCE - Failed
Performing Test DEF_IOS_OK
Performing Test DEF_IOS_OK - Failed
Performing Test OSTREAM_FORM_OK
Performing Test OSTREAM_FORM_OK - Failed
Check if the system is big endian
Searching 16 bit integer
Check size of unsigned short
Check size of unsigned short - failed
Check size of unsigned int
Check size of unsigned int - failed
Check size of unsigned long
Check size of unsigned long - failed
CMake Error at /usr/share/cmake-2.8/Modules/TestBigEndian.cmake:44 (MESSAGE):
no suitable type found
Call Stack (most recent call first):
CMakeLists.txt:637 (TEST_BIG_ENDIAN)

Should I be looking for an issue with LLVM or an issue with emscripten? In either case, any suggestions will be appreciated!

[cmundi]

I forgot to mention... my build platform is Linux x86_64. So 64-bit little endian.

[cmundi]

Also, the failures on "Performing Test..." happen with the native (gcc) toolchain. That's not surprising. The surprise is just for the endianess test. The code for the test is

/* A 16 bit integer is required. */
typedef @CMAKE_16BIT_TYPE@ cmakeint16;

/* On a little endian machine, these 16bit ints will give "THIS IS LITTLE ENDIAN."
   On a big endian machine the characters will be exchanged pairwise. */
const cmakeint16 info_little[] =  {0x4854, 0x5349, 0x4920, 0x2053, 0x494c, 0x5454, 0x454c, 0x4520, 0x444e, 0x4149, 0x2e4e, 0x0000};

/* on a big endian machine, these 16bit ints will give "THIS IS BIG ENDIAN."
   On a little endian machine the characters will be exchanged pairwise. */
const cmakeint16 info_big[] =     {0x5448, 0x4953, 0x2049, 0x5320, 0x4249, 0x4720, 0x454e, 0x4449, 0x414e, 0x2e2e, 0x0000};

#ifdef __CLASSIC_C__
int main(argc, argv) int argc; char *argv[];
#else
int main(int argc, char *argv[])
#endif
{
  int require = 0;
  require += info_little[argc];
  require += info_big[argc];
  (void)argv;
  return require;
}

[kripken]

Looks like the test uses the program return value as output? I'm not sure if node.js lets us return that properly. Worth trying to do a printf there instead, and seeing if that is identical between C and JS builds.

[cmundi]

Alon,

Thank you for your insight. I will explore this and report with the next
couple days.

Carlos
On Apr 19, 2012 1:48 PM, "Alon Zakai" <
reply@reply.github.com>
wrote:

Looks like the test uses the program return value as output? I'm not sure
if node.js lets us return that properly. Worth trying to do a printf there
instead, and seeing if that is identical between C and JS builds.

---

Reply to this email directly or view it on GitHub:
#400 (comment)

[cmundi]

Ok, I think I have found the problem but not the root cause.

The problem is not in emscripten. But it definitely prevents building some projects with emscripten.

First, the problem is not the test code I posted above. The cmake does fail during the endianess check, but the failure occurs before that code is even built. In fact, as we will see, the failure suggests a gross misconfiguration of Cmake 2.8.5, which I have verified on two separate installations of Ubuntu 11.10. Whether the blame belongs with me, with Canonical, with Debian or with Cmake is not something I have determined. Read on.

I instrumented the /usr/share/cmake-2.8/Modules/TestBigEndian.cmake and found that the test was failing early and that the C code above was not getting compiled, much less run! The early failure is in the attempt to identify an integer type with a 16-bit representation, checking unsigned short, unsigned int and unsigned long. (Remember, we're deliberately trying to accommodate platforms besides x86.) Remarkably, all three tests fail!

Let that soak in for a moment. Something is really wrong. Let's hope it's a typo. No such luck.

Drilling deeper, we see that TestBigEndian.cmake uses /usr/share/cmake-2.8/Modules/CheckSizeType.cmake to make and run the tests for each of those three integer types, one at a time. Guess what? The try_compile clause fails for all three! The sources are there. You can even watch cmake copy them into the temporary build directory. But all three builds fail. Fail!

Well, we can't expect to check endianess if we can't get the compiler to build code... This is seriously not good.

By this time you should be asking, "can cmake build the project with the gcc toolchain?" The answer is yes!

The next question should be, "can cmake build the project with clang/clang++ directly, without emscripten?"
I have not tried yet.

I speculate that building with the gcc toolchain -- or at least without emscript -- results in some sizeof() results either being cached or -- more likely -- defined in platform headers so that the CheckTypeSize.cmake never gets invoked by cmake.

But how and why is the try_compile failing? I'm sorry, I'm just not enough of a build system junkie to run that one to ground. Ok, I am that hardcore, but I'm also behind schedule and down to my last six-pack of Red Bull.

So what did I do? I selfishly wrote

SET(CMAKE_16BIT_TYPE "unsigned short")

in /usr/share/cmake-2.8/Modules/TestBigEndian.cmake and got on with my build so that I can go home and sleep tonight. :)

Hopefully someone who knows cmake will be able to help us out. I did find another emscripten issue #345 which I bet is the same problem. And Google turned up a few discussions unrelated to emscripten which seem to corroborate my observations.

[kripken]

Can you perhaps paste/gist the source of one of the tests that fails?

[cmundi]

Yes. I will do that tomorrow, because I am exhausted and no longer trust myself without sleep.

Meanwhile I have verified that cmake works just fine if I specify the LLVM toolchain directly to cmake, i.e. I do not use emscripten. So it appears -- and I stress that this is preliminary -- that builds with the gcc and llvm toolchain are possible but somehow emscripten is missing some small thing which is preventing cmake from configuring smoothly. I really suspect gcc and llvm are relying on some side-effect but I have not proved this.

[cmundi]

Ok, one more update! If I run cmake as cmake-gui (which I usually do) then if I also use emscripten like this

$EMSCRIPTEN_HOME/emconfigure cmake-gui

** and **

when I (always!) start with a fresh cmake cache, I select "specify native tools" instead of "use native default tools" then evenything works.

Note that I had been using "specify native tools" when I was having problems. The fact that emscripten "convinced" cmake to use clang/clang++ for the baseline compiler tests lulled me into a false sense of safety. Plus, it always worked on other projects. Somehow, this compiler selection can be "forgotten" in more complex builds, like the one I'm doing now. This is starting to make sense. The try_compile failed, because... the compiler wasn't working! Telling cmake explciitly to use clang and clang++ solved the problem. Of course I'm not sure what if anything emconfigure really accomplishes in this case, but a build is a build!

I will try to post a useful snippet of code tomorrow.

[cmundi]

Here you go Alon: https://gist.github.com/2430230

This gist is enough to demonstrate the issue I am experiencing with emscripten + cmake.

The TestBigEndian module fails because the CheckTypeSize module fails. I chose this slightly more complex than strictly necessary example to provide an indication of the scope of the issue.

Thanks for your interest!

[kripken]

That CMakeLists.txt file works for me with emconfigure. So the difference must be something with how CMake is set up on our systems. I know very little CMake though so I don't have any guesses. I am on CMake 2.8.5 if that helps.

[cmundi]

That is very interesting.

I also have CMake 2.8.5 (via 2.8.5-1ubuntu1) which I uninstalled & reinstalled just to be safe. So I agree that this sounds like a configuration issue.

Since "cmake ." works for me and "emconfigure cmake ." fails for me -- both in the same shell session -- I will check my ~/.emscripten file. Are there any other configuration files for emscripten which I should check?

Thanks!

[cmundi]

Strange. I checked my ~/.emscripten file. The only things which are not default are the paths I set for EMSCRIPTEN_ROOT and LLVM_ROOT and SPIDERMONKEY_ENGINE. Surely none of those should affect cmake.

Here's a summary for anyone who finds this and wonders if it applies to them:

My platform is...
64-bit Ubuntu 11.10
CMake 2.8.5 (2.8.5-1ubuntu1)
gcc 4.6.1 (4:4.6.1-2ubuntu5)
LLVM & Clang 3.0 built in Release

I mention the compilers because cmake runs them before invoking CheckTypeSize (e.g. via TestBigEndian), so in principle the mere act of running the compiler could cause cmake to update its cache and thereby affect the results.

Again, I stress that you will see the problem only if your project builds with cmake and CheckTypeSize is (directly or indirectly) called out by a CMakeList.txt file and you wrap cmake with emconfigure and you do NOT explicitly pass clang/clang++ as the native compiler options to cmake. If you do not meet all four of these conditions, then you will not see the problem.

And apparently some systems don't show the problem at all. So there is at least one more variable.

It seems I had to push hard and be somewhat unlucky to see this problem at all, but I have it on both of my Ubuntu boxes (both x64).

[cmundi]

UPDATE: I have narrowed the problem down a little further.

Refer to my gist above. Running 'emconfigure cmake .' writes a temp file to override the default compiler settings -- as expected. See class Building in tools/shared.py.

Here is an example of such a temp file created on my system:

# the name of the target operating system
SET(CMAKE_SYSTEM_NAME Linux)

# which C and C++ compiler to use
SET(CMAKE_C_COMPILER   /home/cmundi/Projects/emscripten/emscripten//emcc)
SET(CMAKE_CXX_COMPILER /home/cmundi/Projects/emscripten/emscripten//em++)
SET(CMAKE_AR           /home/cmundi/Projects/emscripten/emscripten//emar)
SET(CMAKE_RANLIB       /home/cmundi/Projects/emscripten/emscripten//emranlib)
SET(CMAKE_C_FLAGS      )
SET(CMAKE_CXX_FLAGS    )

# here is the target environment located
SET(CMAKE_FIND_ROOT_PATH  /home/cmundi/Projects/emscripten/emscripten//system/include )

# adjust the default behaviour of the FIND_XXX() commands:
# search headers and libraries in the target environment, search
# programs in the host environment
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE BOTH)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

Unfortunately -- as I reported above -- on my system the configure fails on TestBigEndian (because CheckTypeSize fails to build). But if just for fun we comment out just the redefinition of the c compiler in the temp file thus

#SET(CMAKE_C_COMPILER   /home/cmundi/Projects/emscripten/emscripten//emcc)

leaving all the otehr lines alone and then run manually as

cmake -DCMAKE_TOOLCHAIN_FILE=/tmp/tmphvfCoz.txt .     <== that's a typical tempfile generated by emconfigure

then the configure passes. TestBigEndian runs as it should.

Now note that uncommenting the line we just commented and commenting out all the CMAKE_FIND_ROOT* definitions, thus

# the name of the target operating system
SET(CMAKE_SYSTEM_NAME Linux)

# which C and C++ compiler to use
SET(CMAKE_C_COMPILER   /home/kyle/Projects/emscripten/emscripten//emcc)
SET(CMAKE_CXX_COMPILER /home/kyle/Projects/emscripten/emscripten//em++)
SET(CMAKE_AR           /home/kyle/Projects/emscripten/emscripten//emar)
SET(CMAKE_RANLIB       /home/kyle/Projects/emscripten/emscripten//emranlib)
SET(CMAKE_C_FLAGS      )
SET(CMAKE_CXX_FLAGS    )

# here is the target environment located
#SET(CMAKE_FIND_ROOT_PATH  /home/kyle/Projects/emscripten/emscripten//system/include )

# adjust the default behaviour of the FIND_XXX() commands:
# search headers and libraries in the target environment, search
# programs in the host environment
#set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
#set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
#set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE BOTH)
#set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

results in failure again when we run

cmake -DCMAKE_TOOLCHAIN_FILE=/tmp/tmphvfCoz.txt .     <== that's a typical tempfile generated by emconfigure

Of course, this doesn't solve the problem. We obviously still want to run emcc and not gcc. But what this shows is that -- on my 64-bit Ubuntu 11.10 boxes (both of them) with fresh installs of cmake 2.8.5 and gcc 4.6.1, substituting emcc for gcc causes the config to fail, regardless of the CMAKE_FIND_ROOT* definitions.

But we have narrowed the issue to the emcc script. The next step of course would be to dig into the emcc script and see what it is changing in the environment -- irrespective of what tools/shared.py => Building.handle_CMake_toolchain() puts in the tempfile -- which is causing compilation of some (not all!) C sources to fail under cmake.

I doubt I'll get around to poking through emcc. It's a lot bigger than emconfigure. :) I'm just going to use the workaround I found above, as long as it keeps working.

[jeromewu]

Just for people who might come here from googling solution for TestBigEndian issue.

For a newer version of emscripten, there is a cmake/Modules to fix the issue, all you have to do is adding following line to your CMakeLists.txt:

# It will work when using docker image trzeci/emscripten:sdk-tag-1.38.16-64bit
set(CMAKE_MODULE_PATH "${CMAKE_MODULE_PATH};/emsdk_portable/sdk/cmake/Modules")

This will fix the issue without hard code anything.