CMake pulls in wrong libintl on macOS

[JEphron]

OSX 10.13.6
checkout: 6a9f0fc
make can't find my libintl:

~/d/s/f/build ❯❯❯ make
[  0%] Built target CHECK-FISH-BUILD-VERSION-FILE
[  1%] Built target build_fish_pc
[  2%] Built target pofiles_2
[  3%] Built target pofiles_7
[ 82%] Built target fishlib
[ 84%] Built target tinyexpr
[ 85%] Linking CXX executable fish_key_reader
Undefined symbols for architecture x86_64:
  "_libintl_bindtextdomain", referenced from:
      fish_bindtextdomain(char const*, char const*) in libfishlib.a(fallback.cpp.o)
  "_libintl_gettext", referenced from:
      fish_gettext(char const*) in libfishlib.a(fallback.cpp.o)
  "_libintl_textdomain", referenced from:
      fish_textdomain(char const*) in libfishlib.a(fallback.cpp.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [fish_key_reader] Error 1
make[1]: *** [CMakeFiles/fish_key_reader.dir/all] Error 2
make: *** [all] Error 2

I've noticed that CMake is trying to use the libintl from an install of Mono that I have on my system:
(and also the gettext from anaconda, for some reason: although maybe unrelated)

-- ...
-- Found Intl: /Library/Frameworks/Mono.framework/Headers
-- Found Gettext: /anaconda3/bin/msgmerge (found version "0.19.8.1")
-- ...

I'm far from an expert on CMake, so I wasn't sure how to fix this properly. I worked around it by doing a brew link gettext --force, which causes CMake to see libintl at /usr/local/lib/libintl.dylib, but brew warns that this is probably a bad idea and might cause other compilation issues.

So, not sure if this is really a bug report or just my setup being wonky, but I figure it can't hurt to document it here (for posterity).

[mqudsi]

Obvious questions first (sorry): have you tried blowing away the build dir entirely and trying again?

It builds with cmake for me on macOS High Sierra and Mojave.

[JEphron]

Yeah, I've tried that. Same deal.
I've been following the build instructions in the README

mkdir build
cd build
cmake ..
make

Here's the complete output of cmake

-- The C compiler identification is AppleClang 10.0.0.10001044
-- The CXX compiler identification is AppleClang 10.0.0.10001044
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Setting build type to default 'RelWithDebInfo'
-- Found Curses: /usr/lib/libcurses.dylib
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE
-- Looking for backtrace_symbols
-- Looking for backtrace_symbols - found
-- Looking for clock_gettime
-- Looking for clock_gettime - found
-- Looking for ctermid_r
-- Looking for ctermid_r - found
-- Performing Test HAVE_STRUCT_DIRENT_D_TYPE
-- Performing Test HAVE_STRUCT_DIRENT_D_TYPE - Success
-- Looking for dirfd
-- Looking for dirfd - found
-- Looking for C++ include execinfo.h
-- Looking for C++ include execinfo.h - found
-- Looking for flock
-- Looking for flock - found
-- Looking for futimens
-- Looking for futimens - found
-- Looking for futimes
-- Looking for futimes - found
-- Looking for getifaddrs
-- Looking for getifaddrs - found
-- Looking for getpwent
-- Looking for getpwent - found
-- Looking for gettext
-- Looking for gettext - not found
-- Looking for killpg
-- Looking for killpg - found
-- Looking for lrand48_r
-- Looking for lrand48_r - not found
-- Looking for mkostemp
-- Looking for mkostemp - found
-- Looking for include files curses.h, term.h
-- Looking for include files curses.h, term.h - found
-- Looking for C++ include ncurses/term.h
-- Looking for C++ include ncurses/term.h - not found
-- Looking for C++ include siginfo.h
-- Looking for C++ include siginfo.h - not found
-- Looking for C++ include spawn.h
-- Looking for C++ include spawn.h - found
-- Looking for std::wcscasecmp
-- Looking for std::wcscasecmp - not found
-- Looking for std::wcsdup
-- Looking for std::wcsdup - not found
-- Looking for std::wcsncasecmp
-- Looking for std::wcsncasecmp - not found
-- Performing Test HAVE_STRUCT_STAT_ST_CTIME_NSEC
-- Performing Test HAVE_STRUCT_STAT_ST_CTIME_NSEC - Failed
-- Performing Test HAVE_STRUCT_STAT_ST_MTIMESPEC_TV_NSEC
-- Performing Test HAVE_STRUCT_STAT_ST_MTIMESPEC_TV_NSEC - Success
-- Performing Test HAVE_STRUCT_STAT_ST_MTIM_TV_NSEC
-- Performing Test HAVE_STRUCT_STAT_ST_MTIM_TV_NSEC - Failed
-- Looking for sys_errlist
-- Looking for sys_errlist - found
-- Looking for C++ include sys/ioctl.h
-- Looking for C++ include sys/ioctl.h - found
-- Looking for C++ include sys/select.h
-- Looking for C++ include sys/select.h - found
-- Looking for include files sys/types.h, sys/sysctl.h
-- Looking for include files sys/types.h, sys/sysctl.h - found
-- Looking for C++ include termios.h
-- Looking for C++ include termios.h - found
-- Looking for wcscasecmp
-- Looking for wcscasecmp - found
-- Looking for wcsdup
-- Looking for wcsdup - found
-- Looking for wcslcpy
-- Looking for wcslcpy - found
-- Looking for wcsncasecmp
-- Looking for wcsncasecmp - found
-- Looking for wcsndup
-- Looking for wcsndup - not found
-- Looking for _sys_errs
-- Looking for _sys_errs - not found
-- Looking for C++ include sys/types.h
-- Looking for C++ include sys/types.h - found
-- Looking for C++ include stdint.h
-- Looking for C++ include stdint.h - found
-- Looking for C++ include stddef.h
-- Looking for C++ include stddef.h - found
-- Check size of struct winsize
-- Check size of struct winsize - done
-- Looking for TIOCGWINSZ
-- Looking for TIOCGWINSZ - found
-- Check size of wchar_t[8]
-- Check size of wchar_t[8] - done
-- Performing Test TPARM_TAKES_VARARGS
-- Performing Test TPARM_TAKES_VARARGS - Success
-- Performing Test HAVE_STD__MAKE_UNIQUE
-- Performing Test HAVE_STD__MAKE_UNIQUE - Failed
-- Found Intl: /Library/Frameworks/Mono.framework/Headers
-- Found Gettext: /anaconda3/bin/msgmerge (found version "0.19.8.1")
-- Performing Test HAVE__NL_MSG_CAT_CNTR
-- Performing Test HAVE__NL_MSG_CAT_CNTR - Failed
-- Found system PCRE2 library /usr/local/include
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) (Required is at least version "1.8.7")
CMake Deprecation Warning at cmake/Tests.cmake:33 (CMAKE_POLICY):
  The OLD behavior for policy CMP0037 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.
Call Stack (most recent call first):
  CMakeLists.txt:145 (INCLUDE)


-- The following features have been enabled:

 * gettext, translate messages with gettext

-- The following OPTIONAL packages have been found:

 * Intl
 * Gettext

-- The following REQUIRED packages have been found:

 * Curses
 * Threads

-- The following features have been disabled:

 * Documentation, user manual and documentation

-- The following OPTIONAL packages have not been found:

 * Doxygen (required version >= 1.8.7)

-- Configuring done
-- Generating done
-- Build files have been written to: /Users/<name>/devel/sources/fish-shell/build

I added a few logging statements to gettext.cmake to try to see what was going on

...
INCLUDE(FeatureSummary)

OPTION(WITH_GETTEXT "translate messages if gettext is available" ON)
IF(WITH_GETTEXT)
  FIND_PACKAGE(Intl)
  FIND_PACKAGE(Gettext)
  # --- Begin changes
  message (a=${Intl_FOUND}) 
  message (b=${Intl_INCLUDE_DIRS})
  message (c=${Intl_LIBRARY})
  message (d=${Intl_LIBRARIES})
  # --- End changes
  IF(GETTEXT_FOUND)
      SET(HAVE_GETTEXT 1)
      INCLUDE_DIRECTORIES(${Intl_INCLUDE_DIR})
  ENDIF()
ENDIF()
ADD_FEATURE_INFO(gettext GETTEXT_FOUND "translate messages with gettext")
...

relevant output:

a=TRUE
b=/Library/Frameworks/Mono.framework/Headers
c=Intl_LIBRARY-NOTFOUND
d=

Another workaround appears to be cmake .. -DWITH_GETTEXT=OFF

[mqudsi]

Do you have either of LD_LIBRARY_PATH or LIBRARY_PATH exported?

[JEphron]

No, neither are set

[mqudsi]

CMake's heuristic for determining the match for find_library is detailed here (it's pretty easy to follow): https://cmake.org/cmake/help/v3.0/command/find_library.html

Does anything there stand out to you?

[mqudsi]

When I install gettext via homebrew, it says the following:

mqudsi@MacBook ~> brew reinstall gettext
==> Reinstalling gettext 
==> Downloading https://homebrew.bintray.com/bottles/gettext-0.19.8.1.yosemite.b
Already downloaded: /Users/mqudsi/Library/Caches/Homebrew/downloads/f187ebd03fcdacc2945717297f0673beff5a923334ecf75a0206a6faf9959b3f--gettext-0.19.8.1.yosemite.bottle.tar.gz
==> Pouring gettext-0.19.8.1.yosemite.bottle.tar.gz
==> Caveats
gettext is keg-only, which means it was not symlinked into /usr/local,
because macOS provides the BSD gettext library & some software gets confused if both are in the library path.

If you need to have gettext first in your PATH run:
  echo 'set -g fish_user_paths "/usr/local/opt/gettext/bin" $fish_user_paths' >> ~/.config/fish/config.fish

For compilers to find gettext you may need to set:
  set -gx LDFLAGS "-L/usr/local/opt/gettext/lib"
  set -gx CPPFLAGS "-I/usr/local/opt/gettext/include"

==> Summary
🍺  /usr/local/Cellar/gettext/0.19.8.1: 1,934 files, 17.0MB

So by default no lib should be found. The question is why cmake is picking up on these (incorrect) libraries rather than ignoring them altogether.

[JEphron]

I think cmake is pulling them out of $PATH.
I found that removing /bin/anaconda3 from $PATH causes cmake to fail to find gettext.

cmake output

...
-- Could NOT find Gettext (missing: GETTEXT_MSGMERGE_EXECUTABLE GETTEXT_MSGFMT_EXECUTABLE)
...

however, if I then run

set -g fish_user_paths "/usr/local/opt/gettext/bin" $fish_user_paths

cmake finds the correct gettext

I also tried running

  set -gx LDFLAGS "-L/usr/local/opt/gettext/lib"
  set -gx CPPFLAGS "-I/usr/local/opt/gettext/include"

as per brew's suggestion, but even though cmake can see those flags (they show up as env variables that I can print out using message), it seems to ignore them and instead use $PATH

[faho]

I found that removing /bin/anaconda3 from $PATH causes cmake to fail to find gettext.

Ah, that explains it. We're not actually looking for gettext headers here, we're looking for the msgmerge and msgfmt programs, because we call them to operate on our translation files.

Since they are executable programs, cmake finds them not via $LDFLAGS or $CPPFLAGS, but via $PATH. And apparently the first component in your $PATH to have a msgmerge is... anaconda.

I'm not sure what to do about this, though. I don't even know why anaconda would have msgmerge? I don't really get what it is, TBH.

[krader1961]

I don't really get what it is, TBH.

It's a package manager with a strong focus on tools like NumPy and R that are relevant to data science: https://www.anaconda.com/what-is-anaconda/. Although a lot of people like myself use it primarily as a better mechanism for managing multiple python environments than pyenv and the like. So in many my anaconda environments I'll have packages like openssl installed.

If someone has activated a anaconda environment that that includes tools like msgfmt then it's not really any different than if they had installed those tools in /usr/local/bin and included that dir in PATH. Caveat Emptor.

P.S., I normally avoid having any anaconda environment activated when doing things like building fish to avoid such surprises. I really don't think this is the responsibility of fish.

[mqudsi]

But why are we finding something by executable then feeding that to the linker:

Undefined symbols for architecture x86_64:
  "_libintl_bindtextdomain", referenced from:
      fish_bindtextdomain(char const*, char const*) in libfishlib.a(fallback.cpp.o)
  "_libintl_gettext", referenced from:
      fish_gettext(char const*) in libfishlib.a(fallback.cpp.o)
  "_libintl_textdomain", referenced from:
      fish_textdomain(char const*) in libfishlib.a(fallback.cpp.o)

I imagine we need a separate check for libintl vs its bin files.

[JEphron]

I want to stress that I didn't have an Anaconda environment activated when I was building Fish. It's merely installed on my system (I use it as a general-purpose replacement for virtualenvs). AFAIK it adds its bin directory to the path automatically when it's installed.

[mqudsi]

@JEphron noted, but my concern still holds.

[mqudsi]

Ah, that explains it. We're not actually looking for gettext headers here, we're looking for the msgmerge and msgfmt programs, because we call them to operate on our translation files.

This is not quite true. We search separately for libintl and the gettext binaries.
This is the same root problem as with ncurses - CMake has its own (really poor) heuristic for locating libraries to avoid depending on pkg-config.

[zanchey]

I can reproduce this - looks like I've got the Mono libintl, and if I add the brew gettext tools to my path, I get the same problem.

I have spent some time reviewing the current state:

• libintl is not shipped in macOS, but can be picked up by FindPackage(Intl)
• the gettext binaries (msgfmt etc) are not shipped in macOS, but can be picked up by FindPackage(gettext)
• if both libintl and the gettext binaries are detected, the include directories get added to the global include directories
• regardless of the gettext state, the libintl libraries are added to the list of library dependencies of the fishlib target (this is inconsistent with the previous item and probably needs fixing, perhaps through an add_library target)

If you have Mono installed, libintl's headers are picked up by CMake from /Library/Frameworks/Mono.framework, but the libraries are not found in the same prefix (Intl_LIBRARY:FILEPATH=Intl_LIBRARY-NOTFOUND). Unfortunately, this cannot be used as a signal that libintl is not available, because this is exactly the state that glibc is in with libintl compiled into the main library.

This is only exposed as a problem when gettext is in the path, because of the requirement for both gettext and libintl for the include_directory call (as above).

pkg-config would not help here, because it does not ship on macOS.

I think this is a bug in CMake's FindIntl module, which I'll report to them. Someone has a similar problem in EOSIO/eos#1539.

For now, you can work around this either by turning off the use of translation (cmake -DWITH_GETTEXT=0), or enable it by adding an additional library search path (cmake -DCMAKE_LIBRARY_PATH=/Library/Frameworks/Mono.framework/Libraries).

[zanchey]

https://gitlab.kitware.com/cmake/cmake/issues/18921

[ridiculousfish]

Wow that was a journey! Sounds like you helped fish and cmake.