[Bug] configure failing on Mac with bogus missing zlib message

[cmbarton]

Describe the bug
A recent change in GRASS master caused it to report being unable to find zlib, even though zlib and its includes are present in the path for compiling. This happens when I'm creating a Mac binary in an environment created with Anaconda. It is not clear what is really missing or needs to have a path set in configure.

A workaround compiles by adding the following before configure runs

export CONDA_BUILD_SYSROOT=/Users/cmbarton/SDKs/MacOSX10.14.sdk

But this causes problems with compiling GRASS extensions later. So I need to know what 7.9 is now looking for (that it is not looking for in 7.8.4) that is causing this problem.

Here is the last part of the terminal output before configure bombs.

checking for MacOSX architectures...  -arch x86_64
checking for MacOSX SDK... checking for /Users/cmbarton/SDKs/MacOSX10.14.sdk/SDKSettings.plist... yes
checking how to build libraries... shared
checking for additional include dirs... /Applications/GRASS-7.9.app/Contents/Resources/include
checking for additional library dirs... /Applications/GRASS-7.9.app/Contents/Resources/lib
checking for a BSD compatible install... /usr/bin/install -c
checking for flex... flex
checking for yywrap in -lfl... no
checking for bison... bison -y
checking for ranlib... ranlib
checking for ar... ar
checking for env... env
checking for perl... /usr/bin/perl
checking how to run the C preprocessor... /lib/cpp
checking for ANSI C header files... no
checking for limits.h... no
checking for termio.h... no
checking for termios.h... no
checking for unistd.h... no
checking for values.h... no
checking for f2c.h... no
checking for g2c.h... no
checking for sys/ioctl.h... no
checking for sys/mtio.h... no
checking for sys/resource.h... no
checking for sys/time.h... no
checking for sys/timeb.h... no
checking for sys/types.h... no
checking for sys/utsname.h... no
checking for libintl.h... no
checking for iconv.h... no
checking for langinfo.h... no
checking whether time.h and sys/time.h may both be included... yes
checking for off_t... no
checking for uid_t in sys/types.h... no
checking return type of signal handlers... int
checking for Cygwin environment... no
checking for ftime... yes
checking for gethostname... yes
checking for gettimeofday... yes
checking for lseek... yes
checking for nice... yes
checking for time... yes
checking for uname... yes
checking for seteuid... yes
checking for setpriority... yes
checking for setreuid... yes
checking for setruid... yes
checking for drand48... yes
checking for putenv... yes
checking for setenv... yes
checking for nanosleep... yes
checking whether setpgrp takes no argument... yes
checking for long long int... yes
checking for int64_t... no
checking for W11... no
checking for X... no
checking for library containing cuserid... no
checking for asprintf... yes
checking for atan... yes
checking for dlsym... yes
checking for iconv... no
checking for iconv in -liconv... no
checking for iconv in -lgiconv... no
checking for libiconv... no
checking for libiconv in -liconv... yes
checking for socket... yes
checking for location of zlib includes... 
checking for zlib.h... no

I'm attaching also the configure log and my configure script.
config.log
configure79.sh.zip

System description (please complete the following information):

• Operating System: Mac OSX 10.15.6

• GRASS GIS version 7.9
  version=7.9.dev
  date=2020
  revision=64527a73f
  build_date=2020-08-05
  build_platform=x86_64-apple-darwin19.6.0
  build_off_t_size=8

[nilason]

Setting the CONDA_BUILD_SYSROOT is not a workaround, it is needed by conda, see Anaconda compiler tools. Missing that causes configure failure. The fact that that it exits at zlib is just a coincidence and not relevant.

The use of conda for a prepackaged binary for distribution is a quite particular case, which leads to challenges with compiling extensions. Here's what happens: you configure and build on a machine with a particular SDK /Users/cmbarton/SDKs/MacOSX10.14.sdk, this configure setting will end up in include/Make/Platform.make. The Platform.make is later used for compiling the C-based extensions on a machine without this SDK (at that location). This is NOT a GRASS issue, but a logical consequence of using conda this way.

This may be circumvented using CONDA_BUILD_SYSROOT=$(xcrun --show-sdk-path) which will use the SDK of present OS version (on building machine). But this will, in turn, require Xcode for any user who want to compile an extension.

Brief test of removing sysroot from Platform.make (after GRASS build and installation, but before distribution) with:

#!/bin/bash

FILE="/Applications/GRASS-7.9.app/Contents/Resources/include/Make/Platform.make"
sed -i .bak 's|-isysroot /Users/cmbarton/SDKs/MacOSX10.14.sdk||g' $FILE

seems to work, but should be tested thoroughly. (Not tested without Xcode).

[cmbarton]

Here is some additional info on this issue from offline emails and from some additional tests.

I've been building these packages for a couple years with Anaconda and never needed to export CONDA_BUILD_SYSROOT previously. And I don't need it to build 7.8. So something has changed that requires this in 7.9.

The only thing that has changed lately, is that now CC is configurable, before it was static (“gcc”, with your previous patch "clang”). This in itself may, somehow indirect lead to the requirement of CONDA_BUILD_SYSROOT, I don’t know. I do know that a configurable CC is the way it should have been from start.

The issue with compiling extensions may have existed previously and just no one reported it. I'm trying to check this now. This was the case with old non-inclusive package. You had to have the command line dev tools installed. The new apps are supposed to fix that by having tools needed for compiling installed inside the app.

Setting a hard path to "--with-macosx-sdk=“ will lead to setting -isysroot in Platform.make. So, for as long as you have built in this way, compiling extensions was probably broken for other users. Deleting sysroot (pre-distribution) from that file may be the solution for this.

The reason to build with Anaconda is that I wanted to create an all-inclusive binary package that would work as most Mac users expect, without having to install anything extra. This could not be done with the old (and largely obsolete) packaging tools that come with GRASS. There may be other ways to do it, but Anaconda works pretty well and is comparatively easy to manage, especially for getting all of the many dependencies that GRASS needs without me having to compile and update them from scratch.

I understand you perfectly and support your goal. Presently, conda is the best (only) way for this aims.

Maybe there is no need to point to an SDK at all, as you mention. I've been doing it as a legacy from old build protocols that helped ensure that GRASS would be backward compatible with at least a few older OS versions. Maybe with everything inside, it isn't needed and just mucks things up. So I'll try building without it. If CONDA_BUILD_SYSROOT is now required, could it point to the /Contents/Resources folder inside the app? This should have all the needed compiling tools (in /bin, /include, /lib, etc).

This may be a more general way to set

CONDA_BUILD_SYSROOT=$(xcrun --show-sdk-path)

which with 10.15 (and probably back a few versions) will resolve to:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
But you cannot include SDK in binary bundle (legal concerns).

I haven’t been able to figure out actual differences/advantages with using only MACOSX_DEPLOYMENT_TARGET vs. and/or setting -isysroot. But if MACOSX_DEPLOYMENT_TARGET only is enough, that may be the way to go.

I would suggest, now for a start, to keep the 10.14 SDK and CONDA_BUILD_SYSROOT as is, and apply my little sysroot delete script before distribution.

[cmbarton]

The issue with compiling extensions may have existed previously and just no one reported it. I'm trying to check this now. This was the case with old non-inclusive package. You had to have the command line dev tools installed. The new apps are supposed to fix that by having tools needed for compiling installed inside the app.

So with some tests, even though this should be possible, it doesn't seem to work.

Setting a hard path to "--with-macosx-sdk=“ will lead to setting -isysroot in Platform.make. So, for as long as you have built in this way, compiling extensions was probably broken for other users. Deleting sysroot (pre-distribution) from that file may be the solution for this.

For GRASS 7.9, we need to set both CONDA_BUILD_SYSROOT and configure argument --with-macosx-sdk= for setting -isysroot in Platform.make. Leaving either out will cause either configuration or make to fail.

The reason to build with Anaconda is that I wanted to create an all-inclusive binary package that would work as most Mac users expect, without having to install anything extra. This could not be done with the old (and largely obsolete) packaging tools that come with GRASS. There may be other ways to do it, but Anaconda works pretty well and is comparatively easy to manage, especially for getting all of the many dependencies that GRASS needs without me having to compile and update them from scratch.

I understand you perfectly and support your goal. Presently, conda is the best (only) way for this aims.

Maybe there is no need to point to an SDK at all, as you mention. I've been doing it as a legacy from old build protocols that helped ensure that GRASS would be backward compatible with at least a few older OS versions. Maybe with everything inside, it isn't needed and just mucks things up. So I'll try building without it. If CONDA_BUILD_SYSROOT is now required, could it point to the /Contents/Resources folder inside the app? This should have all the needed compiling tools (in /bin, /include, /lib, etc).

Pointing CONDA_BUILD_SYSROOT to the folders inside the app causes configure to fail

This may be a more general way to set

CONDA_BUILD_SYSROOT=$(xcrun --show-sdk-path)

which with 10.15 (and probably back a few versions) will resolve to:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
But you cannot include SDK in binary bundle (legal concerns).

This works. It means that anyone wanting to compile an extension with g.extension must also install Apple's Xcode. Something I wanted to avoid but don't currently see how we can.

I haven’t been able to figure out actual differences/advantages with using only MACOSX_DEPLOYMENT_TARGET vs. and/or setting -isysroot. But if MACOSX_DEPLOYMENT_TARGET only is enough, that may be the way to go.

MACOSX_DEPLOYMENT_TARGET is not enough. We must also set CONDA_BUILD_SYSROOT and --with-macosx-sdk= for both configure and build to work in GRASS 7.9 on the Mac. Probably need to do this generically (as above) on 7.8 too. If Xcode is installed, then extensions will compile properly.

I would suggest, now for a start, to keep the 10.14 SDK and CONDA_BUILD_SYSROOT as is, and apply my little sysroot delete script before distribution.

deleting -isysroot in the distribution causes compiling extensions to fail. So there is no point in doing this.

[nilason]

deleting -isysroot in the distribution causes compiling extensions to fail. So there is no point in doing this.

This worked for me with the binary. Please elaborate.

[cmbarton]

I can delete -isroot in 3 of the 4 places without apparent effects (though I'm only testing with 2 extensions: r.fill.gap and r.geomorphon). But if I delete it in the "CFLAGS =" line, it causes compilation to fail with the following error in the console:

Fetching <r.fill.gaps> from GRASS GIS Addons repository (be patient)...
Compiling...
cell_funcs.c:21:10: fatal error: 'math.h' file not found
#include <math.h>
         ^~~~~~~~
1 error generated.
make: *** [OBJ.x86_64-apple-darwin19.6.0/cell_funcs.o] Error
1
ERROR: Compilation failed, sorry. Please check above error messages.

This error is the same whether I set the path to the SDK in Xcode or to my local one.

The error for r.geomorphon is a bit different, but these errors might be misleading and simply a result of not being able to find a viable SKD.

Fetching <r.geomorphon> from GRASS GIS Addons repository (be patient)...
Compiling...
clang-10: error: no such file or directory: 'k'
make: *** [/tmp/grass7-cmbarton-17762/tmpxvdp4mru/r.geomorph
on/bin/r.geomorphon] Error 1
ERROR: Compilation failed, sorry. Please check above error messages.

[nilason]

Yeah, deleting -isysroot isn't working for me either, I now learned after double checking. Must have done something wrong, sorry.

But an SDK is needed for compiling extensions. There are no more system header files in /usr/include, as that directory is removed with 10.14. I believe Command Line Tools (xcode-select --install) would be enough, Xcode not needed. Which leaves some SDK in /Library/Developer/CommandLineTools/SDKs/. With 10.15 there is a MacOSX10.14.sdk too, you may use that one for a more generally available path. It's possible Platform.make must be modified pre-distribution to point to the user's present default SDK, /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk.

[nilason]

Compiling extension on mac (using the binary) is an old issue, see https://gis.stackexchange.com/questions/325204/error-installing-r-stream-extension-grass-gis-on-macos.

[cmbarton]

Yeah, deleting -isysroot isn't working for me either, I now learned after double checking. Must have done something wrong, sorry.

But an SDK is needed for compiling extensions. There are no more system header files in /usr/include, as that directory is removed with 10.14. I believe Command Line Tools (xcode-select --install) would be enough, Xcode not needed. Which leaves some SDK in /Library/Developer/CommandLineTools/SDKs/. With 10.15 there is a MacOSX10.14.sdk too, you may use that one for a more generally available path. It's possible Platform.make must be modified pre-distribution to point to the user's present default SDK, /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk.

I think this is the best way to go. Is there a way to make this generic, without a hard coded path along the lines of $(xcrun --show-sdk-path)?

[nilason]

I think this is the best way to go. Is there a way to make this generic, without a hard coded path along the lines of $(xcrun --show-sdk-path)?

I'm working on the issue, but in the meantime I would say a hardcoded path to Command Line Tools /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk is the most realistic way for the distribution binary. With a note of the requirement of xcode-select --install for compiling extensions. If I'm not mistaken xcrun --show-sdk-path will only work with Xcode installed.

Please, take a look and test https://github.com/nilason/grass-conda, which I just put up. It's a step to formalise building Grass.app with conda, based on your instructions on grasswiki. I have 100 per cent reproductivity, but can always be improved. Creating a dmg, is still on the todo-list. @cmbarton and everyone else: if you have suggestions, issues, feel free to address them there.

I believe this issue can be closed as it is not a GRASS GIS issue, but a packaging issue.

[cmbarton]

This is a good way forward. IMHO, packaging GRASS is a GRASS issue. This has been a very big issue for the Mac for quite awhile, and attention to it by the GRASS community has made it much better.

[nilason]

Is this still an issue? I believe it's not.

[cmbarton]

No longer an issue. Closing

[lucasjinreal]

clang: error: no such file or directory: 'zlib'