building universal2 binaries from mixed Fortran/C++ code (current issue: ___builtin_nested_func_ptr_created symbol not found)

[slayoo]

Hello,
CC: @s-u

I'm using the packaged gcc-12-branch from https://github.com/R-macos/gcc-12-branch/releases and I'm trying to build universal binaries using the dr.sh driver driver script (https://github.com/R-macos/gcc-12-branch/blob/gcc-12-2-darwin/build/dr.sh). I'm instructing CMake to use it with cmake -DCMAKE_Fortran_COMPILER=dr.sh but it errs with The CMAKE_Fortran_COMPILER: /Users/runner/work/PyPartMC/PyPartMC/dr.sh is not a full path to an existing compiler tool.

What is the intended way to use the dr.sh?

(posting here as the R-macos fork has issues disabled)

Thanks,
Sylwester

[s-u]

@slayoo Can you provide more details? It works for me just fine for me (see below), but make sure the script is executable (chmod a+x dr.sh) and that you have the installed full GNU Fortran installation from the release. If in doubt, point to the project you are trying to compile. It should be a drop-in replacement for gfortran, it just runs the compilers twice and the runs lipo on the result. I didn't test for any quoting issues (e.g. if you have spaces in your paths) so your mileage may vary, but then we'd need more details to look into it.

$ cmake -DCMAKE_Fortran_COMPILER=dr.sh  .. && make
-- The Fortran compiler identification is GNU 12.2.0
-- Checking whether Fortran compiler has -isysroot
-- Checking whether Fortran compiler has -isysroot - yes
-- Checking whether Fortran compiler supports OSX deployment target flag
-- Checking whether Fortran compiler supports OSX deployment target flag - yes
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /Users/urbanek/cmake-test/dr.sh - skipped
-- Configuring done (0.6s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/urbanek/cmake-test/build
[ 50%] Building Fortran object src/CMakeFiles/my_exe.dir/main.f90.o
x86_64
Look for CMakeFiles/my_exe.dir/main.f90.o
arm64
Look for CMakeFiles/my_exe.dir/main.f90.o
lipo -create -arch x86_64 CMakeFiles/my_exe.dir/main.f90.o-x86_64 -arch arm64 CMakeFiles/my_exe.dir/main.f90.o-arm64 -output CMakeFiles/my_exe.dir/main.f90.o
[100%] Linking Fortran executable my_exe
x86_64
-macosx_version_min has been renamed to -macos_version_min
ld: warning: ignoring duplicate libraries: '-lgcc', '-lgcc_s.1.1'
Look for my_exe
arm64
-macosx_version_min has been renamed to -macos_version_min
ld: warning: ignoring duplicate libraries: '-lemutls_w', '-lgcc'
Look for my_exe
lipo -create -arch x86_64 my_exe-x86_64 -arch arm64 my_exe-arm64 -output my_exe
[100%] Built target my_exe

$ file src/my_exe
src/my_exe: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64:Mach-O 64-bit executable arm64]
src/my_exe (for architecture x86_64):	Mach-O 64-bit executable x86_64
src/my_exe (for architecture arm64):	Mach-O 64-bit executable arm64

(I'd change the name of the script to something more obvious like gfortran-universal and put it in the same directory as the binaries, but it doesn't really matter for it to work....)

[slayoo]

@s-u, thank you for quick reply! I was exactly assuming as you describe it - that it's a drop-in replacement for gfortran. Thank you for checking with CMake.

I'm trying to automate building of universal2 wheels for the PyPartMC project. In the CI script, this looks like this:
https://github.com/open-atmos/PyPartMC/pull/291/files
(chmod 755 dr.sh is there)

and the log is here:
https://github.com/open-atmos/PyPartMC/actions/runs/6761737134/job/18376968854?pr=291#step:11:987

I'm investigating further...

[s-u]

This is the problem:

wget https://github.com/R-macos/gcc-12-branch/blob/gcc-12-2-darwin/build/dr.sh

it doesn't pull the script - you need the raw version from GH so it should be

wget https://raw.githubusercontent.com/R-macos/gcc-12-branch/gcc-12-2-darwin/build/dr.sh

[slayoo]

Thank you, @s-u!

[s-u]

@slayoo BTW: As a test I just tried to compile gitmodules/json-fortran from your project, but cmake wouldn't add the necessary -arch flags to FFLAGS for me despite -DCMAKE_OSX_ARCHITECTURES='arm64;x86_64' so I had to use -DCMAKE_Fortran_FLAGS='-arch arm64 -arch x86_64'. I don't know if this is a CMake bug or a problem with the project...

[slayoo]

Thanks! Perhaps CMake doesn't add it since unpatched gfortran doesn't support multiple arch options?

[slayoo]

After setting FC=/opt/gfortran/bin/gfortran-universal -arch arm64 -arch x86_64 (where gfortran-universal is the renamed dr.sh), indeed enables compilation, but CMake still chokes on linking a standard test program:

-- The Fortran compiler identification is GNU 12.2.0
...
-- Checking whether Fortran compiler has -isysroot
-- Checking whether Fortran compiler has -isysroot - yes
-- Checking whether Fortran compiler supports OSX deployment target flag
-- Checking whether Fortran compiler supports OSX deployment target flag - yes
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - failed
-- Check for working Fortran compiler: /opt/gfortran/bin/gfortran-universal
-- Check for working Fortran compiler: /opt/gfortran/bin/gfortran-universal - broken
CMake Error at /Users/runner/hostedtoolcache/cmake/3.26.5/x64/cmake-3.26.5-macos-universal/CMake.app/Contents/share/cmake-3.26/Modules/CMakeTestFortranCompiler.cmake:59 (message):
  The Fortran compiler

    "/opt/gfortran/bin/gfortran-universal"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-via-sdist-r8ez5pvq/PyPartMC-0.6.4.post21/build/temp.macosx-10.9-universal2-cpython-311/_PyPartMC/CMakeFiles/CMakeScratch/TryCompile-74M2NC
    
    Run Build Command(s):/Users/runner/hostedtoolcache/cmake/3.26.5/x64/cmake-3.26.5-macos-universal/CMake.app/Contents/bin/cmake -E env VERBOSE=1 /usr/bin/make -f Makefile cmTC_ae3c0/fast && /Applications/Xcode_13.4.1.app/Contents/Developer/usr/bin/make  -f CMakeFiles/cmTC_ae3c0.dir/build.make CMakeFiles/cmTC_ae3c0.dir/build
    Building Fortran object CMakeFiles/cmTC_ae3c0.dir/testFortranCompiler.f.o
    /opt/gfortran/bin/gfortran-universal  -arch arm64 -arch x86_64   -isysroot /Applications/Xcode_13.4.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -mmacosx-version-min=12.7 -c /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-via-sdist-r8ez5pvq/PyPartMC-0.6.4.post21/build/temp.macosx-10.9-universal2-cpython-311/_PyPartMC/CMakeFiles/CMakeScratch/TryCompile-74M2NC/testFortranCompiler.f -o CMakeFiles/cmTC_ae3c0.dir/testFortranCompiler.f.o
    arm64
    Look for CMakeFiles/cmTC_ae3c0.dir/testFortranCompiler.f.o
    x86_64
    Look for CMakeFiles/cmTC_ae3c0.dir/testFortranCompiler.f.o
    lipo -create -arch arm64 CMakeFiles/cmTC_ae3c0.dir/testFortranCompiler.f.o-arm64 -arch x86_64 CMakeFiles/cmTC_ae3c0.dir/testFortranCompiler.f.o-x86_64 -output CMakeFiles/cmTC_ae3c0.dir/testFortranCompiler.f.o
    Linking Fortran executable cmTC_ae3c0
    /Users/runner/hostedtoolcache/cmake/3.26.5/x64/cmake-3.26.5-macos-universal/CMake.app/Contents/bin/cmake -E cmake_link_script CMakeFiles/cmTC_ae3c0.dir/link.txt --verbose=1
    /opt/gfortran/bin/gfortran-universal  -arch arm64 -arch x86_64  -isysroot /Applications/Xcode_13.4.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -mmacosx-version-min=12.7 CMakeFiles/cmTC_ae3c0.dir/testFortranCompiler.f.o -o cmTC_ae3c0 
    arm64
    collect2: fatal error: cannot find ‘ld’

Any hints?
Thank you for your help!

[s-u]

Highly likely you are missing /opt/gfortran/bin on your $PATH, so make sure you run

export PATH=/opt/gfortran/bin:$PATH

before.

[slayoo]

Thank you, @s-u!
Setting the path indeed helped, and (save for a warning which apparently originates from linking C++ and Fortran originating .o files together), the compilation succeeds. The resultant .so file when loaded from Python via import ... causes the following failure:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/PyPartMC/__init__.py", line 44, in <module>
    import _PyPartMC
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_PyPartMC.cpython-311-darwin.so, 0x0002):
 symbol not found in flat namespace (___builtin_nested_func_ptr_created)
Error: Process completed with exit code 1.

which leads to https://gcc.gnu.org/bugzilla//show_bug.cgi?id=106487 ... and further (TBC)

[slayoo]

same error message was reported, e.g., here: iains/gcc-darwin-arm64#34 (comment)

[s-u]

This is no longer related to the universal wrapper - that looks like a problem in your linking (failing to link the necessary libgcc run-time - that's where ___builtin_nested_func_ptr_created is defined). Note that linking C++ with Fortran is quite tricky - you have the merge flags for both compilers, they cannot do it for you, because they don't know about each other's flags (you are dealing with clang C/C++ toolchain and GCC Fortran so you need to tell the linker to include both run-times - and the drivers add only their own run-time by default).

[slayoo]

Thank you, @s-u

Concerning ___builtin_nested_func_ptr_created, for the record, Googling further, I've found the following patch in Homebrew gcc codebase:

+@item -foff-stack-trampolines
+@opindex foff-stack-trampolines
+Certain platforms (such as the Apple M1) do not permit an executable
+stack. Generate calls to @code{__builtin_nested_func_ptr_created} and
+@code{__builtin_nested_func_ptr_deleted} in order to allocate and
+deallocate trampoline space on the executable heap. Please note that
+these functions are implemented in libgcc, and will not be compiled in
+unless you provide @option{--enable-off-stack-trampolines} when
+building gcc. 

(https://raw.githubusercontent.com/Homebrew/formula-patches/1d184289/gcc/gcc-12.2.0-arm.diff)

... still digesting what it means for this workflow...

Anyhow, let me also ask about this warning which points to /opt/gfortran/lib/gcc/:

[ 96%] Linking CXX shared module /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-via-sdist-3m25uo7r
/PyPartMC-0.6.4.post23/build/lib.macosx-10.9-universal2-cpython-311/_PyPartMC.cpython-311-darwin.so
ld: warning: ignoring file /opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/12.2.0/libgfortran.dylib,
 building for macOS-x86_64 but attempting to link with file built for macOS-arm64

Does it mean that the linker is unable to correctly link against libgfortran? Could it be related with the above libgcc error?

Thanks!

[s-u]

@slayoo as I said above - you are using the wrong linking flags (look at the paths - it's using arm64 Fortran flags when linking x86_64 which obviously cannot work). This has nothing to do with the compiler, it works just fine, so the patches and links are red herrings that are distracting you from the real issue.

Ok, let me explain a bit more in detail what I meant above: In most projects, the linker (ld) is not called directly, but rather by the language driver so it can add the necessary flags for the run-time. For example, consider a Fortran driver being used for linking:

$ gfortran -v hello.o
Driving: /opt/gfortran/bin/aarch64-apple-darwin20.0-gfortran -v hello.o -mmacosx-version-min=14.0.0 
 -asm_macosx_version_min=14.0 -nodefaultexport -l gfortran
[...]
/opt/gfortran/libexec/gcc/aarch64-apple-darwin20.0/12.2.0/collect2 -syslibroot /opt/gfortran/SDK/ 
 -dynamic -arch arm64 -macosx_version_min 14.0.0 -o a.out 
 -L/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/12.2.0 
 -L/opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/12.2.0/../../.. 
 hello.o -lgfortran -lemutls_w -lgcc -lquadmath -lemutls_w 
 -lgcc -lSystem -lgcc -no_compact_unwind -rpath @loader_path 
 -rpath /opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/12.2.0 -rpath /opt/gfortran/lib

Notice all the Fortran run-time linking flags adding the gcc library paths and in particular notice the -lgcc which is where ___builtin_nested_func_ptr_created is defined and pulled from /opt/gfortran/lib/gcc/aarch64-apple-darwin20.0/12.2.0/libgcc.a.

So if you use the Fortran driver to link, everything is just fine and everything works as is should.

Now the problem starts if you are mixing languages and toolchains. So let's look at the C++ driver instead which comes from Apple's clang:

$ clang++ -v foo.o
[...]
"/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" 
 -demangle -lto_library 
 Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib 
 -dynamic -arch arm64 -platform_version macos 14.0.0 14.0 -syslibroot 
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk 
 -o a.out -L/usr/local/lib foo.o -lc++ -lSystem 
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/15.0.0/lib/darwin/libclang_rt.osx.a

Notice that there is no -lgcc and the run-time now comes from the clang directories. So simply put: if you link C++ and Fortran code this way, it will fail, because the C++ driver has no idea about the Fortran flags.

This is where things get really complicated: in order to create binary with both C++ and Fortran code (simple lesson: don't do that!), you have to manually find out all the run-time flags and in the right order.

In your last example above, obviously some part of the configuration has been trying to detect the flags, but it only did it for arm64 and then tried to use the same (thus wrong!) flags for x86_64. Have a look at gfortran -v -arch arm64 and gfortran -v -arch x86_64 - they are different as the directories contain the architecture name.

Similarly, in you first example, it was using CXX for linking which didn't include the necessary libgcc.a which is necessary for the ___builtin_nested_func_ptr_created symbol. So the problem is not the existence ___builtin_nested_func_ptr_created which is completely fine, it is the wrong linker flags that don't link the necessary libraries.

I suspect that your project is simply not setup to handle dual-archs correctly. I think CMake is simply incapable of linking Fortran and C++ automatically (which is possibly why -DCMAKE_OSX_ARCHITECTURES='arm64;x86_64' doesn't work for Fortran), so you would have to make it work in your project.

The links you posted above are unrelated - there is nothing wrong here, the Fortran compiler works just fine - most of the posts above are simply proofs of bad run-time mismatches, because you get the error with ___builtin_nested_func_ptr_created if your linker flags are simply wrong for the run-time you're using (typically trying to use clang drivers to link gcc binaries without the gcc flags -- or in one of the linked pages it's simply wrong version of gcc since you have to use the one from the Fortran branch).

I hope this helps to explain. I think the crux of the problem is the attempt to use C++ linker for Fortran code. I'd say it would be a lot easier to use the Fortran linker instead and add the C++ flags as they are far simpler to add - so that would be my recommendation.

[slayoo]

Thank you, once again, @s-u!
The project in question (PyPartMC) is essentially a set of C++ bindings for Fortran code with no additional logic, hence linking Fortran and C++ is the very key purpose here. We're routinely releasing binary packages of it (as Python wheels) for Linux, Windows and macOS, and it works both on arm64 and x86_64 there - the codebase is not something too experimental at this stage. We're trying to add cross compilation to produce macOS universal wheels on Github Actions.

Based on your explanation, I was able to silence the "attempting to link with file built for macOS-arm64" warning by doing:

sudo lipo -create \
  -arch x86_64 /opt/gfortran/lib/gcc/x86_64*/*/libgcc.a \
  -arch arm64 /opt/gfortran/lib/gcc/aarch64*/*/libgcc.a \
  -output /opt/gfortran/lib/libgcc_universal.a
FC="gfortran-universal -arch arm64 -arch x86_64 -L/opt/gfortran/lib -lgcc_universal"

Still, it complains at runtime about the missing symbol. No luck yet with using the gfortran linker to assemble the final .so file. Will try to create a minimal example to narrow the scope and be able to ask more precise questions. Meanwhile, let me close this "issue" and thank again for your help!

[s-u]

@slayoo I don't know what you did or didn't do, but if you can't link per architecture then you need to lipo all the run-time libraries (by default they are dynamic - that's where your second error came from - it was unrelated to your first one!!!). Note that the resulting binary will be useless unless you force static Fortran run-time or ship the universal run-time dylibs with your binary (since Apple doesn't ship them unlike C++).

If I was you I'd build separate targets and lipo them at the end - it is the most reliable (and simple way) - that's what I was doing for the compilers as well since there is no way in hell you can create them in one multi-arch pass - it's too big a can of worms and it only works if the project is setup for it.