autotools builds not working on apple silicon native #4059

maddanio · 2021-01-02T11:48:09Z

Package and Environment Details (include every applicable attribute)

Operating System+version: macos 11.0.1
Compiler+version: apple clang
Conan version: conan 1.32.1
Python version: Python 3.8.2

Conan profile (output of `conan profile show default` or `conan profile show <profile>` if custom profile is in use)

[settings]
os=Macos
os_build=Macos
arch=armv8
arch_build=armv8
compiler=apple-clang
compiler.version=12.0
compiler.libcxx=libc++
build_type=Release
[options]
[build_requires]
[env]

Steps to reproduce (Include if Applicable)

require any library built with autotools, like libjpeg, yasm, iconv...

Logs (Include/Attach if Applicable)

Click to expand log

yasm/1.3.0: 
yasm/1.3.0: WARN: Package binary is corrupted, removing: 3eb32b976fb0626f3c40a8b87ace72156aa44204
yasm/1.3.0: WARN: Build folder is dirty, removing it: /Users/daniel/.conan/data/yasm/1.3.0/_/_/build/3eb32b976fb0626f3c40a8b87ace72156aa44204
yasm/1.3.0: ERROR: Package '3eb32b976fb0626f3c40a8b87ace72156aa44204' build failed
yasm/1.3.0: WARN: Build folder /Users/daniel/.conan/data/yasm/1.3.0/_/_/build/3eb32b976fb0626f3c40a8b87ace72156aa44204
ERROR: yasm/1.3.0: Error in build() method, line 67
	autotools = self._configure_autotools()
while calling '_configure_autotools', line 60
	self._autotools.configure(args=conf_args, configure_dir=self._source_subfolder)
	ConanException: Error 77 while executing source_subfolder/configure '--enable-debug=no' '--disable-rpath' '--disable-nls' '--prefix=/Users/daniel/.conan/data/yasm/1.3.0/_/_/package/3eb32b976fb0626f3c40a8b87ace72156aa44204' '--bindir=${prefix}/bin' '--sbindir=${prefix}/bin' '--libexecdir=${prefix}/bin' '--libdir=${prefix}/lib' '--includedir=${prefix}/include' '--oldincludedir=${prefix}/include' '--datarootdir=${prefix}/share' --build=x86_64-apple-darwin --host=aarch64-apple-darwin

The text was updated successfully, but these errors were encountered:

maddanio · 2021-01-02T11:51:47Z

at least in the conanfile for yasm (here) there is nothing specific to macos, so it seems that AutoToolsBuildEnvironment makes some wrong choices in what to pass to configure, namely these arguments to configure seem suspicious:
--build=x86_64-apple-darwin --host=aarch64-apple-darwin

SSE4 · 2021-01-02T12:19:33Z

probably, this is due to the arm64 mis-detection: conan-io/conan#8262

maddanio · 2021-01-02T12:37:28Z

Ah, ok, so guess i will need to test with master?

maddanio · 2021-01-02T12:38:29Z

But does that also fix build arch? My profile seens to require armv8?

maddanio · 2021-01-02T14:11:03Z

ok, tested, didnt change anything :(. btw, why do i need to manually set the profile? shouldnt conan detect it as default?

maddanio · 2021-01-02T15:36:00Z

phew, trying to debug, its all rather confusing. as far as I can see it all goes wrong in AutoToolsBuildEnvironment._get_host_build_target_flags which seems should return None, None, target if not cross building, but in order to do so it would actually have to find the target settings. these it wants from get_target_os_arch from the conan file which would require os_target and arch_target there. I specify those in my profile, but that doesnt seem to get it into conanfile.settings which only contains os and arch, albeit those two being correct (Macos and armv8 respectively). is there a concept behind all of this and is it documented?

maddanio · 2021-01-02T15:37:57Z

first off: what are target, build, and host os/arch? and to which of these does the plain os/arch settings belong?

maddanio · 2021-01-02T15:52:31Z

ok, so it seems to work if I hack host/target/build to be the gnu triplet derived from the os and arch settings being aarch64-apple-darwin, now just need to hack autotols env in a sensible way...I guess....

maddanio · 2021-01-02T16:08:26Z

ok, actually iconv still fails, will check more on monday...

jmarrec · 2021-03-26T16:52:56Z

Any update on this? I have trouble with libffi on an M1 chip. Seems like the tripplet is badly recognized as arm-apple-darwin20.3.0 and it fails with missing symbols during testing. If I hack the recipe by adding build=aarch64-apple-darwin20.3.0 then it seems to work

conan-center-index/recipes/libffi/all/conanfile.py

Line 126 in 30d525e

    
           self._autotools.configure(args=config_args, configure_dir=self._source_subfolder, build=build, host=host)

Before hack:

checking build system type... arm-apple-darwin20.3.0
checking host system type... arm-apple-darwin20.3.0
checking target system type... arm-apple-darwin20.3.0

[...]

[ 50%] Building C object CMakeFiles/test_package.dir/test_package.c.o
[100%] Linking C executable bin/test_package
Undefined symbols for architecture arm64:
  "_ffi_call", referenced from:
      _main in test_package.c.o
  "_ffi_prep_cif_machdep", referenced from:
      _ffi_prep_cif_core in libffi.a(prep_cif.o)
  "_ffi_prep_cif_machdep_var", referenced from:
      _ffi_prep_cif_core in libffi.a(prep_cif.o)
  "_ffi_prep_closure_loc", referenced from:
      _ffi_prep_closure in libffi.a(prep_cif.o)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [bin/test_package] Error 1
make[1]: *** [CMakeFiles/test_package.dir/all] Error 2
make: *** [all] Error 2
ERROR: libffi/3.3 (test package): Error in build() method, line 12
	cmake.build()
	ConanException: Error 2 while executing cmake --build '/Users/irvinemac/git/conan/conan-center-index/recipes/libffi/all/test_package/build/06788ff71b0efa6d28472f37ac096c14ff4f7493' '--' '-j8'

maddanio · 2021-03-26T19:12:26Z

I gave up and we moved to a modified version of cget

SSE4 · 2021-03-27T02:50:29Z

@jmarrec what's your conan profile? what's the output of the following command:

conan profile new xxx --detect && conan profile show xxx

as I understand, after conan-io/conan#8262 it should contain arch = armv8.

I don't have M1 machine to test, but as far as I can tell, there are several evidences it should compute aarch64-apple-darwin instead of arm-apple-darwin on M1:
#3966
#4692

e.g. both logs contain --host=aarch64-apple-darwin.

/cc @a4z @theodelrieu

a4z · 2021-03-27T06:09:48Z

I would say, to spot the problem you need to look into the whole configure part, not just one line

   def _configure_autotools(self):
        if self._autotools:
            return self._autotools
        self._autotools = AutoToolsBuildEnvironment(self, win_bash=tools.os_info.is_windows)
        yes_no = lambda v: "yes" if v else "no"
        config_args = [
            "--enable-debug={}".format(yes_no(self.settings.build_type == "Debug")),
            "--enable-shared={}".format(yes_no(self.options.shared)),
            "--enable-static={}".format(yes_no(not self.options.shared)),
        ]
        self._autotools.defines.append("FFI_BUILDING")
        if self.options.shared:
            self._autotools.defines.append("FFI_BUILDING_DLL")
        if self.settings.compiler == "Visual Studio":
            if "MT" in str(self.settings.compiler.runtime):
                self._autotools.defines.append("USE_STATIC_RTL")
            if "d" in str(self.settings.compiler.runtime):
                self._autotools.defines.append("USE_DEBUG_RTL")
        build = None
        host = None
        if self.settings.compiler == "Visual Studio":
            build = "{}-{}-{}".format(
                "x86_64" if "64" in platform.machine() else "i686",
                "pc" if self.settings.arch == "x86" else "w64",
                "cygwin")
            host = "{}-{}-{}".format(
                "x86_64" if self.settings.arch == "x86_64" else "i686",
                "pc" if self.settings.arch == "x86" else "w64",
                "cygwin")
        else:
            if self._autotools.host and "x86-" in self._autotools.host:
                self._autotools.host = self._autotools.host.replace("x86", "i686")
        self._autotools.configure(args=config_args, configure_dir=self._source_subfolder, build=build, host=host)
        return self._autotools

because there are packages building with autotools on M1 (have libcurl for example)

my guess: the else part from if self.settings.compiler == "Visual Studio": looks a little bit hardcoded to one architecture

so I do not think that the general statement, that autotools builds not working, can be made

jmarrec · 2021-03-29T08:40:21Z

@SSE4 conan does detect arch=armv8

[settings]
os=Macos
os_build=Macos
arch=armv8
arch_build=armv8
compiler=apple-clang
compiler.version=12.0
compiler.libcxx=libc++
build_type=Release
[options]
[build_requires]
[env]

a4z · 2021-03-29T08:47:21Z

of course it does @jmarrec , but you might need to update your conan installation to a recent version

[a4z@41012:~/elux](09:45:12)
M1::) conan --version
Conan version 1.34.1

[a4z@41012:~/](09:45:17)
M1::) conan profile new --detect defaultM1
Found apple-clang 12.0
Profile created with detected settings: /Users/a4z/.conan/profiles/defaultM1

[a4z@41012:~/](09:45:34)
M1::) conan profile show defaultM1
Configuration for profile defaultM1:

[settings]
os=Macos
os_build=Macos
arch=armv8
arch_build=armv8
compiler=apple-clang
compiler.version=12.0
compiler.libcxx=libc++
build_type=Release
[options]
[build_requires]
[env]

jmarrec · 2021-03-29T08:49:59Z

I use 1.34.1. I set up this dev machine last friday, so everything is bleeding edge.

SSE4 · 2021-03-29T08:54:21Z

okay, seems like we have a bug where autotools host/build/arch isn't correctly computed for M1 in some cases. I can't debug it, as I don't have machine, but I can point places where to look.
tools.get_gnu_triplet should be the right place to check:

conan-center-index/recipes/json-c/all/conanfile.py

Line 54 in d013176

host = tools.get_gnu_triplet(str(self.settings.os), str(self.settings.arch))

the easiest way to check if it works:

$ python                                                                                                                                                                                                                   Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from  conans.client.tools.oss import get_gnu_triplet
>>> get_gnu_triplet('Macos', 'armv8')
'aarch64-apple-darwin'

if you're getting arm-apple-darwin, then it's a bug

a4z · 2021-03-29T08:56:04Z

[a4z@41012:~/](09:45:41)
M1::) python3
Python 3.8.2 (default, Dec 21 2020, 15:06:03)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from  conans.client.tools.oss import get_gnu_triplet
>>> get_gnu_triplet('Macos', 'armv8')
'aarch64-apple-darwin'

jmarrec · 2021-03-29T09:02:05Z

(I'd like to provide an answer, but I'm having trouble connecting to that remote machine at the moment... sorry)

SSE4 · 2021-03-29T09:04:14Z

no problem, not urgent.
I want to understand where are these coming from first:

checking build system type... arm-apple-darwin20.3.0
checking host system type... arm-apple-darwin20.3.0
checking target system type... arm-apple-darwin20.3.0

if you have a complete log, it would be nice (e.g. one containing the full command line).
I have a suspicion that conan might not be passing --build/--host/--target at all, but it's auto-detected by config.sub or config.guess of the project itself.

jmarrec · 2021-03-29T10:19:54Z

(py39)irvinemac@git$ conan--version
Conan version 1.34.1

(py39)irvinemac@git$ python --version
Python 3.9.2

(py39)irvinemac@git$ cat ~/.conan/profiles/default 
[settings]
os=Macos
os_build=Macos
arch=armv8
arch_build=armv8
compiler=apple-clang
compiler.version=12.0
compiler.libcxx=libc++
build_type=Release
[options]
[build_requires]
[env]

(py39)irvinemac@git$ python
Python 3.9.2 (default, Mar 26 2021, 04:14:33) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from  conans.client.tools.oss import get_gnu_triplet
>>> get_gnu_triplet('Macos', 'armv8')
'aarch64-apple-darwin'
>>>

And I got disconnected by Teamviewer after 30 seconds again... So I can't give a full libffi log. The machine being in another country where everyone is sleeping, there's not much I can do at the moment. I'll try again in 4 hours...

What I can say is that I had built libffi by checking out conan-center index, using my default profile above.

cd recipes/libffi/all
create create . libffi/3.3@ -b missing

SSE4 · 2021-03-29T10:30:23Z

probably, it might be possible that it's not a conan fault, see libffi/libffi#576
e.g. 3.3 still contains the old config.guess: https://github.com/libffi/libffi/blob/v3.3/config.guess#L1298

jmarrec · 2021-03-29T10:40:53Z

That seems pretty spot-on indeed, good sleuthing @SSE4 !

Now is left the question of what conan should do about it?

work around it by passing build_type in conanfile.py
Patch v3.3 sources to bring the new config.guess and config.sub
Nothing (which I would dislike very much, 5 years have passed between libffi 3.2.1 and 3.3)

SSE4 · 2021-03-29T10:44:47Z

workaround in recipe is okay. you may even explicitly pass --build/--host/--target for M1, or just add the same patch to update config.sub/config.guess from the upstream.
feel free to submit PR.
I personally prefer second approach (updating config.guess), as it should potentially fix more platforms than just M1 - so will make the recipe more portable in general.

jmarrec · 2021-03-29T12:43:02Z

@SSE4 I'll make a patch and submit a PR, thanks for your help. cf #5085

maddanio added the bug Something isn't working label Jan 2, 2021

SSE4 mentioned this issue Mar 29, 2021

[libffi/3.2.1] / [libffi/3.3]Make a patch for 3.2.1 and 3.3 to pull the latest config.guess/config.sub to support aarch64 (M1 macs) #5085

Merged

4 tasks

conan-center-bot closed this as completed in #5085 May 13, 2021

gundelsw mentioned this issue Jan 2, 2024

opencv/4.8.1 opencv/4.5.5 dep libjpeg/9e: linker error not a mach-o file in libjpeg.a building for Macos M1 #22112

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

autotools builds not working on apple silicon native #4059

autotools builds not working on apple silicon native #4059

maddanio commented Jan 2, 2021

maddanio commented Jan 2, 2021

SSE4 commented Jan 2, 2021

maddanio commented Jan 2, 2021

maddanio commented Jan 2, 2021

maddanio commented Jan 2, 2021

maddanio commented Jan 2, 2021

maddanio commented Jan 2, 2021

maddanio commented Jan 2, 2021

maddanio commented Jan 2, 2021

jmarrec commented Mar 26, 2021 •

edited

maddanio commented Mar 26, 2021

SSE4 commented Mar 27, 2021

a4z commented Mar 27, 2021

jmarrec commented Mar 29, 2021

a4z commented Mar 29, 2021

jmarrec commented Mar 29, 2021

SSE4 commented Mar 29, 2021 •

edited

a4z commented Mar 29, 2021

jmarrec commented Mar 29, 2021

SSE4 commented Mar 29, 2021

jmarrec commented Mar 29, 2021 •

edited

SSE4 commented Mar 29, 2021

jmarrec commented Mar 29, 2021

SSE4 commented Mar 29, 2021

jmarrec commented Mar 29, 2021 •

edited

autotools builds not working on apple silicon native #4059

autotools builds not working on apple silicon native #4059

Comments

maddanio commented Jan 2, 2021

Package and Environment Details (include every applicable attribute)

Conan profile (output of conan profile show default or conan profile show <profile> if custom profile is in use)

Steps to reproduce (Include if Applicable)

Logs (Include/Attach if Applicable)

maddanio commented Jan 2, 2021

SSE4 commented Jan 2, 2021

maddanio commented Jan 2, 2021

maddanio commented Jan 2, 2021

maddanio commented Jan 2, 2021

maddanio commented Jan 2, 2021

maddanio commented Jan 2, 2021

maddanio commented Jan 2, 2021

maddanio commented Jan 2, 2021

jmarrec commented Mar 26, 2021 • edited

maddanio commented Mar 26, 2021

SSE4 commented Mar 27, 2021

a4z commented Mar 27, 2021

jmarrec commented Mar 29, 2021

a4z commented Mar 29, 2021

jmarrec commented Mar 29, 2021

SSE4 commented Mar 29, 2021 • edited

a4z commented Mar 29, 2021

jmarrec commented Mar 29, 2021

SSE4 commented Mar 29, 2021

jmarrec commented Mar 29, 2021 • edited

SSE4 commented Mar 29, 2021

jmarrec commented Mar 29, 2021

SSE4 commented Mar 29, 2021

jmarrec commented Mar 29, 2021 • edited

Conan profile (output of `conan profile show default` or `conan profile show <profile>` if custom profile is in use)

jmarrec commented Mar 26, 2021 •

edited

SSE4 commented Mar 29, 2021 •

edited

jmarrec commented Mar 29, 2021 •

edited

jmarrec commented Mar 29, 2021 •

edited