Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fail to make libffi.a on MacOS #2360

Closed
chrispiech opened this issue Apr 7, 2022 · 13 comments
Closed

fail to make libffi.a on MacOS #2360

chrispiech opened this issue Apr 7, 2022 · 13 comments

Comments

@chrispiech
Copy link

when I run
pyodide % sudo make

It looks fine until it gets to this error
source pyodide/emsdk/emsdk/emsdk_env.sh
make[1]: source: No such file or directory
make[1]: *** [pyodide/cpython/installs/python-3.10.2/lib/libffi.a] Error 1
make: *** [pyodide/cpython/installs/python-3.10.2/lib/python3.10] Error 2

when I check I can see that there is in fact no file called libffi.a in
pyodide/cpython/installs/python-3.10.2/lib/

Any help would be greatly apprecaited!

@ryanking13
Copy link
Member

ryanking13 commented Apr 7, 2022

Which OS and shell are you using? The error message says that source command not exists.

Maybe your sudo shell is sh not bash. Try make without sudo, or run it inside the docker image that we provide (./run_docker make).

@MichaelCharles
Copy link

I'm also encountering this and I'm running make without sudo.

The output is below.

[2022-04-23 09:34:33] Building cpython...
make -C /Users/michael/Projects/learn-with/pyodide/cpython
rm -rf /Users/michael/Projects/learn-with/pyodide/cpython/build/libffi
git clone https://github.com/hoodmane/libffi-emscripten /Users/michael/Projects/learn-with/pyodide/cpython/build/libffi --shallow-exclude upstream-base
Cloning into '/Users/michael/Projects/learn-with/pyodide/cpython/build/libffi'...
remote: Enumerating objects: 1674, done.
remote: Counting objects: 100% (1674/1674), done.
remote: Compressing objects: 100% (797/797), done.
remote: Total 1674 (delta 1029), reused 1344 (delta 848), pack-reused 0
Receiving objects: 100% (1674/1674), 883.19 KiB | 8.57 MiB/s, done.
Resolving deltas: 100% (1029/1029), done.
source /Users/michael/Projects/learn-with/pyodide/emsdk/emsdk/emsdk_env.sh
make[1]: source: No such file or directory
make[1]: *** [/Users/michael/Projects/learn-with/pyodide/cpython/installs/python-3.10.2/lib/libffi.a] Error 1
make: *** [/Users/michael/Projects/learn-with/pyodide/cpython/installs/python-3.10.2/lib/python3.10] Error 2

I'm on an Intel Macbook Pro using zsh. I've checked on sh, bash and zsh and I'm able to manually execute source with no problem.

Switching from zsh to bash or sh, and then running make does not change the output.

@MichaelCharles
Copy link

Attempting to use ./run_docker make results in a different error:

checking whether the C compiler works... no
configure: error: in `/src/cpython/build/libffi':
configure: error: C compiler cannot create executables
See `config.log' for more details
emconfigure: error: './configure --host=wasm32-unknown-linux --prefix=/src/cpython/build/libffi/target --enable-static --disable-shared --disable-dependency-tracking --disable-builddir --disable-multi-os-directory --disable-raw-api --disable-docs' failed (returned 77)
make[1]: *** [Makefile:123: /src/cpython/installs/python-3.10.2/lib/libffi.a] Error 1
make[1]: Leaving directory '/src/cpython'
make: *** [Makefile:229: /src/cpython/installs/python-3.10.2/lib/python3.10] Error 2

This also seems related to libffi?

@MichaelCharles
Copy link

Solved it for my case.

Despite there being no changes in the files tracked by git, I thought maybe there might some sort of cache in the untracked files (sorry I haven't looked too closely into what's going on).

Anyway, I deleted the entire folder the Pyodide source was in, and then cloned it again fresh.

After that, I tried running ./run_docker make, and I was able to successfully build.

@rth
Copy link
Member

rth commented Apr 23, 2022

Yes, MacOS build support is not very good right now. So it's expected that you would run into some edge cases. Docker on linux should work, and probably indeed better not to mix the two.

@dopplershift
Copy link
Contributor

So I hit this and managed to get past it:

diff --git a/cpython/Makefile b/cpython/Makefile
index 5f069db0..37cc5d37 100644
--- a/cpython/Makefile
+++ b/cpython/Makefile
@@ -119,8 +119,7 @@ $(INSTALL)/lib/libbz2.a: $(BZIP2TARBALL)
 $(INSTALL)/lib/libffi.a :
        rm -rf $(FFIBUILD)
        git clone $(LIBFFIREPO) $(FFIBUILD) --shallow-exclude upstream-base
-   source $(PYODIDE_ROOT)/emsdk/emsdk/emsdk_env.sh
-   cd $(FFIBUILD) && git checkout $(LIBFFI_COMMIT) && ./build.sh && make install
+ . $(PYODIDE_ROOT)/emsdk/emsdk/emsdk_env.sh && cd $(FFIBUILD) && git checkout $(LIBFFI_COMMIT) && pwd && ./build.sh && make install
        cp $(FFIBUILD)/target/include/*.h $(BUILD)/Include/
        mkdir -p $(INSTALL)/lib
        cp $(FFIBUILD)/target/lib/libffi.a $(INSTALL)/lib/

It seems one should not try to source in a Makefile. Now, after installing autoconf from Homebrew, I now get:

autoreconf: running: /usr/local/Cellar/autoconf/2.71/bin/autoconf --force
configure.ac:8: warning: The macro `AC_CANONICAL_SYSTEM' is obsolete.
configure.ac:8: You should run autoupdate.
./lib/autoconf/general.m4:2081: AC_CANONICAL_SYSTEM is expanded from...
configure.ac:8: the top level
configure.ac:74: warning: The macro `AC_TRY_COMPILE' is obsolete.
configure.ac:74: You should run autoupdate.
./lib/autoconf/general.m4:2847: AC_TRY_COMPILE is expanded from...
lib/m4sugar/m4sh.m4:692: _AS_IF_ELSE is expanded from...
lib/m4sugar/m4sh.m4:699: AS_IF is expanded from...
./lib/autoconf/general.m4:2249: AC_CACHE_VAL is expanded from...
./lib/autoconf/general.m4:2270: AC_CACHE_CHECK is expanded from...
acinclude.m4:3: AC_FUNC_MMAP_BLACKLIST is expanded from...
configure.ac:74: the top level
configure.ac:91: warning: The macro `AC_HEADER_STDC' is obsolete.
configure.ac:91: You should run autoupdate.
./lib/autoconf/headers.m4:704: AC_HEADER_STDC is expanded from...
configure.ac:91: the top level
configure.ac:118: warning: The macro `AC_TRY_COMPILE' is obsolete.
configure.ac:118: You should run autoupdate.
./lib/autoconf/general.m4:2847: AC_TRY_COMPILE is expanded from...
lib/m4sugar/m4sh.m4:692: _AS_IF_ELSE is expanded from...
lib/m4sugar/m4sh.m4:699: AS_IF is expanded from...
./lib/autoconf/general.m4:2249: AC_CACHE_VAL is expanded from...
./lib/autoconf/general.m4:2270: AC_CACHE_CHECK is expanded from...
m4/asmcfi.m4:1: GCC_AS_CFI_PSEUDO_OP is expanded from...
configure.ac:118: the top level
configure.ac:122: warning: The macro `AC_TRY_LINK' is obsolete.
configure.ac:122: You should run autoupdate.
./lib/autoconf/general.m4:2920: AC_TRY_LINK is expanded from...
lib/m4sugar/m4sh.m4:692: _AS_IF_ELSE is expanded from...
lib/m4sugar/m4sh.m4:699: AS_IF is expanded from...
./lib/autoconf/general.m4:2249: AC_CACHE_VAL is expanded from...
./lib/autoconf/general.m4:2270: AC_CACHE_CHECK is expanded from...
configure.ac:122: the top level
configure.ac:138: warning: The macro `AC_TRY_COMPILE' is obsolete.
configure.ac:138: You should run autoupdate.
./lib/autoconf/general.m4:2847: AC_TRY_COMPILE is expanded from...
lib/m4sugar/m4sh.m4:692: _AS_IF_ELSE is expanded from...
lib/m4sugar/m4sh.m4:699: AS_IF is expanded from...
./lib/autoconf/general.m4:2249: AC_CACHE_VAL is expanded from...
./lib/autoconf/general.m4:2270: AC_CACHE_CHECK is expanded from...
configure.ac:138: the top level
configure.ac:185: warning: The macro `AC_TRY_COMPILE' is obsolete.
configure.ac:185: You should run autoupdate.
./lib/autoconf/general.m4:2847: AC_TRY_COMPILE is expanded from...
lib/m4sugar/m4sh.m4:692: _AS_IF_ELSE is expanded from...
lib/m4sugar/m4sh.m4:699: AS_IF is expanded from...
./lib/autoconf/general.m4:2249: AC_CACHE_VAL is expanded from...
./lib/autoconf/general.m4:2270: AC_CACHE_CHECK is expanded from...
configure.ac:185: the top level
configure.ac:310: warning: The macro `AC_HELP_STRING' is obsolete.
configure.ac:310: You should run autoupdate.
./lib/autoconf/general.m4:204: AC_HELP_STRING is expanded from...
configure.ac:310: the top level
configure.ac:418: warning: The macro `AC_HELP_STRING' is obsolete.
configure.ac:418: You should run autoupdate.
./lib/autoconf/general.m4:204: AC_HELP_STRING is expanded from...
acinclude.m4:353: LIBFFI_ENABLE_SYMVERS is expanded from...
configure.ac:418: the top level
configure.ac:418: warning: AC_PROG_LD is m4_require'd but not m4_defun'd
acinclude.m4:251: LIBFFI_CHECK_LINKER_FEATURES is expanded from...
acinclude.m4:353: LIBFFI_ENABLE_SYMVERS is expanded from...
configure.ac:418: the top level
configure.ac:418: warning: The macro `AC_TRY_RUN' is obsolete.
configure.ac:418: You should run autoupdate.
./lib/autoconf/general.m4:2997: AC_TRY_RUN is expanded from...
acinclude.m4:251: LIBFFI_CHECK_LINKER_FEATURES is expanded from...
acinclude.m4:353: LIBFFI_ENABLE_SYMVERS is expanded from...
configure.ac:418: the top level
configure.ac:418: warning: The macro `AC_TRY_LINK' is obsolete.
configure.ac:418: You should run autoupdate.
./lib/autoconf/general.m4:2920: AC_TRY_LINK is expanded from...
acinclude.m4:353: LIBFFI_ENABLE_SYMVERS is expanded from...
configure.ac:418: the top level
configure.ac:41: error: possibly undefined macro: AC_PROG_LIBTOOL
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
configure:8529: error: possibly undefined macro: AC_PROG_LD
autoreconf: error: /usr/local/Cellar/autoconf/2.71/bin/autoconf failed with exit status: 1
make[1]: *** [/Users/rmay/repos/pyodide/cpython/installs/python-3.10.2/lib/libffi.a] Error 1
make: *** [/Users/rmay/repos/pyodide/cpython/installs/python-3.10.2/lib/python3.10] Error 2

Any idea what version of autoconf/automake should work?

@dopplershift
Copy link
Contributor

Ok, latest autoconf (2.71) from Homebrew was fine. That error was due to missing libtool.

@rth rth changed the title fail to make libffi.a fail to make libffi.a on MacOS May 6, 2022
@rth
Copy link
Member

rth commented May 6, 2022

Any idea what version of autoconf/automake should work?

Yes, a reasonably recent one should work. FYI I also tried to make the build work with conda on MacOS in #1912 but got blocked by conda compilers not interacting well with the emscripten ones. Probably because I wasn't not doing something right, I don't have a Mac.

If you manage to make it work with Homebrew that would be perfect. Also make sure you have a recent enough version of patch. See also previous effort in https://github.com/pyodide/pyodide/pull/590/files

@rth
Copy link
Member

rth commented May 6, 2022

Otherwise using the Docker image is still the easiest way to build including on MacOS (both intel and M1 CPUs)

@dopplershift
Copy link
Contributor

Thanks for the heads up. I'm currently relying (I believe) on the native Mac compilers and everything else from Homebrew. I'm stubborn, so I'll keep at it natively. It's now failing with:

❯ make
./tools/dependency-check.sh
npx rollup -c src/js/rollup.config.js

./src/js/pyodide.ts → dist/pyodide.mjs...
[!] (plugin Typescript) TS1134: Variable declaration expected.
src/js/pyproxy.gen.ts (136:13)

136 declare var (1 << 8): number;
                ~


make: *** [src/js/_pyodide.out.js] Error 1

but I haven't been able to dig it out any further.

@hoodmane
Copy link
Member

hoodmane commented May 6, 2022

That's a problem with sed. As per the comment on line 23, lines 21 through 49 are supposed to be removed from src/core/pyproxy.ts by sed on line 114 of the Makefile: sed '/^\/\/\s*pyodide-skip/,/^\/\/\s*end-pyodide-skip/d' | \ but it clearly isn't working right for you.

@rth
Copy link
Member

rth commented May 6, 2022

I'm stubborn, so I'll keep at it natively.

Thanks a lot for your perseverance!

Yes, sed is another one that's not exactly the same. Maybe installing gnu-sed from Homebrew would help.

dopplershift added a commit to dopplershift/pyodide that referenced this issue May 12, 2022
@dopplershift
Copy link
Contributor

Installing gnu-sed from Homebrew (and putting it on the PATH) was the magic. Successful build with the Makefile fix I posted above (contributed in #2554).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants