Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maybe need add libwebp.7.dylib to Frameworks folder? #85

Closed
ccqpein opened this issue Jul 7, 2023 · 24 comments · Fixed by #88
Closed

Maybe need add libwebp.7.dylib to Frameworks folder? #85

ccqpein opened this issue Jul 7, 2023 · 24 comments · Fixed by #88

Comments

@ccqpein
Copy link

ccqpein commented Jul 7, 2023

Problems:

After build the emacs, and codesign (#64). My mac return me the panic report that:

dyld[16543]: Library not loaded: @rpath/libwebp.7.dylib
  Referenced from: <8B34F8AD-74B5-371E-AED8-5F8BC7340372> /Applications/Emacs.app/Contents/Frameworks/libwebpdemux.2.dylib
  Reason: tried: '/Applications/Emacs.app/Contents/Frameworks/../lib/libwebp.7.dylib' (no such file), '/Applications/Emacs.app/Contents/Frameworks/libwebp.7.dylib' (no such file), '/Applications/Emacs.app/Contents/Frameworks/libwebp.7.dylib' (no such file), '/usr/local/lib/libwebp.7.dylib' (no such file), '/usr/lib/libwebp.7.dylib' (no such file, not in dyld cache)

Temporary solution:

After some research and debugging, I find it the @rpath value is path @executable_path/../Frameworks (offset 12). After I add homebrew's lib path to @rpath by install_name_tool -add_rpath /opt/homebrew/lib /Applications/Emacs.app/Contents/MacOS/Emacs. The problem is solved. It can get the libwebp.7.dylib now.

I checked the code and find line 170 define extra_libs which copy to Frameworks in embed (maybe? I don't know Ruby).

Maybe we can also copy libwebp.7.dylib too? Thanks!

My env:

macOS 13.4.1
M2 Max
GNU Emacs master branch

@ccqpein
Copy link
Author

ccqpein commented Jul 7, 2023

After manually copy libwebp.7.dylib to Framework, I find there is another thing need to copy to Framework. libsharpyuv.0.dylib also need to copy to Framework

cp /opt/homebrew/lib/libsharpyuv.0.dylib /Applications/Emacs.app/Contents/Frameworks/

Maybe script can add an optional config that add the homebrew lib path to @rpath if we don't want to embedding?

@ccqpein
Copy link
Author

ccqpein commented Jul 7, 2023

Make PR #86

@stevemolitor
Copy link

stevemolitor commented Jul 17, 2023

I ran into the same issues building 29.0.2, and #86 fixes the issue for me. So 👍 from me fwiw. ;)

@jimeh
Copy link
Owner

jimeh commented Jul 18, 2023

Thanks for the report and PR @ccqpein. I’ll try and find some time to have a proper look at this before the end of the week.

I have a feeling it’s a combination of new optional configure flags in emacs which automatically enable themselves if specific brew packages happen to be installed, along with what looks like a potential bug with the shared library embedding code in the build script.

@jimeh
Copy link
Owner

jimeh commented Jul 31, 2023

@ccqpein Apologies, it took me longer to get around to this than I thought. However I can't re-produce it on my own machines. I don't have any Apple Silicon machines available to me, but I don't think it's an Intel vs AS issue.

For what it's worth, in my pre-builds project, the 29.0.92-pretest build for example includes the libwebp7.dylib and libsharpyuv.0.dylib files just fine.

So I'm not fully sure what might be going on. The fact that it's linked to via @rpath, means the lib embedder logic detects it as being a shared library from within Homebrew, and hence copies it into the app bundle and updates the share library link. Possibly there was some random glitch while performing the file copy operation.

Since I can't re-produce it myself, I'm afraid I might need to ask you debug things a bit further. I have just pushed a new v0.6.42 of build-emacs-for-macos, which has a new --log-level flag. If you set the log level to debug, you'll get a lot more details printed towards the end when the various libraries embedding/bundling logic runs.

You can also try to use the --no-archive option to skip the archiving step at the end, leaving you with a folder under builds to speed things up. Then you can compare sources/**/nextstep/Emacs.app and builds/**/Emacs.app, the later of which will have had all the library embedding steps performed. Between the debug output and otool -L results for both apps, hopefully something might become obvious.

Let me know if there's anything more information I can give you, or help you with further debugging in general.

@ccqpein
Copy link
Author

ccqpein commented Aug 1, 2023

@jimeh no need to apologies. I am trying to rebuild with your update now. I will keep this issue updated. Thank you!

@ccqpein
Copy link
Author

ccqpein commented Aug 1, 2023

There are outputs of build-emacs-for-macos/builds/Emacs.2023-08-01.74d6604.master.macOS-13.arm64/Emacs.app/Contents/MacOS/Emacs and /build-emacs-for-macos/sources/emacs-mirror-emacs-74d6604/nextstep/Emacs.app/Contents/MacOS/Emacs with the --no-archive argument:

/build-emacs-for-macos/builds/Emacs.2023-08-01.74d6604.master.macOS-13.arm64/Emacs.app/Contents/MacOS/Emacs:
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 2299.50.120)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon (compatibility version 2.0.0, current version 169.0.0)
	/System/Library/Frameworks/IOSurface.framework/Versions/A/IOSurface (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore (compatibility version 1.2.0, current version 1.11.0)
	/System/Library/Frameworks/UniformTypeIdentifiers.framework/Versions/A/UniformTypeIdentifiers (compatibility version 1.0.0, current version 709.0.0)
	@rpath/libtiff.6.dylib (compatibility version 7.0.0, current version 7.1.0)
	@rpath/libjpeg.8.dylib (compatibility version 8.0.0, current version 8.3.2)
	@rpath/libpng16.16.dylib (compatibility version 57.0.0, current version 57.0.0)
	@rpath/libgif.dylib (compatibility version 0.0.0, current version 7.2.0)
	@rpath/libwebpdemux.2.dylib (compatibility version 3.0.0, current version 3.13.0)
	@rpath/libwebpdecoder.3.dylib (compatibility version 5.0.0, current version 5.7.0)
	/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit (compatibility version 1.0.0, current version 615.1.26)
	/usr/lib/libxml2.2.dylib (compatibility version 10.0.0, current version 10.9.0)
	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
	@rpath/libgnutls.30.dylib (compatibility version 66.0.0, current version 66.0.0)
	@rpath/liblcms2.2.dylib (compatibility version 3.0.0, current version 3.15.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	@rpath/libjansson.4.dylib (compatibility version 19.0.0, current version 19.0.0)
	@rpath/libgmp.10.dylib (compatibility version 15.0.0, current version 15.1.0)
	@rpath/libgccjit.0.dylib (compatibility version 0.0.0, current version 0.0.1)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
	@rpath/libtree-sitter.0.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libsqlite3.dylib (compatibility version 9.0.0, current version 346.0.0)
	/System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork (compatibility version 1.0.0, current version 1406.0.4)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1971.0.0)
	/System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics (compatibility version 64.0.0, current version 1690.5.4)
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1228.0.0)
	/System/Library/Frameworks/CoreText.framework/Versions/A/CoreText (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1971.0.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
/build-emacs-for-macos/sources/emacs-mirror-emacs-74d6604/nextstep/Emacs.app/Contents/MacOS/Emacs:
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 2299.50.120)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon (compatibility version 2.0.0, current version 169.0.0)
	/System/Library/Frameworks/IOSurface.framework/Versions/A/IOSurface (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore (compatibility version 1.2.0, current version 1.11.0)
	/System/Library/Frameworks/UniformTypeIdentifiers.framework/Versions/A/UniformTypeIdentifiers (compatibility version 1.0.0, current version 709.0.0)
	/opt/homebrew/opt/libtiff/lib/libtiff.6.dylib (compatibility version 7.0.0, current version 7.1.0)
	/opt/homebrew/opt/jpeg-turbo/lib/libjpeg.8.dylib (compatibility version 8.0.0, current version 8.3.2)
	/opt/homebrew/opt/libpng/lib/libpng16.16.dylib (compatibility version 57.0.0, current version 57.0.0)
	/opt/homebrew/opt/giflib/lib/libgif.dylib (compatibility version 0.0.0, current version 7.2.0)
	/opt/homebrew/opt/webp/lib/libwebpdemux.2.dylib (compatibility version 3.0.0, current version 3.13.0)
	/opt/homebrew/opt/webp/lib/libwebpdecoder.3.dylib (compatibility version 5.0.0, current version 5.7.0)
	/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit (compatibility version 1.0.0, current version 615.1.26)
	/usr/lib/libxml2.2.dylib (compatibility version 10.0.0, current version 10.9.0)
	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
	/opt/homebrew/opt/gnutls/lib/libgnutls.30.dylib (compatibility version 66.0.0, current version 66.0.0)
	/opt/homebrew/opt/little-cms2/lib/liblcms2.2.dylib (compatibility version 3.0.0, current version 3.15.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/opt/homebrew/opt/jansson/lib/libjansson.4.dylib (compatibility version 19.0.0, current version 19.0.0)
	/opt/homebrew/opt/gmp/lib/libgmp.10.dylib (compatibility version 15.0.0, current version 15.1.0)
	/opt/homebrew/opt/libgccjit/lib/gcc/current/libgccjit.0.dylib (compatibility version 0.0.0, current version 0.0.1)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
	/opt/homebrew/opt/tree-sitter/lib/libtree-sitter.0.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libsqlite3.dylib (compatibility version 9.0.0, current version 346.0.0)
	/System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork (compatibility version 1.0.0, current version 1406.0.4)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1971.0.0)
	/System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics (compatibility version 64.0.0, current version 1690.5.4)
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1228.0.0)
	/System/Library/Frameworks/CoreText.framework/Versions/A/CoreText (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1971.0.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

@ccqpein
Copy link
Author

ccqpein commented Aug 1, 2023

Then I searched the output of buildin. For libwebp, there is some log:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/{user}/build-emacs-for-macos/builds/Emacs.2023-08-01.74d6604.master.macOS-13.arm64/Emacs.app/Contents/MacOS/Emacs
==> DEBUG: -- Bundling shared library: /opt/homebrew/opt/webp/lib/libwebpdemux.2.dylib
==> DEBUG: -- -- Copying to: Contents/Frameworks/libwebpdemux.2.dylib
==> DEBUG: -- -- Relinking to: @rpath/libwebpdemux.2.dylib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/{user}/build-emacs-for-macos/builds/Emacs.2023-08-01.74d6604.master.macOS-13.arm64/Emacs.app/Contents/MacOS/Emacs
==> DEBUG: -- Bundling shared library: /opt/homebrew/opt/webp/lib/libwebpdecoder.3.dylib
==> DEBUG: -- -- Copying to: Contents/Frameworks/libwebpdecoder.3.dylib
==> DEBUG: -- -- Relinking to: @rpath/libwebpdecoder.3.dylib

But I don't see anything about libsharpyuv. Looks like both missing lib files are came from libwebp

@jimeh
Copy link
Owner

jimeh commented Aug 1, 2023

@ccqpein I believe I've tracked down the root of the issue. The original libwebpdemux.2.dylib file links to other webp shared libraries by using @rpath, which the library embedder does not understand at the moment. I'll try and get a fix in place soon.

@ccqpein
Copy link
Author

ccqpein commented Aug 13, 2023

I just build the Emacs master branch, and looks like there are some dylib missing too. But this time from @load_path

dyld[55871]: Library not loaded: @loader_path/../../../../opt/gettext/lib/libintl.8.dylib
  Referenced from: <A95699CE-2BA7-3EA0-A074-A47CAB443C4D> /Applications/Emacs.app/Contents/Frameworks/libgnutls.30.dylib
  Reason: tried: '/Applications/Emacs.app/Contents/Frameworks/../../../../opt/gettext/lib/libintl.8.dylib' (no such file), '/usr/local/lib/libintl.8.dylib' (no such file), '/usr/lib/libintl.8.dylib' (no such file, not in dyld cache)

Looks like it still the old load_path. The one in M2 chip mac should be

/Applications/Emacs.app/Contents/Frameworks/../../../../opt/homebrew/opt/gettext/lib/libintl.8.dylib
instead of
/Applications/Emacs.app/Contents/Frameworks/../../../../opt/gettext/lib/libintl.8.dylib

@jimeh
Copy link
Owner

jimeh commented Aug 14, 2023

Thanks for the details :)... Thankfully I've already come across that one too. It's because the shared library links to other shared libraries using @load_path, which confuses the custom shared library embedding/bundling code in the build script.

These issues have necessitated a rewrite of said code, and I have a nearly working version of it locally. I just need to find some more time this week to iron out a few bugs before it'll work properly.

@igorepst
Copy link

Hi @jimeh. Any news on this? :)
I have the following which prevents me from starting Emacs:

dyld[43042]: Library not loaded: @loader_path/../../../../opt/freetype/lib/libfreetype.6.dylib
  Referenced from: <C9101C0F-3D61-3F2C-8DBD-F79F68FD092A> /Applications/Emacs.app/Contents/Frameworks/libharfbuzz.0.dylib
  Reason: tried: '/Applications/Emacs.app/Contents/Frameworks/../../../../opt/freetype/lib/libfreetype.6.dylib' (no such file), '/usr/local/lib/libfreetype.6.dylib' (no such file), '/usr/lib/libfreetype.6.dylib' (no such file, not in dyld cache)Library not loaded: @loader_path/../../../../opt/icu4c/lib/libicui18n.73.dylib
  Referenced from: <FF684384-280D-3F59-8741-D9AE10E07338> /Applications/Emacs.app/Contents/Frameworks/libxml2.2.dylib
  Reason: tried: '/Applications/Emacs.app/Contents/Frameworks/../../../../opt/icu4c/lib/libicui18n.73.dylib' (no such file), '/usr/local/lib/libicui18n.73.dylib' (no such file), '/usr/lib/libicui18n.73.dylib' (no such file, not in dyld cache)Library not loaded: @loader_path/../../../../opt/gettext/lib/libintl.8.dylib
  Referenced from: <A95699CE-2BA7-3EA0-A074-A47CAB443C4D> /Applications/Emacs.app/Contents/Frameworks/libgnutls.30.dylib
  Reason: tried: '/Applications/Emacs.app/Contents/Frameworks/../../../../opt/gettext/lib/libintl.8.dylib' (no such file), '/usr/local/lib/libintl.8.dylib' (no such file), '/usr/lib/libintl.8.dylib' (no such file, not in dyld cache)

Thanks!

@ccqpein
Copy link
Author

ccqpein commented Sep 18, 2023

Hi @jimeh How is everything going? Do you need more test or log info I can help with? Thanks.

@garyo
Copy link

garyo commented Oct 11, 2023

I'm also experiencing this on my M1 Macbook Pro, running Ventura 13.5.2.
But the fix in #86 is not enough for me; with those two files copied into Frameworks I get this:

dyld[92706]: Library not loaded: @loader_path/../../../../opt/icu4c/lib/libicui18n.73.dylib
  Referenced from: <FF684384-280D-3F59-8741-D9AE10E07338> /Applications/Emacs.app/Contents/Frameworks/libxml2.2.dylib
  Reason: tried: '/Applications/Emacs.app/Contents/Frameworks/../../../../opt/icu4c/lib/libicui18n.73.dylib' (no such file), '/usr/local/lib/libicui18n.73.dylib' (no such file), '/usr/lib/libicui18n.73.dylib' (no such file, not in dyld cache)
Library not loaded: @loader_path/../../../../opt/gettext/lib/libintl.8.dylib
  Referenced from: <A95699CE-2BA7-3EA0-A074-A47CAB443C4D> /Applications/Emacs.app/Contents/Frameworks/libgnutls.30.dylib
  Reason: tried: '/Applications/Emacs.app/Contents/Frameworks/../../../../opt/gettext/lib/libintl.8.dylib' (no such file), '/usr/local/lib/libintl.8.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/usr/lib/libintl.8.dylib' (no such file, not in dyld cache)

and indeed libxml2.2.dylib looks for libicui18n.73.dylib in this path:

% otool -L /Applications/Emacs.app/Contents/Frameworks/libxml2.2.dylib
/Applications/Emacs.app/Contents/Frameworks/libxml2.2.dylib:
	@rpath/libxml2.2.dylib (compatibility version 14.0.0, current version 14.5.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	@loader_path/../../../../opt/icu4c/lib/libicui18n.73.dylib (compatibility version 73.0.0, current version 73.2.0)
	@loader_path/../../../../opt/icu4c/lib/libicuuc.73.dylib (compatibility version 73.0.0, current version 73.2.0)
	@loader_path/../../../../opt/icu4c/lib/libicudata.73.dylib (compatibility version 73.0.0, current version 73.2.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

@garyo
Copy link

garyo commented Oct 11, 2023

My build log (just the bundling part) is in this gist: https://gist.github.com/garyo/6a6dec778dfa98c35e4a03623df21ce6 in case it's helpful.

@jimeh
Copy link
Owner

jimeh commented Oct 11, 2023

Apologies for the radio silence everyone. Tail end of the summer got a bit crazy so I've not had any time available to look at this (and a few other things). Things are returning to normal this month though, so I'll be getting this sorted out soon.

@garyo
Copy link

garyo commented Oct 11, 2023

FWIW, I did these additional commands and now it's working on my M1 mac:

for libname in libicui18n.73.dylib libicuuc.73.dylib libicudata.73.dylib ; do 
  cp /opt/homebrew/Cellar/icu4c/73.2/lib/$libname builds/Emacs.2023-10-11.cfed3bb.master.macOS-13.arm64/Emacs.app/Contents/Framework; 
  install_name_tool -change @loader_path/../../../../opt/icu4c/lib/$libname @rpath/$libname builds/Emacs.2023-10-11.cfed3bb.master.macOS-13.arm64/Emacs.app/Contents/Frameworks/libxml2.2.dylib; 
  echo $libname done; 
  done

@ccqpein
Copy link
Author

ccqpein commented Oct 12, 2023

@jimeh Hope everything is fine. Let me know if you need more logs or test. Thanks!

@bklebe
Copy link

bklebe commented Oct 17, 2023

Is there any workaround for this issue? This appears to be the only Emacs build that offers casks with native-comp, I would love to help out here in any way I can.

@garyo
Copy link

garyo commented Oct 26, 2023

Just a quick note to say that @jimeh 's refactor-shared-library-bundler-logic branch is actually 100% working for me. I know the git log says it's not working yet and I'm sure there is work to be done still but a basic no-args build works fine on my M1 Macbook running Ventura. (I had to install a newer ruby via rbenv to avoid some build problems, but then the build worked fine.)

@ccqpein
Copy link
Author

ccqpein commented Oct 27, 2023

@garyo @jimeh Thanks for notification. I just tried branch refactor-shared-library-bundler-logic and it builds fine. And it starts fine without any crushes. (I just need to install the sudo gem install ruby-macho).

@jimeh
Copy link
Owner

jimeh commented Nov 5, 2023

@garyo @ccqpein That's great to hear, and thanks for testing that branch out. Last time I touched it, it was failing spectacularly for me. But it was middle of the night and I was half asleep, so I was probably doing something stupid... lol

I've got time set aside over the next couple of weeks to sort this issue out properly. And I'm also getting my first Apple Silicon machine this month, so I'll be able to properly try and document the process there too :)

@jimeh
Copy link
Owner

jimeh commented Nov 14, 2023

@garyo @ccqpein Thanks for your patience. I have updated #88 and fixed the outstanding issues on it, which was that it did not actually re-link anything after copying in all the relevant shared libraries into the Emacs.app bundle.

If you have time and are willing, feel free to try out the latest changes in #88. My plan is to wrap up and merge it tomorrow if no issues have cropped up with it.

The only catch on Apple Silicon is that you need to run codesign --force --deep -s - Emacs.app to ad-hoc self-sign the app. (I'll be automating that soon as a means of resolving #64). (this is now included in the same PR)

Personally I've tested Emacs builds from it which seem to fully work on completely fresh installs of macOS with only the Xcode Command Line Tools installed. And I've tested it on Apple Silicon, as I finally upgraded from an old Intel-mac last week.

Later this week my plan is to focus on getting the nightly builds repo and homebrew tap updated, along with ARM64 builds for Apple Silicon machines too.

@ccqpein
Copy link
Author

ccqpein commented Nov 16, 2023

@jimeh Thanks for you amazing work! #88 is good to me. I build and run perfectly without code sign. I think this issue can be deleted anytime. (I don't close it now for people who might searching the same issue and try to find the solution in issue)

@jimeh jimeh linked a pull request Nov 16, 2023 that will close this issue
@jimeh jimeh closed this as completed in #88 Nov 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
6 participants