qemu: `invalid signature (code or signature have been modified)` (regression in 8.0.4, fixed in 8.0.4-1, re-regression in 8.1.0) #140244

AkihiroSuda · 2023-08-23T05:28:05Z

`brew gist-logs <formula>` link OR `brew config` AND `brew doctor` output

$ brew config
HOMEBREW_VERSION: 4.1.6-8-g4564628
ORIGIN: https://github.com/Homebrew/brew
HEAD: 4564628eaf54264c9f3c69ba83c045ceb1742de2
Last commit: 17 hours ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 08799a75aac5d4bcee13abe6113063dae904861b
Core tap last commit: 5 hours ago
Core tap branch: master
Core tap JSON: 23 Aug 05:21 UTC
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: /private/tmp/com.apple.launchd.XXXXXXXXXX/org.xquartz:0
HOMEBREW_MAKE_JOBS: 8
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: octa-core 64-bit icelake
Clang: 14.0.3 build 1403
Git: 2.42.0 => /usr/local/bin/git
Curl: 8.1.2 => /usr/bin/curl
macOS: 13.5-x86_64
CLT: 14.3.1.0.1.1683849156
Xcode: 14.3

$ brew doctor
Your system is ready to brew.

Verification

My "brew doctor output" says Your system is ready to brew. and am still able to reproduce my issue.
I ran brew update and am still able to reproduce my issue.
I have resolved all warnings from brew doctor and that did not fix my problem.
I searched for recent similar issues at https://github.com/Homebrew/homebrew-core/issues?q=is%3Aissue and found no duplicates.

What were you trying to do (and why)?

/usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64 -accel hvf for running a QEMU machine with Hypervisor.framework (HVF)

What happened (include all command output)?

It crashes, as the signature on the binary is broken:

$ /usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64 -accel hvf
qemu-system-x86_64: -accel hvf: Error: HV_DENIED
Abort trap: 6

$ codesign --verify /usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64 
/usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64: invalid signature (code or signature have been modified)
In architecture: x86_64

What did you expect to happen?

The signature shouldn't be broken.

Step-by-step reproduction instructions (by running `brew` commands)

$ brew install qemu

$ /usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64 -accel hvf
qemu-system-x86_64: -accel hvf: Error: HV_DENIED
Abort trap: 6

$ codesign --verify /usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64 
/usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64: invalid signature (code or signature have been modified)
In architecture: x86_64

This was a regression in 8.0.4 bottle (#139409), and was once fixed in its rebuild 1, (#139492 , Homebrew/brew#15864), but broken again in 8.1.0.

Workarounds

Option 1: Downgrade QEMU to v8.0.3

brew uninstall qemu
curl -OSL https://raw.githubusercontent.com/Homebrew/homebrew-core/dc0669eca9479e9eeb495397ba3a7480aaa45c2e/Formula/qemu.rb
brew install ./qemu.rb

Option 2: Install QEMU from the source

brew uninstall qemu
brew install --build-from-source qemu

Option 3: Sign the QEMU binary locally

Lima v0.17.2 shows a prompt to suggest applying this workaround.

cat >entitlements.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.hypervisor</key>
    <true/>
</dict>
</plist>
EOF

codesign --sign - --entitlements entitlements.xml --force /usr/local/bin/qemu-system-$(uname -m | sed -e s/arm64/aarch64/)

The text was updated successfully, but these errors were encountered:

cho-m · 2023-08-23T06:40:13Z

I think the previous analysis in #139409 (comment) is most likely reason.

Qemu codesigns the binaries as part of Meson install (https://github.com/qemu/qemu/blob/v8.1.0/meson.build#L3852-L3854). Homebrew modifies the binaries (e.g. things like RPATH rewrites) invalidating it.

Most likely needs to be handled on brew side.

RPATH modification is one possible reason for invalidation. Looking at RPATH of bottles (pre-poured), the working binary uses placeholder @@HOMEBREW_PREFIX@@ which gets substituted with absolute path on pour while problematic binaries have been RPATH modified to use @loader_path.

Though, working binary may have issues on systems that use non-default prefix as the RPATH would be different than original CI path.

8.1.0:

❯ otool -L qemu/8.1.0/bin/qemu-system-x86_64 | head -5
qemu/8.1.0/bin/qemu-system-x86_64:
    /System/Library/Frameworks/Hypervisor.framework/Versions/A/Hypervisor (compatibility version 1.0.0, current version 1.0.0)
    @loader_path/../../../../opt/pixman/lib/libpixman-1.0.dylib (compatibility version 43.0.0, current version 43.2.0)
    @loader_path/../../../../opt/capstone/lib/libcapstone.4.dylib (compatibility version 4.0.0, current version 4.0.2)
    @loader_path/../../../../opt/libpng/lib/libpng16.16.dylib (compatibility version 57.0.0, current version 57.0.0)

8.0.4 rev 1

❯ otool -L qemu/8.0.4/bin/qemu-system-x86_64 | head -5
qemu/8.0.4/bin/qemu-system-x86_64:
    /System/Library/Frameworks/Hypervisor.framework/Versions/A/Hypervisor (compatibility version 1.0.0, current version 1.0.0)
    @@HOMEBREW_PREFIX@@/opt/pixman/lib/libpixman-1.0.dylib (compatibility version 43.0.0, current version 43.2.0)
    @@HOMEBREW_PREFIX@@/opt/capstone/lib/libcapstone.4.dylib (compatibility version 4.0.0, current version 4.0.2)
    @@HOMEBREW_PREFIX@@/opt/libpng/lib/libpng16.16.dylib (compatibility version 57.0.0, current version 57.0.0)

8.0.4

❯ otool -L qemu/8.0.4/bin/qemu-system-x86_64 | head -5
qemu/8.0.4/bin/qemu-system-x86_64:
    /System/Library/Frameworks/Hypervisor.framework/Versions/A/Hypervisor (compatibility version 1.0.0, current version 1.0.0)
    @loader_path/../../../../opt/pixman/lib/libpixman-1.0.dylib (compatibility version 43.0.0, current version 43.2.0)
    @loader_path/../../../../opt/capstone/lib/libcapstone.4.dylib (compatibility version 4.0.0, current version 4.0.2)
    @loader_path/../../../../opt/libpng/lib/libpng16.16.dylib (compatibility version 57.0.0, current version 57.0.0)

Local builds may also behave differently unless HOMEBREW_RELOCATABLE_INSTALL_NAMES is set.

See Homebrew/homebrew-core#140244.

AkihiroSuda · 2023-08-27T12:14:40Z

Fixed in 8.1.0-1, thank you

ngocphamm · 2023-08-27T12:25:44Z

Still doesn't seem to work for me. I have done.

colima delete
brew uninstall --ignore-dependencies qemu
rm -rf ~/Library/Caches/Homebrew/downloads/* to remove cached downloads

brew install qemu shows

==> Fetching qemu
==> Downloading https://ghcr.io/v2/homebrew/core/qemu/manifests/8.1.0-1
########################################################################################################################################### 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/qemu/blobs/sha256:2871c264e94c3e4e3dbb0cce11b697c6e2a5db9b18603b2d48eabfe0260c279b
########################################################################################################################################### 100.0%
==> Pouring qemu--8.1.0.ventura.bottle.1.tar.gz
🍺  /usr/local/Cellar/qemu/8.1.0: 167 files, 529.3MB
==> Running `brew cleanup qemu`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

Yet

~ »  codesign --verify /usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64
/usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64: invalid signature (code or signature have been modified)
In architecture: x86_64

My system

HOMEBREW_VERSION: 4.1.6
ORIGIN: https://github.com/Homebrew/brew
HEAD: 3cd72905ce3d36a84994cc6c6b9ebbdafff4e922
Last commit: 9 days ago
Core tap JSON: 27 Aug 12:15 UTC
HOMEBREW_PREFIX: /usr/local
HOMEBREW_BAT_THEME: base16-256
HOMEBREW_CASK_OPTS: []
HOMEBREW_EDITOR: nvim
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_MAKE_JOBS: 16
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: 16-core 64-bit kabylake
Clang: 14.0.3 build 1403
Git: 2.39.2 => /Library/Developer/CommandLineTools/usr/bin/git
Curl: 8.1.2 => /usr/bin/curl
macOS: 13.5.1-x86_64
CLT: 14.3.1.0.1.1683849156
Xcode: N/A

AkihiroSuda · 2023-08-27T12:29:54Z

Works for me 🤔

$ brew install qemu
==> Fetching qemu
==> Downloading https://ghcr.io/v2/homebrew/core/qemu/manifests/8.1.0-1
Already downloaded: /Users/suda/Library/Caches/Homebrew/downloads/2bb99c11ad687048433e4d0eef147b2bac43066c34cd0a3626bb134d4e9eeafd--qemu-8.1.0-1.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/qemu/blobs/sha256:2871c264e94c3
Already downloaded: /Users/suda/Library/Caches/Homebrew/downloads/8503bc0f0d222df99467e771c1ba3a939d93bea16b6beae890ef021d35daddfe--qemu--8.1.0.ventura.bottle.1.tar.gz
==> Pouring qemu--8.1.0.ventura.bottle.1.tar.gz
🍺  /usr/local/Cellar/qemu/8.1.0: 167 files, 529.3MB
==> Running `brew cleanup qemu`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

$ codesign --verify /usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64

$ echo $?
0

$ sha256sum /usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64
e75feafdd8ded67d220a3fe34bb098bb85420b0f386af50fa6bce1a1b5d53cbc  /usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64

$ brew config
HOMEBREW_VERSION: 4.1.6-30-gb0c5405
ORIGIN: https://github.com/Homebrew/brew
HEAD: b0c54058f30af1256114c787c8336cce9a8cc5d9
Last commit: 8 hours ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: bf6ac0be7aaf7ef66d89e8e7683f926ae981fcfa
Core tap last commit: 81 minutes ago
Core tap branch: master
Core tap JSON: 27 Aug 12:27 UTC
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: /private/tmp/com.apple.launchd.DENWfo8GI6/org.xquartz:0
HOMEBREW_MAKE_JOBS: 8
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: octa-core 64-bit icelake
Clang: 14.0.3 build 1403
Git: 2.42.0 => /usr/local/bin/git
Curl: 8.1.2 => /usr/bin/curl
macOS: 13.5-x86_64
CLT: 14.3.1.0.1.1683849156
Xcode: 14.3

ngocphamm · 2023-08-27T12:37:14Z

@AkihiroSuda I notice a couple (major?) differences in our set up I guess. Not sure what is causing the issue.

HOMEBREW_VERSION. You seems to be connected to HEAD?
Xcode. I don't have it installed. I just install the CommandLineTools

I don't think mine have the same sha256 though

~ »  sha256sum /usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64
d0afabef422f043183620c96a494f7fd60aa147849de454c0317a5aedb1d2e40  /usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64

rfay · 2023-08-27T12:37:19Z

Does it work for you @AkihiroSuda because Already downloaded: /Users/suda/Library/Caches/Homebrew/downloads/8503bc0f0d222df99467e771c1ba3a939d93bea16b6beae890ef021d35daddfe--qemu--8.1.0.ventura.bottle.1.tar.gz - key difference between the two pastes.

And THANKS for all your work shepherding this fix through.

AkihiroSuda · 2023-08-27T12:39:40Z

@ngocphamm brew update may work?

ngocphamm · 2023-08-27T13:15:07Z

@AkihiroSuda Unforutunately no 😢

brew uninstall --ignore-dependencies qemu -> remove cached download files -> brew update -> brew install qemu. I still get the same sha256 as I did before.

cho-m · 2023-08-27T16:25:35Z

This time, it looks like bottle is using placeholders again (not sure why):

❯ otool -L bin/qemu-system-x86_64 | head -5
bin/qemu-system-x86_64:
	/System/Library/Frameworks/Hypervisor.framework/Versions/A/Hypervisor (compatibility version 1.0.0, current version 1.0.0)
	@@HOMEBREW_PREFIX@@/opt/pixman/lib/libpixman-1.0.dylib (compatibility version 43.0.0, current version 43.2.0)
	@@HOMEBREW_PREFIX@@/opt/capstone/lib/libcapstone.4.dylib (compatibility version 4.0.0, current version 4.0.2)
	@@HOMEBREW_PREFIX@@/opt/libpng/lib/libpng16.16.dylib (compatibility version 57.0.0, current version 57.0.0)

Which means some RPATH updates (specifically placeholder substitutions) are happening on installation, e.g. try with --debug --verbose and you should see things like following (second part only if on newer/unreleased brew, which means signatures are invalid in bottle and need to be resigned):

==> Changing install name in /usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64
  from @@HOMEBREW_PREFIX@@/opt/pixman/lib/libpixman-1.0.dylib
    to /usr/local/opt/pixman/lib/libpixman-1.0.dylib
/usr/bin/env codesign --verify /usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64
==> Codesigning /usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64
/usr/bin/env codesign --display --file-list - /usr/local/Cellar/qemu/8.1.0/bin/qemu-system-x86_64

This does mean you will need my commit Homebrew/brew#15903 which is only available in HEAD of brew and not in a release.

Bo98 · 2023-08-27T17:44:30Z

This time, it looks like bottle is using placeholders again (not sure why):

That will be because we don't have HOMEBREW_RELOCATABLE_INSTALL_NAMES in the rebottle workflow.

Tbh I'd be fine with a revision bump here if it's affecting a number of people.

alexanderxc · 2023-08-27T23:56:02Z

@AkihiroSuda is there a plan to bring this solution for users not on HEAD? I agree with @Bo98, a version bump might be the quickest solution to avoid manual codesign?

Thanks in advance.

AkihiroSuda · 2023-08-28T00:18:00Z

@AkihiroSuda is there a plan to bring this solution for users not on HEAD? I agree with @Bo98, a version bump might be the quickest solution to avoid manual codesign?

Thanks in advance.

I can't answer, as I'm not a maintainer.

Reopening the issue to get attention from maintainers.

cho-m · 2023-08-28T00:24:33Z

New brew tags/releases may take some time to make sure all changes are stable. Someone up-to-date on latest merged brew PRs will need to comment on state particularly if any backwards incompatible changes were added.

We can just bump qemu revision for now. Can also have separate PR(s) to add HOMEBREW_RELOCATABLE_INSTALL_NAMES to other workflows.

EDIT: Though, if there are any other modifications that happen at install-time that may invalidate code signature, then may still need new brew release.

cho-m · 2023-08-28T02:14:54Z

Closing as should hopefully be fixed with #140643 (binaries have been re-signed in bottle with @loader_path RPATHs). Didn't see any binary modifications happening during pour this time. Can let us know if anyone is still seeing an issue.

chenrui333 · 2023-08-28T03:43:28Z

Can someone on this thread verify the latest bottle and confirm the fix? Thanks! 🙏

MikeMcQuaid · 2023-08-28T07:35:32Z

New brew tags/releases may take some time to make sure all changes are stable

There is a new tag (4.1.7) including this released now.

alexanderxc · 2023-08-28T07:44:11Z

I can confirm solution for qemu is working.

Thanks for being so fast.

AkihiroSuda added the bug Reproducible Homebrew/homebrew-core bug label Aug 23, 2023

xRomZak mentioned this issue Aug 23, 2023

qemu exited unexpectedly with exit code -1, stderr: qemu-system-x86_64: Error: HV_DENIED containers/podman#19708

Closed

cho-m mentioned this issue Aug 23, 2023

extend/os/mac/keg: codesign on Intel if invalid signature Homebrew/brew#15903

Merged

7 tasks

tamird added a commit to aya-rs/aya that referenced this issue Aug 23, 2023

github: build qemu from source if bad signature

b91d90d

See Homebrew/homebrew-core#140244.

AkihiroSuda mentioned this issue Aug 25, 2023

Colima not starting "FATA[0002] error starting vm: error at 'starting': exit status 1" abiosoft/colima#786

Closed

5 tasks

rfay mentioned this issue Aug 25, 2023

Colima/Lima broken and unusable on Intel/amd64 ddev/ddev#5298

Closed

1 task

AkihiroSuda closed this as completed Aug 27, 2023

AkihiroSuda reopened this Aug 28, 2023

cho-m mentioned this issue Aug 28, 2023

qemu: revision bump to re-codesign Intel binaries #140643

Merged

6 tasks

cho-m closed this as completed Aug 28, 2023

shamil-mubarakshin mentioned this issue Sep 7, 2023

colima start no longer works on macos-12 runner 20230812.3 actions/runner-images#8104

Closed

10 tasks

github-actions bot added the outdated PR was locked due to age label Sep 28, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

qemu: `invalid signature (code or signature have been modified)` (regression in 8.0.4, fixed in 8.0.4-1, re-regression in 8.1.0) #140244

qemu: `invalid signature (code or signature have been modified)` (regression in 8.0.4, fixed in 8.0.4-1, re-regression in 8.1.0) #140244

AkihiroSuda commented Aug 23, 2023 •

edited

Loading

cho-m commented Aug 23, 2023

AkihiroSuda commented Aug 27, 2023

ngocphamm commented Aug 27, 2023

AkihiroSuda commented Aug 27, 2023

ngocphamm commented Aug 27, 2023

rfay commented Aug 27, 2023

AkihiroSuda commented Aug 27, 2023

ngocphamm commented Aug 27, 2023

cho-m commented Aug 27, 2023 •

edited

Loading

Bo98 commented Aug 27, 2023 •

edited

Loading

alexanderxc commented Aug 27, 2023

AkihiroSuda commented Aug 28, 2023

cho-m commented Aug 28, 2023 •

edited

Loading

cho-m commented Aug 28, 2023

chenrui333 commented Aug 28, 2023

MikeMcQuaid commented Aug 28, 2023

alexanderxc commented Aug 28, 2023

qemu: invalid signature (code or signature have been modified) (regression in 8.0.4, fixed in 8.0.4-1, re-regression in 8.1.0) #140244

qemu: invalid signature (code or signature have been modified) (regression in 8.0.4, fixed in 8.0.4-1, re-regression in 8.1.0) #140244

Comments

AkihiroSuda commented Aug 23, 2023 • edited Loading

brew gist-logs <formula> link OR brew config AND brew doctor output

Verification

What were you trying to do (and why)?

What happened (include all command output)?

What did you expect to happen?

Step-by-step reproduction instructions (by running brew commands)

Workarounds

Option 1: Downgrade QEMU to v8.0.3

Option 2: Install QEMU from the source

Option 3: Sign the QEMU binary locally

cho-m commented Aug 23, 2023

AkihiroSuda commented Aug 27, 2023

ngocphamm commented Aug 27, 2023

AkihiroSuda commented Aug 27, 2023

ngocphamm commented Aug 27, 2023

rfay commented Aug 27, 2023

AkihiroSuda commented Aug 27, 2023

ngocphamm commented Aug 27, 2023

cho-m commented Aug 27, 2023 • edited Loading

Bo98 commented Aug 27, 2023 • edited Loading

alexanderxc commented Aug 27, 2023

AkihiroSuda commented Aug 28, 2023

cho-m commented Aug 28, 2023 • edited Loading

cho-m commented Aug 28, 2023

chenrui333 commented Aug 28, 2023

MikeMcQuaid commented Aug 28, 2023

alexanderxc commented Aug 28, 2023

qemu: `invalid signature (code or signature have been modified)` (regression in 8.0.4, fixed in 8.0.4-1, re-regression in 8.1.0) #140244

qemu: `invalid signature (code or signature have been modified)` (regression in 8.0.4, fixed in 8.0.4-1, re-regression in 8.1.0) #140244

AkihiroSuda commented Aug 23, 2023 •

edited

Loading

`brew gist-logs <formula>` link OR `brew config` AND `brew doctor` output

Step-by-step reproduction instructions (by running `brew` commands)

cho-m commented Aug 27, 2023 •

edited

Loading

Bo98 commented Aug 27, 2023 •

edited

Loading

cho-m commented Aug 28, 2023 •

edited

Loading