CodeQL breaks iOS builds

[ls-valentinas-bakaitis]

Hi,

I'm trying to roll out CodeQL scanning to some of our iOS (swift) repositories and I have issues with CodeQL breaking the build steps. The same steps that succeed by themselves will fail when CodeQL init is added before them.

If I run this workflow with CodeQL step commented out, it succeeds:

name: CodeQL

on:
  push:
    branches: [ master, main ]
  pull_request:
    branches: [ master, main ]
  schedule:
  - cron: 30 12 * * 3
jobs:
  analyze:
    strategy:
      matrix:
        language: [ 'swift' ]
    runs-on: ['self-hosted', 'arm64', 'macOS', 'xcode-15.3']
    permissions:
      actions: read
      contents: read
      security-events: write
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Checkout vend actions
        uses: actions/checkout@v4
        with:
          repository: vend/github-actions
          ref: 'master'
          token: ${{ <redacted> }}
          path: xs-actions
#      - name: Initialize CodeQL
#        uses: github/codeql-action/init@v3
#        with:
#          languages: ${{ matrix.language }}
      - name: Setup Git and iOS dependencies
        uses: './xs-actions/.github/actions/ios_setup'
        with:
          install-gemfile-deps: "true"
          install-rbenv: "true"
          install-rosetta: "true"
          skip-xcode-macro-validation: 'true'
          ssh-private-key: ${{ <redacted> }}
      - name: Run fastlane test command
        uses: './xs-actions/.github/actions/ios_fastlane-run'
        with:
          fastlane-lane: 'test'

However once CodeQL init portion is uncommented I get this error in the "Setup Git and iOS dependencies" step:

Run /usr/sbin/softwareupdate --install-rosetta --agree-to-license
  /usr/sbin/softwareupdate --install-rosetta --agree-to-license
  shell: /bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    CODEQL_ACTION_FEATURE_MULTI_LANGUAGE: false
    CODEQL_ACTION_FEATURE_SANDWICH: false
    CODEQL_ACTION_FEATURE_SARIF_COMBINE: true
    CODEQL_ACTION_FEATURE_WILL_UPLOAD: true
    CODEQL_ACTION_VERSION: 3.25.10
    JOB_RUN_UUID: 9ffea00d-6649-4691-8625-ddbf1bf0db6b
    CODEQL_ACTION_INIT_HAS_RUN: true
    CODEQL_ACTION_ANALYSIS_KEY: .github/workflows/codeql-analysis-swift.yml:analyze
    CODEQL_WORKFLOW_STARTED_AT: 2024-06-18T02:34:54.572Z
    CODEQL_RAM: 14950
    CODEQL_THREADS: 8
    CODEQL_SCRATCH_DIR: /Users/admin/actions-runner/_work/_temp/codeql_databases/working
    CODEQL_VERBOSITY: warnings
    CODEQL_DIST: /Users/admin/actions-runner/_work/_tool/CodeQL/2.17.5/arm64/codeql
    CODEQL_PLATFORM: osx64
    CODEQL_PLATFORM_DLL_EXTENSION: .dylib
    CODEQL_JAVA_HOME: /Users/admin/actions-runner/_work/_tool/CodeQL/2.17.5/arm64/codeql/tools/osx64/java-aarch64
    CODEQL_EXTRACTOR_SWIFT_ROOT: /Users/admin/actions-runner/_work/_tool/CodeQL/2.17.5/arm64/codeql/swift
    CODEQL_EXTRACTOR_SWIFT_WIP_DATABASE: /Users/admin/actions-runner/_work/_temp/codeql_databases/swift
    CODEQL_EXTRACTOR_SWIFT_DIAGNOSTIC_DIR: /Users/admin/actions-runner/_work/_temp/codeql_databases/swift/diagnostic/extractors/swift
    CODEQL_EXTRACTOR_SWIFT_LOG_DIR: /Users/admin/actions-runner/_work/_temp/codeql_databases/swift/log
    CODEQL_EXTRACTOR_SWIFT_SCRATCH_DIR: /Users/admin/actions-runner/_work/_temp/codeql_databases/swift/working
    CODEQL_EXTRACTOR_SWIFT_TRAP_DIR: /Users/admin/actions-runner/_work/_temp/codeql_databases/swift/trap/swift
    CODEQL_EXTRACTOR_SWIFT_SOURCE_ARCHIVE_DIR: /Users/admin/actions-runner/_work/_temp/codeql_databases/swift/src
    CODEQL_EXTRACTOR_SWIFT_THREADS: 8
    CODEQL_EXTRACTOR_SWIFT_RAM: 14950
    CODEQL_TRACER_LOG: /Users/admin/actions-runner/_work/_temp/codeql_databases/log/build-tracer.log
    CODEQL_TRACER_DIAGNOSTICS_DIR: /Users/admin/actions-runner/_work/_temp/codeql_databases/diagnostic/tracer
    SEMMLE_COPY_EXECUTABLES_ROOT: /Users/admin/actions-runner/_work/_temp/codeql_databases/working/copy-root
    CODEQL_TOOL_PATH: /Users/admin/flutter:/Users/admin/flutter/bin/:/Users/admin/flutter/bin/cache/dart-sdk/bin:/Users/admin/.rbenv/shims:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Users/admin/android-sdk/cmdline-tools/latest/bin:/Users/admin/android-sdk/platform-tools:/Users/admin/android-sdk/emulator
    CODEQL_TRACER_LANGUAGES: swift
    SEMMLE_PRELOAD_libtrace: /Users/admin/actions-runner/_work/_tool/CodeQL/2.17.5/arm64/codeql/tools/osx64/libtrace.dylib
    CODEQL_RUNNER: /Users/admin/actions-runner/_work/_tool/CodeQL/2.17.5/arm64/codeql/tools/osx64/runner
    DYLD_INSERT_LIBRARIES: /Users/admin/actions-runner/_work/_tool/CodeQL/2.17.5/arm64/codeql/tools/osx64/libtrace.dylib
    SSH_AUTH_SOCK: /var/folders/zt/b4_8gf3n2wn8ylvm8wy7svc00000gn/T//ssh-sx7Xfm7bN6Bu/agent.1708
    SSH_AGENT_PID: 1711
Installing Rosetta 2 on this system is not supported.
Error: Process completed with exit code 1.

If I try to move CodeQL init after the "Setup Git and iOS dependencies" step, then the next step breaks - there are too many log lines to post here, but this is the error that occurs at fastlane step when CodeQL init is included just before it:

/Users/admin/actions-runner/_work/iOS.MobileSelling/iOS.MobileSelling/<redacted>/<redacted>:16:16: external macro implementation type 'DependenciesMacrosPlugin.DependencyClientMacro' could not be found for macro 'DependencyClient()'

    public let startAuthSession: @Sendable (_ domain: String) async throws -> AuthenticatedSession

It looks like CodeQL init is doing something that is making Rosetta 2 installation and subsequent fastlane compilation fail.

[mbg]

Hi @ls-valentinas-bakaitis 👋

Thanks for opening this issue. We rely on Rosetta 2 being installed on an arm-based macOS runners for CodeQL to work correctly. CodeQL should only be initialised as late as possible in your workflow (i.e. after all setup steps, but before the actual build starts), so putting it after your "Setup Git and iOS dependencies" step makes sense.

If I understand correctly, with that ordering, Rosetta 2 gets installed successfully?

We would probably need to see more of the logs to understand what's causing the issue in the fastlane step later on. If you can share more publicly here, then that would be great. Otherwise, you can open a support ticket referencing this issue and we can pick things up from there.

[ls-valentinas-bakaitis]

@mbg Thank you, it might be easier via a support ticket - I have opened one (ID 2847745), however I was unable to provide complete logs with it as they were too big and the ticket form wouldn't accept them. Please let me know on that ticket what is the best way forward.

[jakobholmgrenhiq]

If you don't mind, please share whatever solution you came up with in the support ticket publicly here as well if possible. I have very similar issues and have been following #2043 for a long time.

@mbg do you think I'd be better off opening my own support ticket?

Please let me know if you do not think my issue is related and I'll happily delete my comment to keep the issue clean and clear.

Cross-posting my message from that ticket here for convenience:

I am running into a similar issue when building with fastlane where my action gets stuck on the codesigning step only when CodeQL is initialized prior to building.

The failing command is the following:

set -o pipefail && xcodebuild -workspace ./REDACTED.xcodeproj/project.xcworkspace -scheme REDACTED -configuration QA-Release -destination 'generic/platform=iOS' -archivePath ./build.xcarchive archive | tee /Users/runner/Library/Logs/gym/REDACTED\ QA.log | xcbeautify

In my case, the action gets stuck indefinitely with the last readable output being the following:

[13:03:49]: ▸ Signing REDACTED.framework (in target 'REDACTED' from project 'REDACTED')

This step usually completes in seconds but will get stuck until the action times out or is cancelled. This does not happen for the exact same pipeline without CodeQL.

The initialization, build, and analysis steps looks as follows:

      - name: Initialize CodeQL
        uses: github/codeql-action/init@v3.22.12
        with:
          languages: swift
          queries: security-and-quality
          tools: https://github.com/github/codeql-action/releases/download/codeql-bundle-v2.15.5/codeql-bundle-osx64.tar.gz

      - name: Build QA
         REDACTED fastlane step

      - name: Perform CodeQL Analysis
        uses: github/codeql-action/analyze@v3.22.12
        with:
          category: "/language:swift"

Runner: macos-13
Xcode-version: 15.0.1

And later these logs were attached:

First match is unique:

[T 13:21:58 9563] Attempting to switch stdout/stderr to 6...
/Users/runner/work/_temp/codeql_databases/working/copy-root/000001F5/usr/bin/codesign.semmle.000023CF.0A82CBC0.slice.x86_64: replacing existing signature
/Users/runner/work/_temp/codeql_databases/working/copy-root/000001F5/Applications/Xcode_15.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang-stat-cache.semmle.000023CF.09925910.slice.x86_64: replacing existing signature
/Users/runner/work/_temp/codeql_databases/working/copy-root/000001F5/usr/bin/codesign.semmle.000023CF.0B6BD7E8.slice.x86_64: replacing existing signature
/Users/runner/work/_temp/codeql_databases/working/copy-root/000001F5/usr/bin/codesign.semmle.000023CF.0BE22128.slice.x86_64: replacing existing signature

Rest looks as follows with slight variations:

[T 13:21:58 9570] Initializing tracer.
[T 13:21:58 9570] Initialising tags...
[T 13:21:58 9570] ID set to 0000000000002562_0000000000000001 (parent 00000000000023CF_0000000000000001)
[T 13:21:58 9570] ==== Candidate to intercept: /usr/bin/codesign (canonical: /usr/bin/codesign) ====
[T 13:21:58 9570] Lua: === Intercepted call to /usr/bin/codesign ===
[T 13:21:58 9570] Lua: Disabling tracing for language swift.
[T 13:21:58 9570] Executing the following tracer actions:
[T 13:21:58 9570] Tracer actions:
[T 13:21:58 9570] pre_invocations(0)
[T 13:21:58 9570] post_invocations(0)
[T 13:21:58 9570] trace_languages(0): []
[T 13:21:58 9570] Disabling tracing for this command.
/Applications/Xcode_15.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/runner/work/_temp/codeql_databases/working/copy-root/000001F5/Applications/Xcode_15.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang-stat-cache.semmle.000023CF.09925910.slice.arm64
[T 13:21:58 9573] Initializing tracer.
[T 13:21:58 9573] Initialising tags...
[T 13:21:58 9573] ID set to 0000000000002565_0000000000000001 (parent 00000000000023CF_0000000000000001)
[T 13:21:58 9573] ==== Candidate to intercept: /usr/bin/codesign (canonical: /usr/bin/codesign) ====
[T 13:21:58 9573] Lua: === Intercepted call to /usr/bin/codesign ===
[T 13:21:58 9573] Lua: Disabling tracing for language swift.
[T 13:21:58 9573] Executing the following tracer actions:
[T 13:21:58 9573] Tracer actions:
[T 13:21:58 9573] pre_invocations(0)
[T 13:21:58 9573] post_invocations(0)
[T 13:21:58 9573] trace_languages(0): []
[T 13:21:58 9573] Disabling tracing for this command.
/Users/runner/work/_temp/codeql_databases/working/copy-root/000001F5/Applications/Xcode_15.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang-stat-cache.semmle.000023CF.09925910.slice.arm64: replacing existing signature

[mbg]

@jakobholmgrenhiq: 👋 No worries for asking here -- since you have been able to share your logs already, I don't currently see any reason for you to open a support ticket. I have read through the discussion in #2043 as well. Currently, it's unclear what the issue that @ls-valentinas-bakaitis is facing is, so I can't say whether you have the same issue or not. I will post an update once that one is resolved.

In the meantime, a few suggestions:

• Could you confirm whether you are running the above workflow on a GitHub-hosted runner or your own?
• If it is your own runner, you could try to (temporarily) disable SIP on it to rule out whether that interferes with something.

[ls-valentinas-bakaitis]

@mbg Hi, I have attached the logs to the support ticket (ID 2847745). Thanks!

[redsun82]

I see the zendesk ticket has been closed. Is this issue still relevant?

[ls-valentinas-bakaitis]

@redsun82 I didn't manage to reply to the ticket in time and it auto-closed. I have replied and reopened it now. This issue is still relevant and we still don't have CodeQL working in our iOS repositories.

[redsun82]

👋 @ls-valentinas-bakaitis sorry to hear that! Sorry if I'm a bit confused about the error of the build, as sifting through this issue I see three different errors being mentioned:

• Installing Rosetta 2 on this system is not supported (but I understand that was overcome by moving Initialize CodeQL after the setup step, which is a good move)
• a compile error: external macro implementation type 'DependenciesMacrosPlugin.DependencyClientMacro' could not be found for macro 'DependencyClient()'
• getting stuck on codesign

The latter might be worked around by passing CODE_SIGNING_REQUIRED=NO CODE_SIGNING_ALLOWED=NO to xcodebuild (which I'm not exactly sure how to do when using fastlane)

[ls-valentinas-bakaitis]

@redsun82 I am experiencing compile errors like this one:

external macro implementation type 'DependenciesMacrosPlugin.DependencyClientMacro' could not be found for macro 'DependencyClient()'

There are more like it in the log, but they all look similar. Note that when I remove the github/codeql-action/init step - the build works. So it seems like something that CpdeQL init action does is breaking the build.

I have attached the logs to the support ticket (ID 2847745) - these are private jobs in private repos so I don't want to share them publicly, but happy to share them privately.

[ls-valentinas-bakaitis]

@redsun82 another log entry that appears in our workflow logs is this:

2024-08-25T21:35:41.4434700Z Could not determine current commit SHA using git. Continuing with data from user input or environment. dyld[1492]: terminating because inserted dylib '/Users/admin/actions-runner/_work/_tool/CodeQL/2.18.2/arm64/codeql/tools/osx64/libtrace.dylib' could not be loaded: tried: '/Users/admin/actions-runner/_work/_tool/CodeQL/2.18.2/arm64/codeql/tools/osx64/libtrace.dylib' (fat file, but missing compatible architecture (have 'x86_64,arm64', need '')), '/System/Volumes/Preboot/Cryptexes/OS/Users/admin/actions-runner/_work/_tool/CodeQL/2.18.2/arm64/codeql/tools/osx64/libtrace.dylib' (no such file), '/Users/admin/actions-runner/_work/_tool/CodeQL/2.18.2/arm64/codeql/tools/osx64/libtrace.dylib' (fat file, but missing compatible architecture (have 'x86_64,arm64', need ''))
2024-08-25T21:35:41.4439430Z dyld[1492]: tried: '/Users/admin/actions-runner/_work/_tool/CodeQL/2.18.2/arm64/codeql/tools/osx64/libtrace.dylib' (fat file, but missing compatible architecture (have 'x86_64,arm64', need '')), '/System/Volumes/Preboot/Cryptexes/OS/Users/admin/actions-runner/_work/_tool/CodeQL/2.18.2/arm64/codeql/tools/osx64/libtrace.dylib' (no such file), '/Users/admin/actions-runner/_work/_tool/CodeQL/2.18.2/arm64/codeql/tools/osx64/libtrace.dylib' (fat file, but missing compatible architecture (have 'x86_64,arm64', need ''))

It doesn't seem to stop the run though, so I'm not sure if it's related or not 🤔

[redsun82]

hi @ls-valentinas-bakaitis thanks for reaching back.

It seems like there's a problem with injecting our tracer into some binaries (I see at least df and git being problematic in the logs). While this doesn't stop the build right away, there may be other instances preventing the build system from gathering correct information about the environment. On my machine both df and git are being reported by lipo -info as being fat binaries with x86_64 and arm64e architectures. While arm64e is problematic for us, if both architectures are present in the file this should not be a problem, so long as we get to run in x86_64 mode using Rosetta.

What I would check next, is

• the output of lipo -info $(which df) and lipo -info $(which git). If the two only have arm64e for some reason, then we have no chance of executing them within a traced build.
• whether you can force x86_64 mode by encapsulating the build in an arch -x86_64 command. I think this could also be achieved by using shell: arch -x86_64 bash -eu -o pipefail instead of shell: bash in the action steps triggering the build.

I'm afraid that if it's an arm64e compatibility problem, there's nothing much more we can do at this time: we're discussing workarounds or more principled solutions, and we want to have a better support for Apple silicon, but we don't have a concrete roadmap to get there yet.

[ls-valentinas-bakaitis]

@redsun82

both df and git seem to have x86 and arm architectures:

Architectures in the fat file: /bin/df are: x86_64 arm64e 
Architectures in the fat file: /usr/bin/git are: x86_64 arm64e 

I did try using the shell: arch -x86_64 bash -eu -o pipefail shell, but it seems like our build process doesn't work on x86 regardless if code scanning steps are added or not.

Thank you for your help confirming that it's not going to work at this time.

I understand that there is no easy fix, but Apple discontinued intel based machines a little while ago and we need a way to run code-ql on Apple silicon. Once GitHub starts working on this problem - I would be more than happy to help do some beta testing.