Installing semgrep via pypi or GitHub on M1 Pro, Max, etc. (arm64) Mac installs semgrep-core compiled for x86_64

[j-martin]

Describe the bug

Arguably a feature request

We use pip to install semgrep in our dev setup to ensure everybody uses the same version and therefore gets replicable ouput. Unfortunately, currently, there is no wheel for macOS arm64 so pip installs x86-64 binaries.

https://pypi.org/project/semgrep/#files

The same issue with the artifacts published to github.

brew install semgrep works fine because semgrep-core is compiled for arm64 (unless I am missing something). Unfortunately, the pinning of versions with brew is not great and we also have a mix of Linux users.

# Semgrep installed via pip ---> x86-64

$ file /Users/jm/Library/Caches/pypoetry/virtualenvs/semgrep-standalone-vCBicEn
N-py3.9/lib/python3.9/site-packages/semgrep/bin/semgrep-core
/Users/jm/Library/Caches/pypoetry/virtualenvs/semgrep-standalone-vCBicEnN-py3.9/lib/python3.9/site-packages/semgrep/bin/semgrep-core: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped

# Semgrep installed via brew --> arm64

$ file /opt/homebrew/Cellar/semgrep/0.73.0/libexec/lib/python3.9/site-packages/semgrep/bin/semgrep-core
/opt/homebrew/Cellar/semgrep/0.73.0/libexec/lib/python3.9/site-packages/semgrep/bin/semgrep-core: Mach-O 64-bit executable arm64

To Reproduce

1. On a M1 based (arm64) machine, run pip install semgrep.
2. Running a semgrep --lang java -e 'DSL.or(...)' inside the virtualenv.

$ /Users/jm/Library/Caches/pypoetry/virtualenvs/semgrep-standalone-vCBicEnN-
py3.9/bin/semgrep --lang java -e 'DSL.or(...)'
Traceback (most recent call last):
  File "/Users/jm/Library/Caches/pypoetry/virtualenvs/semgrep-standalone-vCBicEnN-py3.9/bin/semgrep", line 8, in <module>
    sys.exit(main())
  File "/Users/jm/Library/Caches/pypoetry/virtualenvs/semgrep-standalone-vCBicEnN-py3.9/lib/python3.9/site-packages/semgrep/__main__.py", line 20, in main
    cli()
  File "/Users/jm/Library/Caches/pypoetry/virtualenvs/semgrep-standalone-vCBicEnN-py3.9/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jm/Library/Caches/pypoetry/virtualenvs/semgrep-standalone-vCBicEnN-py3.9/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/jm/Library/Caches/pypoetry/virtualenvs/semgrep-standalone-vCBicEnN-py3.9/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/jm/Library/Caches/pypoetry/virtualenvs/semgrep-standalone-vCBicEnN-py3.9/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/jm/Library/Caches/pypoetry/virtualenvs/semgrep-standalone-vCBicEnN-py3.9/lib/python3.9/site-packages/semgrep/cli.py", line 683, in cli
    semgrep.semgrep_main.main(
  File "/Users/jm/Library/Caches/pypoetry/virtualenvs/semgrep-standalone-vCBicEnN-py3.9/lib/python3.9/site-packages/semgrep/semgrep_main.py", line 202, in main
    ) = CoreRunner(
  File "/Users/jm/Library/Caches/pypoetry/virtualenvs/semgrep-standalone-vCBicEnN-py3.9/lib/python3.9/site-packages/semgrep/core_runner.py", line 414, in invoke_semgrep
    ) = self._run_rules_direct_to_semgrep_core(rules, target_manager, profiler)
  File "/Users/jm/Library/Caches/pypoetry/virtualenvs/semgrep-standalone-vCBicEnN-py3.9/lib/python3.9/site-packages/semgrep/core_runner.py", line 354, in _run_rules_direct_to_semgrep_core
    core_run = sub_run(
  File "/Users/jm/Library/Caches/pypoetry/virtualenvs/semgrep-standalone-vCBicEnN-py3.9/lib/python3.9/site-packages/semgrep/util.py", line 133, in sub_run
    result = subprocess.run(cmd, **kwargs)  # nosem: python.lang.security.audit.dangerous-subprocess-use.dangerous-subprocess-use
  File "/Users/jm/.pyenv/versions/3.9.7/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Users/jm/.pyenv/versions/3.9.7/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/jm/.pyenv/versions/3.9.7/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '


# Executing semgrep-core directly on M1

$ /Users/jm/Library/Caches/pypoetry/virtualenvs/semgrep-standalone-vCBicEnN-py3.9/lib/python3.9/site-packages/semgrep/bin/semgrep-core
zsh: exec format error: /Users/jm/Library/Caches/pypoetry/virtualenvs/semgrep-standalone-vCBicEnN-py3.9/lib/python3.9/site-packages/semgrep/bin/semgrep-core

Expected behavior

There should be a wheel built for arm64 too.

Screenshots

What is the priority of the bug to you?

• P0: blocking your adoption of Semgrep or workflow
• P1: important to fix or quite annoying
• P2: regular bug that should get fixed

Right now, people on M1 machines have to install whatever latest version of semgrep brew wants to install. Using rosetta is not really an option since the python executable is compiled for arm64.

Environment
pypi on python 3.9.7

[j-martin]

I updated the ticket to mention that the artifacts on GitHub are also x86-64 only. https://github.com/returntocorp/semgrep/releases/tag/v0.74.0

[aryx]

Is this fixed @brendongo or should we lower its priority?

[j-martin]

@brendongo is there any timeline on fixing this?

[jesper]

Can confirm that this issue still persists in v0.77.0 of sempgrep when installed via pip on M1 macs.

[brendongo]

@j-martin Probably will have time for this in the second half of Feb? Currently busy with porting some semgrep-action functionality into semgrep that should be wrapped up by then.

[j-martin]

Sounds good. Thanks for the update. :)