-
Notifications
You must be signed in to change notification settings - Fork 586
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
semgrep thows exception on python setup.py install #2054
Comments
Setup in question https://github.com/MobSF/Mobile-Security-Framework-MobSF/blob/setup/setup.py Repro python3 -m venv v
. v/bin/activate
python setup.py install |
Thanks for the heads up @ajinabraham - we'll look into this shortly 👍 |
Was able to reproduce this locally but not entirely sure what's happening. @mschwager do you know have any ideas why python is trying to read spacegrep as utf-8 :D |
Was able to reproduce this with semgrep==0.27.0 so it's not just spacegrep, python tries to read semgrep-core as utf-8 as well. |
Might be relevant: https://github.com/pypa/setuptools/blob/master/setuptools/command/easy_install.py#L591 |
Found it. Looks like we were originally installing to The recommendation is to move over to Then, easy_install makes assumptions about things in the |
This Could also consider an upstream patch to easy_install 🤷♂️ |
Indeed, if I rename (venv) ➜ site-packages git:(master) ✗ find . -type d -name 'scripts'
./semgrep-0.32.0-py3.8.egg/EGG-INFO/scripts
# RENAME TO bin/
(venv) ➜ site-packages git:(master) ✗ find . -type d -name 'bin'
./semgrep-0.32.0-py3.8.egg/EGG-INFO/bin
(venv) ➜ pyspider git:(master) ✗ python3 setup.py install
# ...
Searching for semgrep==0.32.0
Best match: semgrep 0.32.0
Processing semgrep-0.32.0-py3.8.egg
semgrep 0.32.0 is already the active version in easy-install.pth
Installing semgrep script to /Users/grayson/sandbox/pyspider/venv/bin
# ...
Finished processing dependencies for pyspider==0.3.2
(venv) ➜ pyspider git:(master) ✗ |
Huzzah! Nice work @minusworld |
Heh, thanks -- we still actually have to solve the issue 🤷♂️ |
@ajinabraham As a temporary workaround, you may try installing without easy_install via another method. This isn't ideal, so we will keep working on this issue. @brendongo @DrewDennison Any suggestions for how to proceed? I see two main options:
|
This is out of my control. This came as a requirement to install and package MobSF(which uses libsast that has a requirement on semgrep) on a pentesting distribution. The maintainer of that project expects a setup.py like most tools. MobSF/Mobile-Security-Framework-MobSF#1585 (comment) |
Note sure if it's relevant, but have you looked into bundling data files or binaries using MANIFEST.in ? |
OK gotcha. MANIFEST.in looks like it could be an option. 👍 We'll look into it and see if it works. |
@minusworld will we be able to try this solution by 0.35? |
TL;DR: After further investigation I can't find a way to make both
The execute bit problem means we receive a So, I see a few options:
Perhaps it'd be best to pull in @blshkv and see what's feasible. Is there a reason you cannot use |
pip is a higher-level interface on top of setuptools or distribute, it might even call setup.py (never used pip, so not sure). So please focus on setuptools solution |
I guess semgrep needs to support pip as that's the easiest and widely used way for people to install semgrep. Other tools that integrate semgrep also have to install semgrep which may not use pip at all. @mschwager In the ideal scenario, instead of semgrep-core as a binary in a python package, there should have been shared objects targeted for multiple arch and OS. So I wouldn't see any issues with setting execute bit as a part of the install process and not a post or manual process. |
Can you point me to where you're seeing the original issue? I can reproduce it, but I can't see where/why this is an issue vs. simply using As mentioned above in my testing, We could package |
@mschwager @blshkv can give more information on that. He had a use case of integrating/bundling tools with Pentoo, a security-focused Linux Distro. For python projects, the standard approach followed by the project is |
You can also have a look at https://snarky.ca/what-the-heck-is-pyproject-toml/ for the complete picture. |
I think the closest software is frida where we solved а similar problem, see: I'm happy with the things as they are now (0.28.0) in general. |
Thanks for the additional information!
Can you point me to where this is being done? That will help with my testing and learning about this issue. I'm trying to brainstorm the best solution here. Would bringing your own
Would that installation workflow work for you? Unfortunately, splitting Semgrep into two packages and using extension modules is more work then we can take on right now. |
At Pentoo/Gentoo we build all software from source usually and create ebuilds using bash (pretty much) for each package. Here is the current Pentoo implementation:
|
Hi @blshkv, I've just merged #2326, which should fix this. This change will go out in the next release in the next few days. The install process should remain similar, but you'll have to make a few changes (note we now have two binaries:
We've also made the |
@blshkv, heads up, I've confirmed the fix, which went out in
If you don't include
If you don't have
Note that we now include an additional binary called |
I have adjusted our ebuilds, SEMGREP_SKIP_BIN works fine. Thanks a lot guys |
Thanks, I was too lazy to ask for it ;-) |
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior, ideally a link to https://semgrep.dev:
Add semgrep to a setup.py script and run
python setup.py install
Expected behavior
install gracefully
Screenshots
If applicable, add screenshots to help explain your problem.
What is the priority of the bug to you?
NA
Environment
pypi, 0.31.1
The text was updated successfully, but these errors were encountered: