[FEATURE] provide setup.py for proper installation and linux distro packaging

[blshkv]

This is a common problem that many tools which we want to distribute do not support system wide installation. Some of these tools provide their own "install.sh" or "setup.sh" with a distro specific commands (sudo apt-install) to install dependencies instead of a proper setup.py.
With a proper setup.py script, an end user can install it using the following single command:
python3 setup.py install --user. It also a distro friendly and makes life for maintainers easier.

The end result, a FHS compliant directory structure should be created.
In case if software produce any output, a local user directory should be used, such as ~.//

There are some custom cases (installing non-python packages, data files etc) which have been resolved in other packages. Bellow, are reference points on such bug reports with solutions (patches)

pentoo/pentoo-overlay#596

[ajinabraham]

User specific install can be done by utilizing USE_HOME.

https://github.com/MobSF/Mobile-Security-Framework-MobSF/blob/master/MobSF/settings.py#L37

All data/config will be used from the user's HOME directory. These changes are done to support distros like Android Tamer, BlackArch etc.

I wish it is as simple as setup.py install but a lot of dependencies and their build requirements make this difficult.

[blshkv]

Thanks for highlighting USE_HOME. I'll utlise it.

But the main issue about setup.py is still there. setup.py handles dependencies properly, plus a proper installation/uninstallation of cause. EDIT: You can just use requirements.txt list in setup.py btw.

[ajinabraham]

I can create a setup.py with pythonic dependencies. But a user need to have other dependencies like these installed before running setup.py

# Install JDK 8+
sudo apt-get install openjdk-8-jdk
# Install the following dependencies
sudo apt install python3-dev python3-venv python3-pip build-essential libffi-dev libssl-dev libxml2-dev libxslt1-dev libjpeg8-dev zlib1g-dev wkhtmltopdf

Ref: https://mobsf.github.io/docs/#/requirements

[blshkv]

yes, that's correct. User would need to install these non-python packages manually. In fact, these instructions are distro specific, so users would need to adjust commands and package names

[ajinabraham]

Hi @blshkv setup.py is now available in the latest master.

once all the requirements here https://mobsf.github.io/docs/#/requirements are met,

you could do

pip install --no-index --find-links=scripts/wheels yara-python
python setup.py install

Migrate DB

mobsfdb

To run MobSF

mobsf 127.0.0.1:8000

Do let me know if this is what you are expecting.

[blshkv]

Ok, great progress thank you.

There is 1 more point to resolve:

• please do not lock dependencies in the requirements.txt file unnecessary. Currently, you require the latest and the greatest using == . However, it would be good if you could relax that requirement and require a specific version only if there is an API change and use >= at least

[blshkv]

also, I changed USE_HOME = True and ran mobsfdb, got the following error:

PermissionError: [Errno 13] Permission denied: '/usr/lib/python3.8/site-packages/mobsf/StaticAnalyzer/migrations'

[ajinabraham]

You do not have to change USE_HOME manually, the setup.py script will automatically change to USE_HOME=True before install.
I don't see the permission error with `usr/lib/python3.8/site-packages/mobsf/StaticAnalyzer/migrations'. This directory is created by django on migration. Not sure if you can change it.

I can change pinning for some of the dependencies. But not all of them.

[blshkv]

confirmed there is no "migration" directory, but mobsfdb is trying to run 0001_initial.py:

  /usr/lib/python3.8/site-packages/mobsf/StaticAnalyzer/migrations/0001_initial.py
    - Create model RecentScansDB
    - Create model StaticAnalyzerAndroid
    - Create model StaticAnalyzerIOS
    - Create model StaticAnalyzerWindows
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.8/mobsfdb", line 33, in <module>
    sys.exit(load_entry_point('mobsf==3.2.6', 'console_scripts', 'mobsfdb')())
  File "/usr/lib/python3.8/site-packages/mobsf/__main__.py", line 14, in db
    execute_from_command_line([
  File "/usr/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3.8/site-packages/django/core/management/base.py", line 85, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/django/core/management/commands/makemigrations.py", line 182, in handle
    self.write_migration_files(changes)
  File "/usr/lib/python3.8/site-packages/django/core/management/commands/makemigrations.py", line 213, in write_migration_files
    os.makedirs(migrations_directory, exist_ok=True)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/usr/lib/python3.8/site-packages/mobsf/StaticAnalyzer/migrations'
[INFO] 18/Jan/2021 04:03:14 - Checking for Update.
[WARNING] 18/Jan/2021 04:03:15 - A new version of MobSF is available, Please update to v3.2.7 Beta from master branch.

[ajinabraham]

So when you run mobsfdb, django tries to create the migration related python files under ...mobsf/StaticAnalyzer/migrations, but looks like there is not enough permission to write under site-packages.

[ajinabraham]

Also, I have unpinned the ones I can. https://github.com/MobSF/Mobile-Security-Framework-MobSF/blob/master/requirements.txt

We normally recommend users to install mobsf and related python dependencies in a venv.

[ajinabraham]

Another option is to pre-generate the migration before running setup.py install
You could do something like

python3 -m venv venv
. venv/bin/activate
(venv) pip install --no-cache-dir -r requirements.txt
(venv) python manage.py makemigrations
(venv) python manage.py makemigrations StaticAnalyzer
deactivate

Now you have the migrations related python files already created. You could do python setup.py install that will include migration files.

[blshkv]

Im packaging your tool for pentoo and can't afford venv for each tool ;-) . So we find the best combination where modules can be reused as it supposed to be

[ajinabraham]

I hope the new changes with requirements.txt can address it. The few remaining pinned ones are unfortunately required for API compatibility.

[blshkv]

I think you fixed everything here. I'm closing the ticket. Thanks a lot for all your work!

[ajinabraham]

Awesome 👍