custom module not loading

[phate1]

Description

ive tried my hand a writing a custom module but seems im doing something stupid as i can't get fame to load it. i tried adding it into the community folder and didnt see it, tried adding a custom folder in the modules folder which also didn't work. ive now uploaded to github and adding it as a repo it clones the repo fine but still doesn't pick up the module.
im not sure what criteria decides if it gets processed or not maybe ive messed something up on the folder structure or something?

any pointers welcome
folder structure
pete@fame:~/fame/fame/modules/private $ tree .
.
├── init.py
├── processing
│   ├── init.py
│   ├── pycache
│   └── yara_proc
│   ├── details.html
│   ├── init.py
│   ├── pycache
│   │   └── yara_proc.cpython-38.pyc
│   ├── requirements.txt
│   └── yara_proc.py
└── pycache

Steps to Reproduce

add custom repo with processing module

Expected behavior

module available in fame

Actual behavior

no mention of the module in worker logs while starting or reloading from ui not available to enable in ui

Debug

seem to have a issue with the mongo auth:
had to add:
"from fame.core import fame_init"
and fame_init() otherwise i got an auth error running the script

pete@fame:~/fame$ utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.

Traceback (most recent call last):
File "utils/troubleshoot.py", line 7, in
from fame.core import fame_init
ModuleNotFoundError: No module named 'fame'
pete@fame:/fame$ vi utils/troubleshoot.py
pete@fame:/fame$ utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.

########## VERSION ##########

OS: Linux-5.4.0-124-generic-x86_64-with-glibc2.29
Python: 3.8.10

########## DEPENDENCIES ###########

WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see pypa/pip#5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
alabaster==0.7.12
amqp==2.6.1
androguard==3.3.5
appdirs==1.4.4
asn1crypto==1.5.1
asttokens==2.0.7
Babel==2.10.3
backcall==0.2.0
backports.zoneinfo==0.2.1
beautifulsoup4==4.11.1
billiard==3.6.4.0
bs4==0.0.1
capstone==4.0.2
celery==4.4.7
certifi==2022.6.15
cffi==1.15.1
chardet==4.0.0
click==8.0.3
colorama==0.3.7
colorclass==2.2.2
compressed-rtf==1.0.6
cryptography==37.0.4
cxxfilt==0.2.2
cycler==0.11.0
decorator==4.4.2
defang==0.5.3
distlib==0.3.5
docker==4.4.0
docutils==0.16
easygui==0.98.3
ebcdic==1.1.1
executing==0.9.1
extract-msg==0.36.1
fasttext==0.9.2
filelock==3.8.0
flare-capa==3.0.2
Flask==2.1.3
Flask-Classful==0.14.2
Flask-Login==0.5.0
flask-paginate==0.7.1
fonttools==4.34.4
funcy==1.16
future==0.18.2
gitdb==4.0.9
GitPython==3.1.27
googleplay-api==0.1.0
halo==0.0.31
hatching-triage==0.1.7
hexdump==3.3
ida-netnode==3.0
ida-settings==2.1.0
idna==2.10
ijson==3.1.4
imagesize==1.4.1
IMAPClient==2.3.1
importlib-metadata==4.12.0
intervaltree==3.1.0
ipython==8.4.0
itsdangerous==2.1.2
javaobj-py3==0.4.3
jbxapi==3.18.0
jedi==0.18.1
Jinja2==3.0.3
joblib==0.16.0
jsbeautifier==1.6.2
kiwisolver==1.4.4
kombu==4.6.11
lark-parser==0.12.0
libvirt-python==7.1.0
lief==0.11.0
lightgbm==3.3.0
log-symbols==0.0.14
lxml==4.9.1
malwareconfig==1.0.4
markdown2==2.3.10
MarkupSafe==2.1.1
matplotlib==3.5.3
matplotlib-inline==0.1.3
msgpack==1.0.4
msoffcrypto-tool==4.11.0
networkx==2.5.1
numpy==1.23.1
olefile==0.46
oletools==0.56
packaging==21.3
parso==0.8.3
pbkdf2==1.3
pcodedmp==1.2.6
peepdf==0.4.2
pefile==2021.9.3
pexpect==4.8.0
pickleshare==0.7.5
Pillow==3.2.0
platformdirs==2.5.2
prompt-toolkit==3.0.30
protobuf==4.21.5
ptyprocess==0.7.0
pure-eval==0.2.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybind11==2.10.0
pycparser==2.21
pycrypto==2.6.1
pycryptodomex==3.15.0
pydot==1.4.2
pyelftools==0.27
Pygments==2.12.0
pymongo==3.11.4
pyparsing==2.4.7
python-dateutil==2.8.1
python-flirt==0.6.3
python-magic==0.4.27
pythonaes==1.0
pytz==2022.1
pytz-deprecation-shim==0.1.0.post0
PyYAML==5.4.1
pyzipper==0.3.6
requests==2.25.1
RTFDE==0.0.2
ruamel.yaml==0.17.16
ruamel.yaml.clib==0.2.6
scikit-learn==0.23.2
scipy==1.9.0
six==1.16.0
smda==1.6.2
smmap==5.0.0
snowballstemmer==2.2.0
sortedcontainers==2.4.0
soupsieve==2.3.2.post1
Sphinx==3.2.1
sphinx-rtd-theme==0.5.2
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-httpdomain==1.7.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
spinners==0.0.24
stack-data==0.3.0
stringsifter==2.20201202
tabulate==0.8.9
termcolor==1.1.0
threadpoolctl==3.1.0
tqdm==4.62.3
traitlets==5.3.0
typing==3.7.4.3
tzdata==2022.1
tzlocal==4.2
urllib3==1.25.11
vine==1.3.0
virtualenv==20.13.4
virustotal-api==1.1.11
viv-utils==0.6.6
vivisect==1.0.5
volatility3==2.0.1
wcwidth==0.2.5
websocket-client==1.3.3
Werkzeug==2.0.3
yara-python==4.0.2
zipp==3.8.1
zxcvbn==4.4.28

########## MongoDB ##########

Version: 6.0.0
Authorization check: True

########## Configuration ##########

types: True
comments: True
extracted: True
email: False
malware_config: False
volatility: True

Modules:

McAfee Antivirus Disabled Configured
Sophos Antivirus Disabled Configured
Symantec Antivirus Disabled Not Configured
virustotal_download Preloading Enabled Configured
cuckoo Processing Disabled Configured
cuckoo_modified Processing Disabled Configured
cutthecrap Processing Disabled Not Configured
document_preview Processing Enabled Configured
email_headers Processing Enabled Configured
eml Processing Enabled Configured
exiftool Processing Enabled Configured
extract Processing Enabled Configured
zip Processing Disabled Configured
flare_capa Processing Enabled Configured
triage Processing Enabled Configured
joe Processing Disabled Not Configured
marcher_config Processing Disabled Configured
msg Processing Enabled Configured
office_macros Processing Enabled Configured
office_password Processing Enabled Configured
peepdf Processing Enabled Configured
stringsifter Processing Enabled Configured
url_download Processing Enabled Configured
url_preview Processing Enabled Configured
virustotal_public Processing Enabled Configured
mem_yara Processing Disabled Not Configured
xlm_deobfuscator Processing Enabled Configured
legacyzip Processing Disabled Configured
mattermost Reporting Disabled Not Configured
slack Reporting Disabled Not Configured
Google Safe Browsing (Lookup API) Threat Intelligence Disabled Not Configured
Google Safe Browsing (Update API) Threat Intelligence Disabled Not Configured
SEKOIA.IO Threat Intelligence Disabled Not Configured
URLhaus Threat Intelligence Disabled Not Configured
Yeti Threat Intelligence Disabled Not Configured
kvm Virtualization Disabled Configured
virtualbox Virtualization Disabled Configured

[Augustin-FL]

Hi,

Few things here :

• In order to import a module, you need to use the add module feature on the web interface (or you need to use do_clone() in a python script...see install.py for details). Editing the file structure directly is not supported, and will not work
• This is because some additional operations are being carried out when cloning/updating a repo. More specifically:
  • After the clone/pull, FAME will try to (re)import all python files of the repo, looking for instances of Module classes in them.
  • If it finds any, it will register it in the database. This will cause the module to appear on the web interface, to be considered by the worker, etc...
  • Note that python files that are invalid (syntax error, import error, etc..) are silently ignored. This is the reason why most modules have try/except blocks when importing external dependencies (the search for Module classes is done before installing requirements.txt)
  • The code carrying out the imports/search of Module classes can be found here.

What likely happens is that your python file yara_proc.py has an error, and that FAME is not able to import it (why? I can't tell without seeing your python module).
You could possibly tune the try/except in module_dispatcher to reveal & analyse the error?

This issue make me think of a new feature to be added : to display a warning message on the web interface when FAME is not being able to import a python file.

Also, regarding troubleshoot.py : indeed, thanks for the report. This will be corrected.

[phate1]

Hey, thanks for the swift reply!
that was indeed the problem it relied on a lib that needed installing, I wrongly assumed the requirements would be processed before trying to run it.
I can now see the module in the UI ... it still doesn't work but I can figure that out from here :)