analysis plugin development
FACT Analysis plugins can work on either the binaries of an object or on results of other plugins, as well as on both. In general a plugin is executed on each file which is (recursively) extracted from the uploaded firmware container and the container itself. If a plugin should be run only on the outer container, this can be specified in the plugin code. This is explained in 1. This allows to run the same set of analyses on each firmware layer. Results of each layer can be propagated by adding a summary to the plugin, thus making partial results visible at the firmware level. The ability to use results of other plugins, additionally allows to create incremental analysis workflows.
FACT detects plugins automatically as long as they are stored in src/plugins/analysis/. A plugin consists of folders and files following the following template:
.
├── __init__.py
├── install.py [OPTIONAL]
├── apt-pkgs-runtime.txt [OPTIONAL]
├── apt-pkgs-build.txt [OPTIONAL]
├── dnf-pkgs-runtime.txt [OPTIONAL]
├── dnf-pkgs-build.txt [OPTIONAL]
├── code
│ ├── __init__.py
│ └── PLUGIN_NAME.py
├── internal [OPTIONAL]
│ └── ADDITIONAL_SOURCES_OR_CODE
├── signatures [YARA PLUGINS ONLY]
├── test [OPTIONAL]
│ ├── __init__.py
│ ├── test_PLUGIN_NAME.py
│ └── data
│ └── SOME DATA FILES TO TEST
└── view
└── PLUGIN_NAME.html [OPTIONAL]
Only the files __init__.py and code/PLUGIN_NAME.py are mandatory for the plugin to work.
Let's have a quick look on the other files:
install.py when provided is automatically triggered by FACT's installation script.
Put your dependencies in the {apt,dnf}-pkgs-{build,runtime}.txt files for debian/fedora packages needed for building/installing your plugin.
A basic install.py should look like this.
#!/usr/bin/env python3
import logging
from pathlib import Path
try:
from plugins.installer import AbstractPluginInstaller
except ImportError:
import sys
SRC_PATH = Path(__file__).absolute().parent.parent.parent.parent
sys.path.append(str(SRC_PATH))
from plugins.installer import AbstractPluginInstaller
class YourInstaller(AbstractPluginInstaller):
base_path = Path(__file__).resolve().parent
# Insert your code here by overwriting functions of AbstractPluginInstaller
# Alias for generic use
Installer = YourInstaller
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)
Installer().install()This optional file can contain a customized view utilizing Jinja2 template features. If there is no custom template, a simple default template is used to show the results of an analysis. This standard template just generates a table from the key-value pairs in the result dictionary including all entries. :warning: Due to the internal FACT processing, this directory should not contain more than one file. That means you cannot add dependencies, like .js or .css files to your view.
ℹ️ To get started, you can copy our minimal functional hello world plugin to start your own development.
Plugins may implement completely genuine analysis techniques, use available python libraries or merely wrap existing third party code.
The analysis happens in the method process_object() inside the AnalysisPlugin class.
As input this function is presented with a FileObject.
During analysis a plugin result should be added to that object.
Finally, the object is returned again.
The relevant parts of the FileObject class (processed_analysis and binary) are described in the comments inside the code template.
Let's have a look on the plugin template:
This is the actual plugin code. The following should mostly be self-explanatory and can be used to write your own plugin.
from analysis.PluginBase import AnalysisBasePlugin
from objects.file import FileObject
class AnalysisPlugin(AnalysisBasePlugin):
'''
Some Description
'''
# mandatory plugin attributes:
NAME = 'plugin_name' # name of the plugin (using snake case)
DESCRIPTION = '...' # a short description of the plugin
VERSION = 'x.y.z' # the version of this plugin (should be updated each time the plugin is changed)
FILE = __file__ # used internally
# optional plugin attributes:
SYSTEM_VERSION = None # version of internally used tool/docker container/etc. (default: `None`)
MIME_BLACKLIST = ['mime_1', ...] # list of MIME types that should be ignored by the plugin (default: `[]`)
DEPENDENCIES = ['plugin_1', ...] # list of plugin names that this plugin relies on (default: `[]`)
RECURSIVE = True # analyze all recursively extracted files (`True`, default) or only the outer firmware container (`False`)
TIMEOUT = 300 # time in seconds after which the analysis is aborted (default: 300)
def process_object(self, file_object: FileObject) -> FileObject:
'''
This function must be implemented by the plugin.
Analysis result must be a dict stored in `file_object.processed_analysis[self.NAME]`
CAUTION: Dict keys must be strings! The contents must be JSON compatible (i.e. no byte strings, etc.)!
CAUTION: The contents must be JSON compatible (i.e. no byte strings, etc.)!
If you want to propagate results to parent objects store a list of strings in
`file_object.processed_analysis[self.NAME]['summary']`.
The file's (binary) content is available via `file_object.binary` (type: bytes).
The file's local file system path is available via `file_object.file_path`.
Results of other plugins can be accessed via `file_object.processed_analysis['PLUGIN_NAME']`.
Do not forget to add these plugins to `DEPENDENCIES`.
'''
return file_objectThe blacklisting feature of FACT is used to increase the system performance, by skipping irrelevant file types. For example compressed file types like ZIP archives will not produce results for most analysis plugins.
Instead of the optional MIME_BLACKLIST, you can also use MIME_WHITELIST to specify a list of allowed MIME types.
This allows plugins to target specific files.
E.g. the exploit mitigations plugin will only work on ELF files so those types can be whitelisted.
If your plugin needs configuration options, you can add them to src/main.cfg in a section named as your plugin. You can access a field (e.g. "option_a") of your configuration with the following Code:
self.config.get(self.NAME, "option_a")The black/whitelists can be set in the configuration file as well by adding mime_blacklist = mime1, mime2, ... (or mime_whitelist) to the section of the plugin.
An important configuration option is the threads value. Looking at the default config you see that most plugins have a value of 2 or higher. The value directly corresponds to the number of concurrent plugin instances FACT will run. Memory-heavy plugins should set this value conservatively while other, less resource-hungry plugins might set a value of 4, 8 or higher (depending on your system). :warning: Default for threads is 1.
The optional folder internal can provide additional code or other stuff needed for the plugin. You can use the following code to get the absolute path of the internal directory:
from pathlib import Path
import sys
INTERNAL_DIRECTORY = sys.path.append(str(Path(__file__).parent.parent / 'internal'))
sys.path.append(INTERNAL_DIRECTORY)The last line is used to add the internal directory to your python path. It is only needed if code is stored there. Once you have added the path like that, you can just import from modules inside the internal directory as from normal libraries.
This file contains your test code. In general, FACT has quite a high code coverage, so if you intend to contribute to the core, tests will be necessary. For internal development tests might not be needed, though they might be helpful in debugging sources of failure.
Since analysis plugins use a complex multithreading approach, instantiating a plugin in tests is not easy.
You might want to use this template to simplify your testing.
The AnalysisPluginTest class used by the template handles all the multithreading stuff and checks some general things (e.g. plugin registration):
from test.unit.analysis.analysis_plugin_test_class import AnalysisPluginTest
from objects.file import FileObject
from ..code.YOUR_PLUGIN import AnalysisPlugin
class TestAnalysisPluginYOURPLUGINNAME(AnalysisPluginTest):
PLUGIN_NAME = 'YOUR_PLUGIN_NAME'
PLUGIN_CLASS = AnalysisPlugin
def setUp(self): # optional
super().setUp()
# additional setup can go here
def _set_config(self): # optional
pass
# config changes (before the plugin is instantiated) can go here
# e.g. `config.set(self.PLUGIN_NAME, 'key', 'value')`
def tearDown(self): # optional
super().tearDown()
# additional cleanup can go here
def test_your_test_code(self):
# your test code. An example could look like this:
test_object = FileObject(file_path='path to some test file')
result = self.analysis_plugin.process_object(test_object)
assert result.processed_analysis[self.PLUGIN_NAME]['some_result'] == 'expected result'
def test_internal_stuff():
pass
# internal tools / static methods / etc. should be tested outside the `AnalysisPluginTest` class for less overheadUsing this template you can just call functions inside your plugin class from the instance given at self.analysis_plugin.
Note that by running the default tests (simply calling pytest from your FACT_core directory) all plugin tests will run as well.
This folder shall contain any additional files, that are needed for your tests. You can address this folder using the following code in your test code file.
from pathlib import Path
TEST_DATA_DIR = Path(__file__).parent / 'data'A lot of analysis is based on simple pattern matching.
Therefore, FACT provides a special YaraBasePlugin utilizing the Yara pattern matching system.
Yara-based plugins are quite easy to create once you have signature files (Yara "rules"):
- Store your Yara rule files to the signature directory (as seen in 0)
- Customize the following plugin template instead of the original template shown in 1
from analysis.YaraPluginBase import YaraBasePlugin
class AnalysisPlugin(YaraBasePlugin):
'''
A Short description
'''
# mandatory plugin attributes:
NAME = 'plugin_name' # name of the plugin (using snake case)
DESCRIPTION = '...' # a short description of the plugin
VERSION = 'x.y.z' # the version of this plugin (should be updated each time the plugin is changed)
FILE = __file__ # used internally
# optional plugin attributes:
SYSTEM_VERSION = None # version of internally used tool/docker container/etc. (default: `None`)
MIME_BLACKLIST = ['mime_1', ...] # list of MIME types that should be ignored by the plugin (default: `[]`)
DEPENDENCIES = ['plugin_1', ...] # list of plugin names that this plugin relies on (default: `[]`)
RECURSIVE = True # analyze all recursively extracted files (`True`, default) or only the outer firmware container (`False`)
TIMEOUT = 300 # time in seconds after which the analysis is aborted (default: 300)
def process_object(self, file_object): # optional
file_object = super().process_object(file_object)
# optional post-processing may happen here
# by default the pattern matching results are already stored in `file_object`
return file_object- Done!
Additional optional steps include configuration of threads, designing an own view and writing tests as for the general plugin development.
FACT has a simple tagging mechanism, that allows to set tags based on your analysis result. By default, tags are set for single files. Optionally, they can be propagated, so that they also appear on the outer firmware container. This allows to display critical information in a more visible way.
To set a tag in your plugin you can use two built-in/helper-functions as seen in this snipped:
from helperFunctions.tag import TagColor
# [...]
self.add_analysis_tag(
file_object=file_object,
tag_name='internal_tag_identifier',
value='Content shown on tag in GUI',
color=TagColor.ORANGE,
propagate=True
)
# [...]The analysis base plugin implements the add_analysis_tag method, so you can use it with self.
The flag propagate controls if the tag should be shown in the outer container as well.
For very significant information this might be useful.
TagColor is an enum style class that contains the values GRAY, BLUE, GREEN, LIGHT_BLUE, ORANGE and RED.
