Skip to content

@mlaferrera mlaferrera released this Jul 27, 2020

Added

  • Add getjson() function to allow plugins to use valid json strings are configuration
    options (@ytreister, #144)

Changed

  • Fix stoq command line to properly parsed --plugin-opts and --request-source
    arguments that contain = or : characters
  • Ensure always_dispatch in stoq.cfg leverages getlist() when Stoq() is
    is instantiated. (#149)
  • Multiple fixes and updates to Dockerfile
Assets 2

@mlaferrera mlaferrera released this Mar 18, 2020 · 32 commits to master since this release

Added

  • Support for asyncio within the framework and plugins
  • Better type checking support
  • self.log may be leveraged from within all plugin classes
  • Add new Error() class for standardizing errors from stoQ and plugins
    Error() will track plugin name, error message, and payload_id (optional)
  • Add configuration properties from [Core] and [Documentation] to each plugin object when loaded
  • PayloadMeta now has a should_scan boolean.
    Allows payloads to be logged and archived, but not scanned by worker plugin.
  • Payload is now updated as results are completed.
    Results from completed scans will be available to other plugins instantly
  • Request() class is passed to all dispatchers, workers, and archiver plugins.
    The Request object contains all payloads, request metadata, results, and errors from
    all other completed plugins. This will allow for all neccessary plugins to have a full
    understanding of the current state of the complete Request.
  • WorkerPlugins now have a configuration option of required_workers.
    This allows for chained worker dependencies. If required_workers is defined, the
    parent plugin will not be run until all required plugins are completed successfully. The
    parent plugin may then use results from other completed plugins for their respective
    scanning tasks.
  • Duplicate extracted payloads are no longer simply skipped, they are appended to
    Payload.results[].extracted_by and Payload.results[].extracted_from
  • Add StoqConfigParser to stoq.helpers to extend options for Stoq and plugin configurations.
  • Parallelization is performed across all of the plugins that can run in a given round,
    instead of parallelizing across all of the plugins to perform on a given payload (#147)
  • Ensure plugin_name is set to the name of the plugin class in case Name is not defined in
    the plugin's configuration.

Changed

  • PayloadResults is now an object of Payload.results, rather than an independent object
  • Most objects have been removed from Payload and are now availabe in Payload.results,
    namely extracted_by, extracted_from, payload_id, size, payload_meta
  • Payload.plugins_run moved to PayloadResults.plugins_runs and is now a Dict[str, List[str]]
    rather than Dict[str, List[List[str]]]
  • PayloadResults.workers is now a Dict[str, Dict] rather than List[Dict[str, Dict]]
  • PayloadMeta is now an object of PayloadResults.payload_meta
  • PayloadResults.extracted_by is now a List[str] rather than str
  • PayloadResults.extracted_from is now a List[str] rather than str
  • Dispatchers run on each payload every round, instead of once per payload. This allows
    the dispatcher to take advantage of the request state model. (#147)
  • Worker plugins can specify additional plugins to run on the payload they scan, effectively giving them dispatch capability.
    With YARA, for example, this allows us to directly scan with YARA and dispatch
    to other plugins by running YARA once. Otherwise, we would run YARA as a dispatcher,
    and then immediately run YARA again as a worker plugin. (#147)
  • Archivers run at the very end along with connectors and decorators because we no
    longer scan a payload to completion at once. (#147)
  • The default value for max_recursion has increased because the average number of
    worker rounds taken to complete a scan is expected to increase. (#147)

Deprecated

  • DeepDispatcher plugin class has been removed
  • Payload.plugins_run has been removed in favor of PayloadResults.plugins_run)
  • Payload.worker_results has been removed in favor of PayloadResults.workers
  • RequestMeta is no longer passed to plugins, in favor of the Request object
  • plugins_opts has been removed from plugin __init__ function. All plugin configuration options
    are only available in self.config
Assets 2
Pre-release
Pre-release

@mlaferrera mlaferrera released this Feb 5, 2020 · 34 commits to master since this release

Added

  • Ensure plugin_name is set to the name of the plugin class in case Name is not defined in the plugin's configuration.

Changed

  • Updated Dockerfile:
    • exiftool updated to 11.86
    • triddefs installed to trid plugin path
    • stoq-framework version to 3.0.0b3
  • Set provider_consumers to 2 to avoid issues with provider timeouts
  • Documentation updates
Assets 2
Pre-release
Pre-release

@mlaferrera mlaferrera released this Dec 4, 2019 · 39 commits to master since this release

Added

  • Support for asyncio within the framework and plugins
  • Better type checking support
  • self.log may be leveraged from within all plugin classes
  • Add new Error() class for standardizing errors from stoQ and plugins
    Error() will track plugin name, error message, and payload_id (optional)
  • Add configuration properties from [Core] and [Documentation] to each plugin object when loaded
  • PayloadMeta now has a should_scan boolean.
    Allows payloads to be logged and archived, but not scanned by worker plugin.
  • Payload is now updated as results are completed.
    Results from completed scans will be available to other plugins instantly
  • Request() class is passed to all dispatchers, workers, and archiver plugins.
    The Request object contains all payloads, request metadata, results, and errors from
    all other completed plugins. This will allow for all neccessary plugins to have a full
    understanding of the current state of the complete Request.
  • WorkerPlugins now have a configuration option of required_workers.
    This allows for chained worker dependencies. If required_workers is defined, the
    parent plugin will not be run until all required plugins are completed successfully. The
    parent plugin may then use results from other completed plugins for their respective
    scanning tasks.
  • Duplicate extracted payloads are no longer simply skipped, they are appended to
    Payload.results[].extracted_by and Payload.results[].extracted_from
  • Add StoqConfigParser to stoq.helpers to extend options for Stoq and plugin configurations.
  • Parallelization is performed across all of the plugins that can run in a given round,
    instead of parallelizing across all of the plugins to perform on a given payload (#147)

Changed

  • PayloadResults is now an object of Payload.results, rather than an independent object
  • Most objects have been removed from Payload and are now availabe in Payload.results,
    namely extracted_by, extracted_from, payload_id, size, payload_meta
  • Payload.plugins_run moved to PayloadResults.plugins_runs and is now a Dict[str, List[str]]
    rather than Dict[str, List[List[str]]]
  • PayloadResults.workers is now a Dict[str, Dict] rather than List[Dict[str, Dict]]
  • PayloadMeta is now an object of PayloadResults.payload_meta
  • PayloadResults.extracted_by is now a List[str] rather than str
  • PayloadResults.extracted_from is now a List[str] rather than str
  • Dispatchers run on each payload every round, instead of once per payload. This allows
    the dispatcher to take advantage of the request state model. (#147)
  • Worker plugins can specify additional plugins to run on the payload they scan, effectively giving them dispatch capability.
    With YARA, for example, this allows us to directly scan with YARA and dispatch
    to other plugins by running YARA once. Otherwise, we would run YARA as a dispatcher,
    and then immediately run YARA again as a worker plugin. (#147)
  • Archivers run at the very end along with connectors and decorators because we no
    longer scan a payload to completion at once. (#147)
  • The default value for max_recursion has increased because the average number of
    worker rounds taken to complete a scan is expected to increase. (#147)

Deprecated

  • DeepDispatcher plugin class has been removed
  • Payload.plugins_run has been removed in favor of PayloadResults.plugins_run)
  • Payload.worker_results has been removed in favor of PayloadResults.workers
  • RequestMeta is no longer passed to plugins, in favor of the Request object
  • plugins_opts has been removed from plugin __init__ function. All plugin configuration options
    are only available in self.config
Assets 2

@mlaferrera mlaferrera released this Dec 3, 2019 · 89 commits to master since this release

Changed

  • Fix plugin requirements path when installing from Github
Assets 2

@mlaferrera mlaferrera released this Nov 8, 2019 · 91 commits to master since this release

Changed

  • Fix issue where deep dispatchers defined from the CLI were not passed to Stoq()
  • Update URL for plugins to include v2 branch when using --github
Assets 2

@mlaferrera mlaferrera released this Jun 7, 2019 · 94 commits to master since this release

Added

  • Provide console output if stoQ configuration file does not exist (Thanks for feedback @jakubgs!)
  • Add command line option --config-file to define stoQ configuration file
  • Add command line option --log-level to allow for setting of the log level
  • Documentation for simplified method of defining plugin options within __init__

Changed

  • Raise StoqPluginException if installing a plugin that is already installed
  • Display plugin_path when plugin is successfully installed
  • Raise StoqPluginNotFound when attempting to load non-existent or invalid plugin
Assets 2

@mlaferrera mlaferrera released this Mar 29, 2019 · 102 commits to master since this release

Added

  • Stoq.reconstruct_all_subresponses() method to allow for reconstructing StoqResponse objects iteratively (@maydewd)

Changed

  • Force payload content to be of type bytes
Assets 2

@mlaferrera mlaferrera released this Feb 15, 2019 · 111 commits to master since this release

Added

  • Allow --plugin-dir from command line to force one or more plugin directories
  • Provide better logging when a plugin is installed from github as a non-root user outside of a venv
  • Gracefully handle exceptions in ConnectorPlugins

Changed

  • Improve handling of plugin configuration options. Plugin options can now also be in stoq.cfg. (Thanks for feedback @chemberger!)
  • Set default precendence for plugin configuration options to be 1) plugin_opts when instantiating Stoq, 2) stoq.cfg, 3) Plugin config file (Thanks for feedback @chemberger!)
  • Make formatted exceptions more legible in results
Assets 2

@mlaferrera mlaferrera released this Jan 15, 2019 · 139 commits to master since this release

Changed

  • Fix erroneous error from being displayed when installing or listing plugins
  • Fix plugin install if requirements.txt does not exist
  • Documentation update for installation (@chemberger)
Assets 2
You can’t perform that action at this time.