Enhancements to ScanYara Scanner for Improved Rule Organization and Metadata Extraction

[phutelmyer]

Description of Change

This update enhances the ScanYara scanner's functionality, introducing the ability for users to:

• Organize YARA rules into "buckets" for improved organization and alerting.
• Extract YARA metadata for specific buckets, as per user configuration.

In previous versions, all ScanYara matches were stored under strelka.yara.matches, containing only the names of the rules. Metadata was available under strelka.yara.meta but lacked organization, making it difficult to parse.

With this update, users can configure keys inside the backend.yml to sort matches into distinct groups and specify whether metadata should be displayed for those groups, including which metadata fields to extract.

Consider the following YARA rule example:

rule meta_test
// This rule verifies the ScanYara scanner works for meta categories
{
    meta:
        author = "John Doe"
        scope = "detection"

    strings:
        $match_str = "Lorem ipsum dolor sit amet"
    condition:
        $match_str
}

Previously, this rule would output a list of matches by filename, along with a separate list of metadata.

Now, with the proposed update, users can create "buckets" of rules, allowing organization by content type:

{
  "collection": [],
  "detection": [
    {
      "name": "meta_test",
      "ruleset": "default",
      "author": "John Doe"
    }
  ],
  "elapsed": "<ANY>",
  "flags": [],
  "hex": [],
  "information": [],
  "matches": [
    "test",
    "hex_extraction_test",
    "meta_test"
  ],
  "meta": [
    {
      "identifier": "StrelkaHexDump",
      "rule": "hex_extraction_test",
      "value": true
    },
    {
      "identifier": "author",
      "rule": "meta_test",
      "value": "John Doe"
    },
    {
      "identifier": "scope",
      "rule": "meta_test",
      "value": "detection"
    }
  ],
  "rules_loaded": 3,
  "tags": []
}

Users must adapt their backend.yml configuration as follows:

'ScanYara':
  ...
  options:
    location: "/etc/strelka/yara/"
    compiled:
      enabled: True
      filename: "rules.compiled"
    category_key: "scope"
    categories:
      collection:
        show_meta: False
      detection:
        show_meta: True
      information:
        show_meta: False
    meta_fields:
      - "author"
      - "description"
      - "hash"
      - "intel"
    show_all_meta: False
    store_offset: True
    offset_meta_key: "StrelkaHexDump"
    offset_padding: 32

Key Configuration Parameters:

• category_key: The YARA metadata key from which the categories will be derived.
• categories: A predefined list of categories across your YARA rules.
• show_meta: Toggles whether metadata for those categories should be extracted.
• meta_fields: Specifies which metadata fields to extract.
• show_all_meta: When enabled, displays all metadata for rules.
  This change maintains backward compatibility with previous versions of ScanYara.

Describe testing procedures
All tests within test_scan_yara pass successfully, and additional testing confirms that existing functionalities remain unaffected by these updates.

Sample output

{
  "collection": [],
  "detection": [
    {
      "name": "meta_test",
      "ruleset": "default",
      "author": "John Doe"
    }
  ],
  "flags": [],
  "hex": [],
  "information": [],
  "matches": [
    "test",
    "hex_extraction_test",
    "meta_test"
  ],
  "meta": [
    {
      "identifier": "StrelkaHexDump",
      "rule": "hex_extraction_test",
      "value": "True"
    },
    {
      "identifier": "author",
      "rule": "meta_test",
      "value": "John Doe"
    },
    {
      "identifier": "scope",
      "rule": "meta_test",
      "value": "detection"
    }
  ],
  "rules_loaded": 3,
  "tags": []
}

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of and tested my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings