deprecate the word blocklist in sigma context and use status instead (#…

…2198)

* deprecate the word blocklist in sigma context and use status instead

* fix sigma verifier

* maybe i should also rename the file itself.

data/timesketch.conf

@@ -251,7 +251,7 @@ EXTERNAL_HOST_URL = 'https://localhost'
 SIGMA_RULES_FOLDERS = ['/etc/timesketch/sigma/rules/']
 SIGMA_CONFIG = '/etc/timesketch/sigma_config.yaml'
 SIGMA_TAG_DELAY = 5
-SIGMA_BLOCKLIST_CSV = '/etc/timesketch/sigma_blocklist.csv'
+SIGMA_RULE_STATUS_CSV = '/etc/timesketch/sigma_rule_status.csv'
 
 #-------------------------------------------------------------------------------
 # Flask Settings

docker/dev/build/docker-entrypoint.sh

@@ -18,7 +18,7 @@ if [ "$1" = 'timesketch' ]; then
   cp /usr/local/src/timesketch/data/data_finder.yaml /etc/timesketch/
   cp /usr/local/src/timesketch/data/bigquery_matcher.yaml /etc/timesketch/
   ln -s /usr/local/src/timesketch/data/sigma_config.yaml /etc/timesketch/sigma_config.yaml
-  ln -s /usr/local/src/timesketch/data/sigma_blocklist.csv /etc/timesketch/sigma_blocklist.csv
+  ln -s /usr/local/src/timesketch/data/sigma_rule_status.csv /etc/timesketch/sigma_rule_status.csv
   ln -s /usr/local/src/timesketch/data/sigma /etc/timesketch/
 
 

docker/e2e/Dockerfile

@@ -8,19 +8,19 @@ ARG PPA_TRACK=stable
 RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
-    software-properties-common \
-    python3-pip \
-    python3-wheel \
-    python3-setuptools \
-    python3-psycopg2 \
-    git \
-    wget \
+  software-properties-common \
+  python3-pip \
+  python3-wheel \
+  python3-setuptools \
+  python3-psycopg2 \
+  git \
+  wget \
   && rm -rf /var/lib/apt/lists/*
 
 # Install Plaso
 RUN add-apt-repository -y ppa:gift/$PPA_TRACK
 RUN apt-get update && apt-get install -y --no-install-recommends \
-    plaso-tools \
+  plaso-tools \
   && rm -rf /var/lib/apt/lists/*
 
 # Install Timesketch from master to get the latest code
@@ -31,7 +31,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 ADD . /tmp/timesketch
 RUN pip3 install -r /tmp/timesketch/requirements.txt
 RUN pip3 install /tmp/timesketch && pip3 install /tmp/timesketch/api_client/python && \
-    pip3 install /tmp/timesketch/importer_client/python
+  pip3 install /tmp/timesketch/importer_client/python
 
 # Copy Timesketch config files into /etc/timesketch
 RUN mkdir /etc/timesketch
@@ -43,7 +43,7 @@ RUN cp /tmp/timesketch/data/features.yaml /etc/timesketch/
 RUN cp /tmp/timesketch/data/plaso.mappings /etc/timesketch/
 RUN cp /tmp/timesketch/data/generic.mappings /etc/timesketch/
 RUN cp /tmp/timesketch/data/sigma_config.yaml /etc/timesketch/
-RUN cp /tmp/timesketch/data/sigma_blocklist.csv /etc/timesketch/
+RUN cp /tmp/timesketch/data/sigma_rule_status.csv /etc/timesketch/
 RUN cp /tmp/timesketch/data/data_finder.yaml /etc/timesketch/
 RUN cp /tmp/timesketch/data/bigquery_matcher.yaml /etc/timesketch/
 RUN cp -r /tmp/timesketch/data/sigma /etc/timesketch/sigma

docs/guides/analyzers/sigma_analyzer.md

@@ -8,15 +8,15 @@ To make a rule be used in the Analyzer:
 
 - place it in a folder that is mentioned in the `SIGMA_RULES_FOLDERS=[]` section of `data/timesketch.conf`
 - go to the Sigma tab of a sketch and check the rule is parsed correctly and the corresponding `es_query` is accurate
-- add an entry in `sigma_blocklist.csv` with column `status` value: `good`
+- add an entry in `sigma_rule_status.csv` with column `status` value: `good`
 - go again to the Sigma tab and open the details of that rule and check the value of `'ts_use_in_analyzer': True`
 
 Not every rule installed on a Timesketch server will be used by the Analyzer.
 Reasons might be because:
 
 - the rule caused parsing error
 - the rule uses concepts that are not implemented from the Sigma project for the Timesketch / OpenSearch backend (e.g. Aggregations)
-- the rule is marked 'bad' in `sigma_blocklist.csv` file
+- the rule is marked 'bad' in `sigma_rule_status.csv` file
 - the rule is located in a folder containing the term `deprecated`
 - After parsing a rule the following value is set: `'ts_use_in_analyzer': False`
 

docs/guides/user/sigma.md

@@ -76,19 +76,19 @@ The rules then will be under
 timesketch/data/sigma
 ```
 
-### Sigma Rules Blocklist file
+### Sigma Rules sigma_rule_status file
 
-The `data/sigma_blocklist.csv` is where Timesketch maintains a list of rules. 
+The `data/sigma_rule_status.csv` is where Timesketch maintains a list of rules. 
 Each rule can have one of the following status values: `good,bad,exploratory`.
 * `exploratoy` rules will be shown in the UI but ignored in the Analyzer. So this status can be used to test rules. By default each rule is considered `exploratory`. 
 * `good` rules will be used in the Sigma analyzer. 
 * `bad` will be ignored and not shown in the UI or used in the Sigma analyzer.
 
 It is good practice to add new rules in this file if they are tested and verified to not be compatible.
 
-Each method that reads Sigma rules from the a folder is checking if part of the full path of a rule is mentioned in the `data/sigma_blocklist.csv` file.
+Each method that reads Sigma rules from the a folder is checking if part of the full path of a rule is mentioned in the `data/sigma_rule_status.csv` file.
 
-For example a file at `/etc/timesketch/data/sigma/rules-unsupported/foo/bar.yml` would not be parsed as a line in `data/sigma_blocklist.csv` mentions:
+For example a file at `/etc/timesketch/data/sigma/rules-unsupported/foo/bar.yml` would not be parsed as a line in `data/sigma_rule_status.csv` mentions:
 
 ```
 /rules-unsupported/,bad,Sigma internal folder name,2021-11-19,
@@ -124,7 +124,7 @@ There are multiple sigma related config variables in `timesketch.conf`.
 SIGMA_RULES_FOLDERS = ['/etc/timesketch/sigma/rules/']
 SIGMA_CONFIG = '/etc/timesketch/sigma_config.yaml'
 SIGMA_TAG_DELAY = 5
-SIGMA_BLOCKLIST_CSV = '/etc/timesketch/sigma_blocklist.csv'
+SIGMA_BLOCKLIST_CSV = '/etc/timesketch/sigma_rule_status.csv'
 ```
 
 The `SIGMA_RULES_FOLDERS` points to the folder(s) where Sigma rules are stored. The folder is the local folder of the Timesketch server (celery worker and webserver). For a distributed system, mounting network shares is possible.

test_tools/sigma_verify_rules.py

@@ -35,49 +35,51 @@
 logging.basicConfig(level=os.environ.get("LOGLEVEL", "INFO"))
 
 
-def get_sigma_blocklist(blocklist_path="./data/sigma_blocklist.csv"):
-    """Get a dataframe of sigma rules to ignore.
+def get_sigma_rule_status(rule_status_path="./data/sigma_rule_status.csv"):
+    """Get a dataframe of sigma rules status.
 
     This includes filenames, paths, ids.
 
     Args:
-        blocklist_path(str): Path to a blocklist file.
-            The default value is './data/sigma_blocklist.csv'
+        rule_status_path(str): Path to a status file.
+            The default value is './data/sigma_rule_status.csv'
 
     Returns:
-        Pandas dataframe with blocklist
+        Pandas dataframe with rule status
 
     Raises:
-        ValueError: Sigma blocklist file is not readabale.
+        ValueError: Sigma rule status file is not readabale.
     """
 
-    if blocklist_path is None or blocklist_path == "":
-        blocklist_path = "./data/sigma_blocklist.csv"
+    if rule_status_path is None or rule_status_path == "":
+        rule_status_path = "./data/sigma_rule_status.csv"
 
-    if not blocklist_path:
-        raise ValueError("No blocklist_file_path set via param or config file")
+    if not rule_status_path:
+        raise ValueError("No rule_status_path set via param or config file")
 
-    if not os.path.isfile(blocklist_path):
+    if not os.path.isfile(rule_status_path):
         raise ValueError(
-            "Unable to open file: [{0:s}], it does not exist.".format(blocklist_path)
+            "Unable to open file: [{0:s}], it does not exist.".format(
+                rule_status_path
+            )
         )
 
-    if not os.access(blocklist_path, os.R_OK):
+    if not os.access(rule_status_path, os.R_OK):
         raise ValueError(
             "Unable to open file: [{0:s}], cannot open it for "
-            "read, please check permissions.".format(blocklist_path)
+            "read, please check permissions.".format(rule_status_path)
         )
 
-    return pd.read_csv(blocklist_path)
+    return pd.read_csv(rule_status_path)
 
 
-def run_verifier(rules_path, config_file_path, blocklist_path=None):
+def run_verifier(rules_path, config_file_path, rule_status_path=None):
     """Run an sigma parsing test on a dir and returns results from the run.
 
     Args:
         rules_path (str): Path to the Sigma rules.
         config_file_path (str): Path to a config file with Sigma mapping data.
-        blocklist_path (str): Optional path to a blocklist file.
+        rule_status_path (str): Optional path to a status file.
             The default value is none.
 
     Raises:
@@ -96,15 +98,19 @@ def run_verifier(rules_path, config_file_path, blocklist_path=None):
         raise IOError("Rules not found at path: {0:s}".format(rules_path))
     if not os.path.isfile(config_file_path):
         raise IOError(
-            "Config file path not found at path: {0:s}".format(config_file_path)
+            "Config file path not found at path: {0:s}".format(
+                config_file_path
+            )
         )
 
-    sigma_config = sigma_util.get_sigma_config_file(config_file=config_file_path)
+    sigma_config = sigma_util.get_sigma_config_file(
+        config_file=config_file_path
+    )
 
     return_verified_rules = []
     return_rules_with_problems = []
 
-    ignore = get_sigma_blocklist(blocklist_path)
+    ignore = get_sigma_rule_status(rule_status_path)
     ignore_list = list(ignore["path"].unique())
 
     for dirpath, dirnames, files in os.walk(rules_path):
@@ -155,7 +161,9 @@ def move_problematic_rule(filepath, move_to_path, reason=None):
         move_to_path: path to move the problematic rules to
         reason: optional reason why file is moved
     """
-    logging.info("Moving the rule: {0:s} to {1:s}".format(filepath, move_to_path))
+    logging.info(
+        "Moving the rule: {0:s} to {1:s}".format(filepath, move_to_path)
+    )
     try:
         os.makedirs(move_to_path, exist_ok=True)
         debug_path = os.path.join(move_to_path, "debug.log")
@@ -183,7 +191,9 @@ def move_problematic_rule(filepath, move_to_path, reason=None):
     )
     epilog = "Remember to feed the tool with proper rule data."
 
-    arguments = argparse.ArgumentParser(description=description, allow_abbrev=True)
+    arguments = argparse.ArgumentParser(
+        description=description, allow_abbrev=True
+    )
     arguments.add_argument(
         "--config_file",
         "--file",
@@ -195,13 +205,13 @@ def move_problematic_rule(filepath, move_to_path, reason=None):
         help=("Path to the file containing the config data to feed sigma "),
     )
     arguments.add_argument(
-        "--blocklist_file",
-        dest="blocklist_file_path",
+        "--rule_status_file",
+        dest="rule_status_path",
         action="store",
         default="",
         type=str,
-        metavar="PATH_TO_BLOCK_FILE",
-        help=("Path to the file containing the blocklist "),
+        metavar="PATH_TO_STATUS_FILE",
+        help=("Path to the file containing the rule status"),
     )
     arguments.add_argument(
         "rules_path",
@@ -211,8 +221,12 @@ def move_problematic_rule(filepath, move_to_path, reason=None):
         metavar="PATH_TO_RULES",
         help="Path to the rules to test.",
     )
-    arguments.add_argument("--debug", action="store_true", help="print debug messages ")
-    arguments.add_argument("--info", action="store_true", help="print info messages ")
+    arguments.add_argument(
+        "--debug", action="store_true", help="print debug messages "
+    )
+    arguments.add_argument(
+        "--info", action="store_true", help="print info messages "
+    )
     arguments.add_argument(
         "--move",
         dest="move_to_path",
@@ -238,18 +252,22 @@ def move_problematic_rule(filepath, move_to_path, reason=None):
         sys.exit(1)
 
     if not os.path.isdir(options.rules_path):
-        print("The path to the rules does not exist ({0:s})".format(options.rules_path))
+        print(
+            "The path to the rules does not exist ({0:s})".format(
+                options.rules_path
+            )
+        )
         sys.exit(1)
 
-    if len(options.blocklist_file_path) > 0:
-        if not os.path.isfile(options.blocklist_file_path):
-            print("Blocklist file not found.")
+    if len(options.rule_status_path) > 0:
+        if not os.path.isfile(options.rule_status_path):
+            print("rule status file not found.")
             sys.exit(1)
 
     sigma_verified_rules, sigma_rules_with_problems = run_verifier(
         rules_path=options.rules_path,
         config_file_path=options.config_file_path,
-        blocklist_path=options.blocklist_file_path,
+        rule_status_path=options.rule_status_path,
     )
 
     if len(sigma_rules_with_problems) > 0: