feat: find common strings in `CONTAINS_PATTERNS` across various packages obtained from helper-script

[peb-peb]

Implement a method to compare and return the common strings for CONTAINS_PATTERNS in helper-script for various packages of same product.

It would work something like this:

$ python3 -m cve_bin_tool.helper_script ../dovecot/dovecot-core_2.3.13+dfsg1-1ubuntu1_amd64.deb --product dovecot --version 2.3.13

class DovecotChecker(Checker):
        CONTAINS_PATTERNS = [
                r"# HELP dovecot_build_info Dovecot build information",
                r"# HELP dovecot_stats_uptime_seconds Dovecot stats service uptime",
                r"# TYPE dovecot_stats_uptime_seconds counter",
                r"BUG: Authentication client %u requested invalid authentication mechanism %s (DOVECOT-TOKEN required)",
                r"DOVECOT_SRAND is not available in non-debug builds",
                r"Dovecot is already running with PID %s (read from %s)",
                r"Dovecot is already running? Socket already exists: %s",
                r"Dovecot version mismatch: Master is v%s, %s is v2.3.13 (if you don't care, set version_ignore=yes)",
                r"Must be started by dovecot master process",
                r"Usage: dovecot [-F] [-c <config file>] [-p] [-n] [-a] [--help] [--version]",
                r"dovecot_build_info{version="2.3.13",revision="89f716dc2"} 1 %ld",
                r"systemd listens on port %d, but it's not configured in Dovecot. Closing.",
        ]

$ python3 -m cve_bin_tool.helper_script ../dovecot/dovecot-2.3.14-1.fc34.i686.rpm 

class DovecotChecker(Checker):
        CONTAINS_PATTERNS = [
                r"# HELP dovecot_build Dovecot build information",
                r"BUG: Authentication client %u requested invalid authentication mechanism %s (DOVECOT-TOKEN required)",
                r"DOVECOT_SRAND is not available in non-debug builds",
                r"Dovecot is already running with PID %s (read from %s)",
                r"Dovecot is already running? Socket already exists: %s",
                r"Dovecot version mismatch: Master is v%s, %s is v2.3.14 (if you don't care, set version_ignore=yes)",
                r"Must be started by dovecot master process",
                r"Reporting-UA: %s; Dovecot Mail Delivery Agent",
                r"Usage: dovecot [-F] [-c <config file>] [-p] [-n] [-a] [--help] [--version]",
                r"dovecot_build_info{version="2.3.14",revision="cee3cbc0d"} 1",
                r"libdovecot-sieve.so.0.0.0-2.3.14-1.fc34.i386.debug" <--- not recommended to use this form of strings
                r"libdovecot.so.0.0.0-2.3.14-1.fc34.i386.debug" <--- not recommended to use this form of strings
                r"systemd listens on port %d, but it's not configured in Dovecot. Closing.",
                r"the (deprecated) vnd.dovecot.duplicate extension cannot be used together with the duplicate extension",
                r"vnd.dovecot.environment: cannot assign to environment variable `env.%s'",
                r"vnd.dovecot.environment: invalid variable name within env namespace `env.%d': encountered numeric variable name",
        ]

Currently as you could see, running the helper-script on a single package returns us a very long list for CONTAINS_PATTERNS and it's very tedious to compare these manually. So, what we intend to do is to take these two lists and return only those, which are common in both of these.

$ python3 -m cve_bin_tool.helper_script ../dovecot/dovecot-2.3.14-1.fc34.i686.rpm ../dovecot/dovecot-core_2.3.13+dfsg1-1ubuntu1_amd64.deb

class DovecotChecker(Checker):
        CONTAINS_PATTERNS = [
                r"BUG: Authentication client %u requested invalid authentication mechanism %s (DOVECOT-TOKEN required)",
                r"DOVECOT_SRAND is not available in non-debug builds",
                r"Dovecot is already running with PID %s (read from %s)",
                r"Dovecot is already running? Socket already exists: %s",
                r"Must be started by dovecot master process",
                r"Usage: dovecot [-F] [-c <config file>] [-p] [-n] [-a] [--help] [--version]",
        ]

Some of the challenges related to this could be:

1. what if the user returns the following filenames of a different product
2. currently, we do are not able to obtain the product name & version number from dovecot-core_2.3.13+dfsg1-1ubuntu1_amd64.deb and dovecot-2.3.14.tgz type of packages
3. Currently helper-script only outputs CONTAINS_PATTERNS if any version-strings are found while scanning. So, A way to overcome 2nd challenge would be to output CONTAINS_PATTERNS regardless of whether version-string is found or not and let the user provide --product dovecot, which should override helper-script's parse_filename function and get the common-strings

---

The previous implementation was:

def main(filenames, product_name=None):
    # finds common strings across multiple files for CONTAIN_PATTERNS
    hs = HelperScript(filenames[1], product_name=product_name)
    binary_string_list_1 = hs.extract_and_parse_file(filenames[1])
    binary_string_list_2 = []

    for filename in filenames[2:]:
        hs = HelperScript(filename)
        binary_string_list_2 = hs.extract_and_parse_file(filename)

        if binary_string_list_2:
            binary_string_list_1 = list(
                set(binary_string_list_1).intersection(set(binary_string_list_2))
            )

            hs.contain_patterns = binary_string_list_1
            binary_string_list_2 = []

[peb-peb]

Currently helper-script only outputs CONTAINS_PATTERNS if any version-strings are found while scanning. So, A way to overcome 2nd challenge would be to output CONTAINS_PATTERNS regardless of whether version-string is found or not and let the user provide --product dovecot, which should override helper-script's parse_filename function and get the common-strings

But, the problem with this approach would be that sometimes CONTAINS_PATTERNS becomes too large (this is the reason for --string-length argument) and despite not finding any valid version-strings, CONTAINS_PATTERNS would be printed. This would not be too helpful for the contributor while creating a checker.

(current output if no version-strings are found) $ python3 -m cve_bin_tool.helper_script ../dovecot/dovecot-2.3.14-1.fc34.i686.rpm 
────────────────────────────────────────────────── DovecotChecker ───────────────────────────────────────────────────

# Copyright (C) 2021 Intel Corporation
# SPDX-License-Identifier: GPL-3.0-or-later


"""
CVE checker for dovecot:

<provide reference links here>
"""
from cve_bin_tool.checkers import Checker


class DovecotChecker(Checker):
        CONTAINS_PATTERNS = [
        ]
        FILENAME_PATTERNS = [
        ]
        VERSION_PATTERNS = [
        ]
        VENDOR_PRODUCT = [ ]
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

(after implementing this if no version-strings are found) $ python3 -m cve_bin_tool.helper_script ../dovecot/dovecot-2.3.14-1.fc34.i686.rpm 
────────────────────────────────────────────────── DovecotChecker ───────────────────────────────────────────────────

# Copyright (C) 2021 Intel Corporation
# SPDX-License-Identifier: GPL-3.0-or-later


"""
CVE checker for dovecot:

<provide reference links here>
"""
from cve_bin_tool.checkers import Checker


class DovecotChecker(Checker):
        CONTAINS_PATTERNS = [
                r"# HELP dovecot_build Dovecot build information",
                r"BUG: Authentication client %u requested invalid authentication mechanism %s (DOVECOT-TOKEN required)",
                r"DOVECOT_SRAND is not available in non-debug builds",
                r"Dovecot is already running with PID %s (read from %s)",
                r"Dovecot is already running? Socket already exists: %s",
                r"Dovecot version mismatch: Master is v%s, %s is v2.3.14 (if you don't care, set version_ignore=yes)",
                r"Must be started by dovecot master process",
                r"Reporting-UA: %s; Dovecot Mail Delivery Agent",
                r"Usage: dovecot [-F] [-c <config file>] [-p] [-n] [-a] [--help] [--version]",
                r"dovecot_build_info{version="2.3.14",revision="cee3cbc0d"} 1",
                r"libdovecot-sieve.so.0.0.0-2.3.14-1.fc34.i386.debug" <--- not recommended to use this form of strings
                r"libdovecot.so.0.0.0-2.3.14-1.fc34.i386.debug" <--- not recommended to use this form of strings
                r"systemd listens on port %d, but it's not configured in Dovecot. Closing.",
                r"the (deprecated) vnd.dovecot.duplicate extension cannot be used together with the duplicate extension",
                r"vnd.dovecot.environment: cannot assign to environment variable `env.%s'",
                r"vnd.dovecot.environment: invalid variable name within env namespace `env.%d': encountered numeric variable name",
        ]
        FILENAME_PATTERNS = [
        ]
        VERSION_PATTERNS = [
        ]
        VENDOR_PRODUCT = [ ]
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

P.S. we do get CONTAINS_PATTERNS, despite not finding a version-string and this could be seen by changing the log level to debug by -l debug