[Bug] Unit tests seem to have gotten slow

[brokensound77]

As of late, tests seem to be really dragging. We should profile the execution of the test suite to find hot spots and see if we can revise anything

[Mikaayenson]

While debugging with @terrancedejesus we noticed the Rule Loader appeared to load multiple instances of the rules which may be slowing things down.

See: manage_versions as an example of how rule loading can affect performance as the rule loader in this example is loaded 1) Package.from_config and 2) default_version_lock.manage_versions.

@dev_group.command('build-release')
@click.argument('config-file', type=click.Path(exists=True, dir_okay=False), required=False, default=PACKAGE_FILE)
@click.option('--update-version-lock', '-u', is_flag=True,
              help='Save version.lock.json file with updated rule versions in the package')
@click.option('--generate-navigator', is_flag=True, help='Generate ATT&CK navigator files')
def build_release(config_file, update_version_lock: bool, generate_navigator: bool, release=None, verbose=True):
    """Assemble all the rules into Kibana-ready release files."""
    config = load_dump(config_file)['package']
    if generate_navigator:
        config['generate_navigator'] = True

    if release is not None:
        config['release'] = release

    if verbose:
        click.echo(f'[+] Building package {config.get("name")}')

    package = Package.from_config(config, verbose=verbose)

    if update_version_lock:
        default_version_lock.manage_versions(package.rules, save_changes=True, verbose=verbose)

    package.save(verbose=verbose)

    if verbose:
        package.get_package_hash(verbose=verbose)
        click.echo(f'- {len(package.rules)} rules included')

    return package

[Mikaayenson]

Running pytest with the duration flag shows the top 10 slowest tests. pytest --durations=10

8.07s setup    tests/test_all_rules.py::TestValidRules::test_all_rule_queries_optimized
0.47s call     tests/test_packages.py::TestPackages::test_rule_versioning
0.45s call     tests/test_packages.py::TestPackages::test_package_summary
0.38s call     tests/test_packages.py::TestPackages::test_package_loader_default_configs
0.33s call     tests/test_all_rules.py::TestValidRules::test_rule_type_changes
0.04s setup    tests/test_mappings.py::TestMappings::test_false_positives
0.04s setup    tests/test_all_rules.py::TestRuleTags::test_casing_and_spacing
0.04s setup    tests/test_all_rules.py::TestRuleTiming::test_eql_interval_to_maxspan
0.04s call     tests/test_all_rules.py::TestRuleFiles::test_rule_file_name_tactic
0.03s setup    tests/test_all_rules.py::TestThreatMappings::test_duplicated_tactics

Rule loader seems to be the bottleneck.