Make it possible to disallow slow queries (or warn user about the problem)

[msm-code]

Your checklist for this pull request

• I've read the contributing guideline.
• I've tested my changes by building and running mquery, and testing changed functionality (if applicable)
• I've added automated tests for my change (if applicable, optional)
• I've updated documentation to reflect my change (if applicable) - not yet, I'll add a separate issue for documentation update

What is the current behaviour?

#308

Currently, people who don't understand how mquery and mwdb work behind the scene can execute very bad yara queries and it'll be (a) expensive (b) inefficient (c) take forever to run.

What is the new behaviour?

If configuration option query_disallow_degenerate is set to "true", users won't be able to run any queries that would end up doing a whole backend scan. Of course they can still run very inefficient queries like "aaa" | "bbb" | ccc" | "ddd" or just {00 00 00}, so this won't help in every case, but that's all we can do at this stage (others will be handled by #297).

I know this is not exactly what #308 asked for. I wonder if anyone really wants to use the "soft" mode - warn only, don't reject the query. If not, I think it's good as just a configuration option.

Closing issues

fixes #308

[ITAYC0HEN]

link is not clickable, I guess it should :)

[msm-code]

Not possible with an error message (without severe hacks, like putting HTML code into a Python exception, or - even worse - parsing the error message on the frontend).

I guess I'll do it the harder way, and put the related alert logic into the frontend. This will require some changes, so I'll address this a bit later (in this PR).

[ITAYC0HEN]

I wonder if anyone really wants to use the "soft" mode - warn only

Sometimes users really want to just list all the files that has a certain imphash value, and it's very important for their research. (i hope this falls under "degenerate query")

pe.imphash() == "b8bb385806b89680e13fc0cf24f4431e"

On other times, users don't really understand the problem.

Combining warn-only mode, and #297 will make sure that even if the user really wants to proceed, they can and it will not be too bad for the system (thanks to the file-limit)

[nazywam]

We could probably call rule.parse() just once, somewhere above? In case it gets expensive in the future

[msm-code]

Performance hit is negliglible, and even if it wasn't, it's already cached:

    def parse(self) -> UrsaExpression:
        if self.__parsed is None:
            self.__parsed = self.__parse_internal()
        return self.__parsed

I worry more about readability. There's no way to assign it to a temporary variable in a list comprehension, but I can rewrite this all to an explicit for loop if you prefer 🤔 It would look like this:

response_rules = []
for rule in rules:
    parsed_rule = rule.parse()
    response_rules.append(
        ParseResponseSchema(
            rule_name=rule.name,
            rule_author=rule.author,
            is_global=rule.is_global,
            is_private=rule.is_private,
            is_degenerate=parsed_rule.is_degenerate,
            parsed=parsed_rule.query,
        )
    )
return response_rules

I prefer the current version, but I'm not overly attached to it if you prefer this one.

[msm-code]

(i hope this falls under "degenerate query")

It does :). No viable ngrams == "degenerate".

Combining warn-only mode, and #297 will make sure that even if the user really wants to proceed, they can and it will not be too bad for the system (thanks to the file-limit)

ACK, thanks for input. I'll rework the code then to just warn the user in the 'non-enforcing' mode.

[ITAYC0HEN]

So 3 modes? Allow\Ignore, Warn, Prevent\Block?

[ITAYC0HEN]

it can also be nice if file-limit can be configured to work:

1. always
2. only in warn and block modes for degenerate queries
3. only in block mode for degenerate queries
4. never

[msm-code]

So 3 modes? Allow\Ignore, Warn, Prevent\Block?

I was considering it, but I think just two modes - Warn and Block are enough. It will be very easy to bypass the warning, and even when we trust users absolutely it may be nice to get an information that rule is inefficient (this can for example happen when copy-pasting rules from the internet - sometimes they look good but have something like or pe.imphash == '1234' that ruins it for mquery. I have a WIP docs page about writing good yara rules).

So in short, I don't see the application for Always mode, but I'm not fundamentally against it if anyone needs it.

it can also be nice if file-limit can be configured to work:

Haven't thought of that yet. I don't immediately like the idea, because:

• it adds a dependency between otherwise independent features)
• Degenerate (sorry for the name btw, best I could do) queries are not the only ones hard for mquery. I've tested with https://github.com/Neo23x0/signature-base on a dataset with 750k malicious files, and even though most of the rules returned way less than 1% of dataset to scan, there were some really bad (for mquery) rules. Especially top3 worst rules were "almost as bad" as degenerate ones: https://github.com/Neo23x0/signature-base/blob/master/yara/gen_excel_auto_open_evasion.yar (712806 files), https://github.com/Neo23x0/signature-base/blob/master/yara/crime_ransom_darkside.yar (695732 files) and https://github.com/Neo23x0/signature-base/blob/master/yara/gen_xor_hunting.yar (662770 files).

But this is probably discussion for #297

[msm-code]

@nazywam @ITAYC0HEN can you re-review?

Changes since the last time:

• I've changed query_disallow_degenerate to query_allow_slow. Nitpicker in me doesn't like the new name (slow queries are still possible, just harder), but it's probably easier to understand
• If that config key is set to "false" or empty, database will reject the query outright, and link the docs (I've updated them too)
• If that config is set to "true", database will ask the user if they're sure, the query button will change to scary red, and the search will start if the user clicks it again:

Also I've sneakily fixed broken CSS in the dropdowns.

Things I didn't change:

• Link is still not clickable. After tackling that for a while, I think this would require quite a few changes in the error handling on both server and client (at least if we want to do it correctly). I don't feel it's important enough improvement to justify the number of changes (link is copy&pasteable so the user can easily copy it). Maybe we can make it a (low priority?) issue.
• As per my previous message, there's no "Allow/Ignore" mode, only "Warn" and "Prevent/Block"

[ITAYC0HEN]

LGTM overall. Didn't check, but I hope that this feature works smartly with API requests as well, in some intuitive way.
Or is it?

[msm-code]

Didn't check, but I hope that this feature works smartly with API requests as well, in some intuitive way. Or is it?

Don't know about smartly, but it works:

• The /api/query/ API has one new optional parameter - force_slow_queries (default is false). It works as expected - without it slow queries will be rejected. With the parameter set to true slow queries will be accepted (if mquery's configuration allows it).
• One caveat is that mquery error responses are not particularly API-friendly - just a JSON with an error message (no error code or something similar). So the API client doesn't know (without parsing the error message) what is the error cause (slow yara rule, syntax error, real internal server error, etc).