Basic support for generic json document scanning

[BrentSouza]

My company is currently evaluating Bridgecrew and one of the things that we're looking for in a policy as code framework is the ability to scan arbitrary json input. The thinking here is that:

1. It's generally easy to get some kind of configuration from various components in our stack as JSON output.
2. It's relatively simple to write checkov custom checks against these config docs
3. It's considerably more effort to create a fully-fledged runner implementation for each component of our stack that we want to check

Using the json parser from the terraform plan implementation, this turns out to be pretty trivial to implement with a very basic feature set. This is a very quick first pass at providing the following functionality:

• Parse JSON documents into a relevant python dict, list, etc
• A base JSON check that should be used to write custom checks against documents
• Narrow the scope of a check to relevant properties within a JSON document using JMESpath-lite syntax, e.g. a.b.c[0]
• Run checks against all items in a JSON array
• Run checks against properties of a JSON object
• Run checks against an entire JSON document

I've tested this locally with some checks we would like to run against our internal GitLab repositories, which can help show how this feature is useful.

Two checks we would like to run are:

1. Users can't push changes directly into a default branch
2. Merge Requests require at least 2 approvals

Here's an example of the MR approvals check:

from checkov.common.models.enums import CheckCategories, CheckResult
from checkov.json.base_json_check import BaseJsonCheck
from checkov.json.enums import BlockType


class MergeRequestRequiresApproval(BaseJsonCheck):
    def __init__(self):
        name = "Merge requests should require at least 2 approvals"
        id = "CKV_GITLAB_2"
        categories = [CheckCategories.CONVENTION]
        super().__init__(
            name=name,
            id=id,
            categories=categories,
            supported_entities=["approvals"],
            block_type=BlockType.OBJECT,
        )

    def scan_entity_conf(self, conf):
        if conf["approvals"]["approvals_before_merge"] < 2:
            for rule in conf["approval_rules"]:
                if rule["approvals_required"] >= 2:
                    return CheckResult.PASSED, rule
            return CheckResult.FAILED, conf
        return CheckResult.PASSED, conf["approvals"]


check = MergeRequestRequiresApproval()

Before we go much further with building this out, I wanted to get the checkov team's feedback on the concept of a generic json runner. Is this something that makes sense to include with the project?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[schosterbarak]

are we populating this var with a value somewhere?

[BrentSouza]

Yes. This is using the json parser that is already in the codebase for terraform plans which captures line numbers along with the deserialized values. The custom checks should return the passing/failing configuration along with the check status. That configuration will have the start/end mark properties as a result of the tf plan parser. Does that make sense?

[BrentSouza]

I was also wondering if you would want to do something like move the tf plan parser into common as a general JSON parser? There doesn't seem to be anything terraform specific about it, but I didn't want to move something like that without discussing it first.

[schosterbarak]

i think it is the right thing to do. you'll need to rebase from master because we had a contribution with performance enhancement there a day ago

[BrentSouza]

@schosterbarak I started pulling all of the json parsing out into common.parsers and common.parsers.json and updated arm, cloudformation, serverless, and terraform runners to use the common json parser. Please let me know if this looks ok!

[nimrodkor]

Very nice @BrentSouza !

Some minor comments

[schosterbarak]

thank you @BrentSouza don't forget to apply for hacktober fest 👍 https://bridgecrew.io/blog/happy-hacktoberfest-2021/

[metahertz]

Epic contribution @BrentSouza!!!!

🎉