More help requested! accessing configuration_modules within the plan

[dwilliams782]

I promise I'll stop asking for help soon.... 😃

I want to use some of the information in the plan to implement some rules that do things like:

1. Check the module source for each module (using configuration.root_module.module_calls)
2. Check for actions (using configuration.resource_changes)

I found you guys are already doing the heavy lifting here but I can't work out how I can actually access these values in a rule?

I'm sure this is down to my lack of knowledge around rego namespaces / imports, so I apologise if this is an obvious question!

[jaspervdj-luminal]

No worries at all!

Yes -- we try to do the heavy lifting, especially when Terraform modules are
involved. One of the ideas behind Regula is that we want to focus on making
rules as easy as possible to write. Terraform plans can be quite complex and
you don't want that "preprocessing" to get in the way of the "business logic"
of the rule.

This is what the resources_view modules do: grab all resources from all
modules and present them as a flat array, with all properties filled in. This
makes it easier to write rules against the "desired state" of resources rather
than going through the changes etc. manually.

However: this currently includes data about the resources themselves.
Meta-information about the resources (like the source of the module) is
currently not present in there. We're open to exploring this option
though, especially if you have a use case. We could do something similar with
meta-information and try to present it in a convenient way to the user.

There is a workaround though -- you can circumvent the resources_view and
operate on the plan directly. We can even pretend that the modules are
resources themselves. The following rule does something like that:

package rules.my_rule
import data.fugue

# Pull out the module calls and make them look like regular resources
module_call_resources := { name: module_resource |
    # Access this through `fugue.plan`
    module = fugue.plan.configuration.root_module.module_call_resources[name]
    module_resource = {
        "id": name,
        "_type": "module_call",
        "_provider": "aws",
        "source": module.source
    }
}

# Tells Regula to work on the entire config rather than passing this rule
# a specific resource
resource_type = "MULTIPLE"
# Some list of approved modules
allowed_modules := {
    "terraform-aws-modules/lambda/aws",
}

# Check that the module source is in the approved list and allow / deny it
policy[p] {
    m = module_call_resources[_]
    allowed_modules[m.source]
    p = fugue.allow({
        "resource": m,
    })
} {
    m = module_call_resources[_]
    not allowed_modules[m.source]
    p = fugue.deny({
        "resource": m,
        "message": sprintf("This module is not in our approved list: %s", [m.source]),
    })
}

That being said; this is a slight workaround and we will look at presenting this
sort of metadata in the resource view so it can be directly accessed by rules.
If you have other use cases for accessing metadata, please let us know as it
would help design this.

On a different note -- we're currently building a CLI for Regula that still uses
Regula as a library, but provides a bunch of additional nice features that we
can't implement in Rego itself (e.g. file discovery). If you're interested in
trying out a prototype of that, please reach out to us at regula@fugue.co!

[dwilliams782]

Hi,

Thanks for a very detailed answer! I'll play with that and let you know how I get on. My two immediate use cases for wanting to access this information are:

1. Make sure that modules are a) from our approved list and b) We want to be able to make sure our users upgrade to new versions of our released modules, so will use the module source tag to compare SemVers
2. Deny changes in the plan such as:
   a) new firewall rules created or deleted in prod networks
   b) no additional / new resources created of resource type X (i.e. allow what is already present but block new types going forward, things like null resources!)

I'll drop an email and have a play with the CLI, sounds interesting!

FYI: We have combined Regula and Conftest, and are utilising this via run atlantis so we have compliance checks built into our PR approval process. With the example above of blocking resource creation, we can then utilise runatlantis's feature of atlantis approve_policies so that we can require designated owners to have to approve anything that fails our regula / conftest checks 😃

[dwilliams782]

Hey @jaspervdj-luminal

I had to change a couple of things in your example, wonder if my version of Regula is behind yours? See commented lines

package rules.my_rule
import data.fugue

# Pull out the module calls and make them look like regular resources
module_call_resources := { name: module_resource |
    # Access this through `fugue.plan`
    #module = fugue.plan.configuration.root_module.module_call_resources[name]
    module = fugue.plan.configuration.root_module.module_calls[name]
    module_resource = {
        "id": name,
        "_type": "module_call",
        "_provider": "aws",
        "source": module.source
    }
}

# Tells Regula to work on the entire config rather than passing this rule
# a specific resource
resource_type = "MULTIPLE"
# Some list of approved modules
allowed_modules := {
    "terraform-aws-modules/lambda/aws",
}

# Check that the module source is in the approved list and allow / deny it
policy[p] {
    m = module_call_resources[_]
    allowed_modules[m.source]
    p = fugue.allow_resource(m)
    #p = fugue.allow({
    #    "resource": m,
    #})
} {
    m = module_call_resources[_]
    not allowed_modules[m.source]
    p = fugue.deny_resource_with_message(m, sprintf("This module is not in our approved list: %s", [m.source]))
    #p = fugue.deny({
     #   "resource": m,
     #   "message": sprintf("This module is not in our approved list: %s", [m.source]),
    #})
}

---

I've built this out to compare the git ref / tag, as we wanted to enforce a minimum version for our modules (i.e. if we release a new version of a module with a required label, we can now ensure everyone upgrades).

It's not clean and I'm sure it can be optimised, but it's getting late on a Friday so it will do for now:

package rules.LH_06_check_module_source
import data.fugue

# TODO: Add support for child modules
# These are configuration.root_module.module_calls.<name>.module.module_calls
# TODO: Add support for git modules with no ref
# TODO: Do we want to allow git HTTPS?

__rego__metadoc__ := {
    "id": "LH_06",
    "title": "All module usage must be preapproved",
    "description": "This checks the module source and tag against the whitelist.",
}

# Approved modules and minimum versions
# We are only evaluating github sources, any local module will pass for now
allowed_modules := {
    "loveholidays/terraform-module-gcs": "v0.9.0",
    "loveholidays/terraform-module-gke-cluster": "v1.5.0",
    "loveholidays/terraform-module-gke-nodepool": "v1.6.0",
    "loveholidays/terraform-module-network": "v1.1.0",
    "loveholidays/terraform-module-service-account": "v1.9.0",
    "loveholidays/terraform-module-shared-vpc": "v1.4.0"
}

# Tells Regula to work on the entire config rather than passing this rule
# a specific resource
resource_type = "MULTIPLE"

# Pull out the module calls and make them look like regular resources
module_call_resources := { name: module_resource |
    # Access this through `fugue.plan`
    module = fugue.plan.configuration.root_module.module_calls[name]
    module_resource = {
        "id": name,
        "_type": "module_call",
        "_provider": "google",
        "source": module.source
    }
}

is_source_github_ssh(source) {
    contains(source, "git@github.com:")
}

module_name_and_version(source) = ret {
    s := split(source, "git@github.com:")[1]
    v := split(s, ".git?ref=")
    ret := {"name": v[0], "version": v[1]}
}

valid_version(source, min) {
    # Provided tag is expected as either semver or branch/short_sha
    v := trim_prefix(source["version"], "v")
    m := trim_prefix(min, "v")
    semver.is_valid(v)
    semver.compare(m, v) <= 0
}

valid_version(source, min) {
    # If branch / short_sha, only direct comparison is possible
    v := trim_prefix(source["version"], "v")
    not semver.is_valid(v)
    source["version"] == min
}

# Check that the module source is in the approved list and allow / deny it
policy[p] {
    m = module_call_resources[_]
    is_source_github_ssh(m.source)
    source := module_name_and_version(m.source)
    min := allowed_modules[source["name"]]
    valid_version(source, min)
    p = fugue.allow_resource(m)
} {
    m = module_call_resources[_]
    # Currently we won't evaluate non-gh modules, just assuming a pass for now
    not is_source_github_ssh(m.source)
    p = fugue.allow_resource(m)
} {
    m = module_call_resources[_]
    is_source_github_ssh(m.source)
    source := module_name_and_version(m.source)
    not allowed_modules[source["name"]]
    msg := sprintf("This module is not in our approved list: %s", [source["name"]])
    p = fugue.deny_resource_with_message(m, msg)
} {
    m = module_call_resources[_]
    is_source_github_ssh(m.source)
    source := module_name_and_version(m.source)
    min := allowed_modules[source["name"]]
    not valid_version(source, min)
    msg := sprintf("Module ref: %s does not meet minimum required: %s", [source["version"], min])
    p = fugue.deny_resource_with_message(m, msg)
}

As seen in the TODO, this currently doesn't support child_modules, I've seen some examples in the codebase of using walk to fix this, so I'll have to look at that next week, for now it's late on a Friday. Thanks again for your help, loving this tool! Feel free to close.

[dwilliams782]

This is what I came up with for nested child modules, bit hacky but it will do until (hopefully) this information is available natively via regula :)

module_call_resources := { name: module_resource |
    [path, value] = walk(fugue.plan.configuration.root_module)
     # path is a list that points to the nested module, such as:

    #"path": [
    #            "module_calls",
    #            "test_child",
    #            "module",
    #            "module_calls",
    #            "nested_inside_child",
    #            "module",
    #            "module_calls",
    #            "nested_inside_child_module"
    #        ],

    # Our end goal is a string that shows the module path:
    # module.test_child.module.nested_inside_child.module.nested_inside_child_module

    # We have erroneous amounts of "module" and "module_calls", so strip them:
    module_names := [ p |
        	p := path[_]
            p != "module"
            p != "module_calls"
        ]

    # This gives us a list of just module names:
    #"module_names": [
    #            "test_child",
    #            "nested_inside_child",
    #            "nested_inside_child_module"
    #            ],

    # We can then concat / sprintf to get our desired path string
    name := sprintf("module.%s", [concat(".module.", module_names)])
    module_resource = {
        "id": name,
        "_type": "module_call",
        "_provider": "google",
        "source": value.source
    }
}

[jaspervdj-luminal]

Awesome, yeah, that looks good! I've added a ticket for making this information more easily available, I'll reach out if we have something working.