Integration testing via example yaml files in each policy directory

[ctab]

We can add integration testing based on each of the policy directories' templates, constraints, and example resource yaml files. Using the standard from #25, we can require that all resources containing "_allowed*.yaml" pass and "_disallowed*.yaml" are rejected. (We'll need to ensure those files exist first -- #26)

We have a couple options here. The kpt tool has a Gatekeeper resource validator to evaluate a resource, template, and constraint against the OPA client. We can also spin up a kind cluster (similar to in the Gatekeeper repo) and apply the resources there. I imagine the kpt approach is faster but kind more representative of real-world usage.

[Kevinma1995]

Hello @ctab, I'm @shomron 's colleague who is interested in getting involved in the gatekeeper project. Just wondering how can I get started and help with this issue?

[ctab]

Hey Kevin, thanks for stepping in to help!

If you haven't already seen it, I have a doc here that summarizes what we are trying to achieve here. https://docs.google.com/document/d/1uQlkIBQcgNNyth8o9ufYaVSfUq_JLJGy9fcuW3VB7vU/edit

For this issue, what I have in mind is something similar to what we have in the Gatekeeper repo with make e2e-bootstrap and make test-e2e. The goal will be to able to spin up a kind cluster (with make e2e-bootstrap) and then iterate through each of the policies in the library, running the following tests:

• apply the template.yaml file
• apply each sample constraint.yaml file. (There is currently only 1 constraint per template but this should support more than one)
• apply each example yaml file in the sample directory and test that the resource in the file is either admitted (if the filename is of the format example_allowed*.yaml) or denied (example_disallowed*.yaml).
• delete this constraint and any examples from the test cluster before moving on to the next one, and the template when done with that directory.

This will depend on #31 which establishes the directory structure for this work. You could branch off of that and get started now, at risk that there may be some changes as it is reviewed, or you may want to wait until that is merged into master. I don't foresee any major changes though since we already discussed the broad strokes of this during our community meetings.

Feel free to message me on the OPA slack if you want to discuss!

[Kevinma1995]

Hey Craig, thanks for the prompt response and detailed explanation. I'm clear about what we want to achieve for this issue and will start working on it shortly!