feat(policies): Add granular policy_sets (#3086)

* Initial work.

* Periodic push.

* Fmt and start adding args to approve_policies cmd.

* keep funcs for now.

* Periodic push.

* Periodic push.

* fmt.

* Move approve policies logic to project_command_runner.

* update some tests

* More test fixes.

* update more tests. fix som logic.

* more tests. add additional info to common data for custom templates.

* fix apply with policies bug. update more tests/fmt

* file perms

* fix error parsing for conftest results.

* Update more tests and linting.

* update documentation.

* Address no-fail case. Address comments.

* Forgot changes.

* fix markdown renderer

* Fix policy fail logic. remove uneeded tmpl var

* targeted policy approvals fix

* Address PR comments.

* empty commit to trigger build

---------

Co-authored-by: PePe Amengual <jose.amengual@gmail.com>
Co-authored-by: rkstrickland <ross.strickland@instacart.com>
Co-authored-by: Dylan Page <dylan.page@autodesk.com>

runatlantis.io/docs/custom-workflows.md

@@ -456,6 +456,8 @@ Or a custom command
   * `SHOWFILE` - Absolute path to the location where Atlantis expects the plan in json format to
   either be generated (by show) or already exist (if running policy checks). Can be used to
   override the built-in `plan`/`apply` commands, ex. `run: terraform show -json $PLANFILE > $SHOWFILE`.
+  * `POLICYCHECKFILE` - Absolute path to the location of policy check output if Atlantis runs policy checks.
+  See [policy checking](/docs/policy-checking.html#data-for-custom-run-steps) for information of data structure.
   * `BASE_REPO_NAME` - Name of the repository that the pull request will be merged into, ex. `atlantis`.
   * `BASE_REPO_OWNER` - Owner of the repository that the pull request will be merged into, ex. `runatlantis`.
   * `HEAD_REPO_NAME` - Name of the repository that is getting merged into the base repository, ex. `atlantis`.

runatlantis.io/docs/policy-checking.md

@@ -16,7 +16,7 @@ Enabling "policy checking" in addition to the [mergeable apply requirement](/doc
 
 ![Policy Check Apply Status Failure](./images/policy-check-apply-status-failure.png)
 
-Any failures need to either be addressed in a successive commit, or approved by a blessed owner. This approval is independent of the approval apply requirement which can coexist in the policy checking workflow. After an approval, the apply can proceed.
+Any failures need to either be addressed in a successive commit, or approved by top-level owner(s) of policies or the owner(s) of the policy set in question. Policy approvals are independent of the approval apply requirement which can coexist in the policy checking workflow. After policies are approved, the apply can proceed.
 
 ![Policy Check Approval](./images/policy-check-approval.png)
 
@@ -44,14 +44,23 @@ policies:
     users:
       - nishkrishnan
   policy_sets:
-    - name: null_resource_warning
-      path: <CODE_DIRECTORY>/policies/null_resource_warning/
+    - name: deny_null_resource
+      path: <CODE_DIRECTORY>/policies/deny_null_resource/
       source: local
+    - name: deny_local_exec
+      path: <CODE_DIRECTORY>/policies/deny_local_exec/
+      source: local
+      approve_count: 2
+      owners:
+        users:
+          - pseudomorph
 ```
 
 - `name` - A name of your policy set.
 - `path` - Path to a policies directory. *Note: replace `<CODE_DIRECTORY>` with absolute dir path to conftest policy/policies.*
 - `source` - Tells atlantis where to fetch the policies from. Currently you can only host policies locally by using `local`.
+- `owners` - Defines the users/teams which are able to approve a specific policy set.
+- `approve_count` - Defines the number of approvals needed to bypass policy checks. Defaults to the top-level policies configuration, if not specified.
 
 By default conftest is configured to only run the `main` package. If you wish to run specific/multiple policies consider passing `--namespace` or `--all-namespaces` to conftest with [`extra_args`](https://www.runatlantis.io/docs/custom-workflows.html#adding-extra-arguments-to-terraform-commands) via a custom workflow as shown in the below example.
 
@@ -158,3 +167,21 @@ workflows:
 ### Quiet policy checks
 
 By default, Atlantis will add a comment to all pull requests with the policy check result - both successes and failures. Version 0.21.0 added the [`--quiet-policy-checks`](server-configuration.html#quiet-policy-checks) option, which will instead only add comments when policy checks fail, significantly reducing the number of comments when most policy check results succeed.
+
+
+### Data for custom run steps
+
+When the policy check workflow runs, a file is created in the working directory which contains information about the status of each policy set tested. This data may be useful in custom run steps to generate metrics or notifications. The file contains JSON data in the following format:
+
+```json
+[
+  {
+    "PolicySetName":  "policy1",
+    "ConftestOutput": "",
+    "Passed":         false,
+    "ReqApprovals":   1,
+    "CurApprovals":   0
+  }
+]
+
+```

runatlantis.io/docs/server-side-repo-config.md

@@ -519,11 +519,12 @@ If you set a workflow with the key `default`, it will override this.
 
 ### Policies
 
-| Key                    | Type            | Default | Required  | Description                              |
-|------------------------|-----------------|---------|-----------|------------------------------------------|
-| conftest_version       | string          | none    | no        | conftest version to run all policy sets  |
-| owners                 | Owners(#Owners) | none    | yes       | owners that can approve failing policies |
-| policy_sets            | []PolicySet     | none    | yes       | set of policies to run on a plan output  |
+| Key                    | Type            | Default | Required  | Description                                              |
+|------------------------|-----------------|---------|-----------|----------------------------------------------------------|
+| conftest_version       | string          | none    | no        | conftest version to run all policy sets                  |
+| owners                 | Owners(#Owners) | none    | yes       | owners that can approve failing policies                 |
+| approve_count          | int             | 1       | no        | number of approvals required to bypass failing policies. |
+| policy_sets            | []PolicySet     | none    | yes       | set of policies to run on a plan output                  |
 
 ### Owners
 | Key         | Type              | Default | Required   | Description                                             |

server/controllers/events/events_controller_e2e_test.go

@@ -1202,6 +1202,7 @@ func setupE2E(t *testing.T, repoDir string, opt setupOption) (events_controllers
 	Ok(t, err)
 
 	projectCommandRunner := &events.DefaultProjectCommandRunner{
+		VcsClient:        e2eVCSClient,
 		Locker:           projectLocker,
 		LockURLGenerator: &mockLockURLGenerator{},
 		InitStepRunner: &runtime.InitStepRunner{

server/controllers/events/testdata/test-repos/policy-checks-apply-reqs/exp-output-auto-policy-check.txt

@@ -1,15 +1,29 @@
 Ran Policy Check for dir: `.` workspace: `default`
 
-**Policy Check Error**
-```
-exit status 1
-Checking plan against the following policies: 
-  test_policy
+**Policy Check Failed**: Some policy sets did not pass.
+#### Policy Set: `test_policy`
+```diff
 FAIL - <redacted plan file> - main - WARNING: Null Resource creation is prohibited.
 
 1 test, 0 passed, 0 warnings, 1 failure, 0 exceptions
 
 ```
-* :heavy_check_mark: To **approve** failing policies an authorized approver can comment:
+
+
+#### Policy Approval Status:
+```
+policy set: test_policy: requires: 1 approval(s), have: 0.
+```
+* :heavy_check_mark: To **approve** this project, comment:
+    * `atlantis approve_policies -d .`
+* :put_litter_in_its_place: To **delete** this plan click [here](lock-url)
+* :repeat: To re-run policies **plan** this project again by commenting:
+    * `atlantis plan -d .`
+
+---
+* :heavy_check_mark: To **approve** all unapplied plans from this pull request, comment:
     * `atlantis approve_policies`
-* :repeat: Or, address the policy failure by modifying the codebase and re-planning.
+* :put_litter_in_its_place: To delete all plans and locks for the PR, comment:
+    * `atlantis unlock`
+* :repeat: To re-run policies **plan** this project again by commenting:
+    * `atlantis plan`

server/controllers/events/testdata/test-repos/policy-checks-diff-owner/exp-output-approve-policies.txt

@@ -1,4 +1,29 @@
+Ran Approve Policies for 1 projects:
+
+1. dir: `.` workspace: `default`
+
+### 1. dir: `.` workspace: `default`
 **Approve Policies Error**
 ```
-contact policy owners to approve failing policies
+1 error occurred:
+	* policy set: test_policy user runatlantis is not a policy owner - please contact policy owners to approve failing policies
+
+
 ```
+#### Policy Approval Status:
+```
+policy set: test_policy: requires: 1 approval(s), have: 0.
+```
+* :heavy_check_mark: To **approve** this project, comment:
+    * `atlantis approve_policies -d .`
+* :put_litter_in_its_place: To **delete** this plan click [here](lock-url)
+* :repeat: To re-run policies **plan** this project again by commenting:
+    * `atlantis plan -d .`
+
+---
+* :heavy_check_mark: To **approve** all unapplied plans from this pull request, comment:
+    * `atlantis approve_policies`
+* :put_litter_in_its_place: To delete all plans and locks for the PR, comment:
+    * `atlantis unlock`
+* :repeat: To re-run policies **plan** this project again by commenting:
+    * `atlantis plan`

server/controllers/events/testdata/test-repos/policy-checks-diff-owner/exp-output-auto-policy-check.txt

@@ -1,15 +1,29 @@
 Ran Policy Check for dir: `.` workspace: `default`
 
-**Policy Check Error**
-```
-exit status 1
-Checking plan against the following policies: 
-  test_policy
+**Policy Check Failed**: Some policy sets did not pass.
+#### Policy Set: `test_policy`
+```diff
 FAIL - <redacted plan file> - main - WARNING: Null Resource creation is prohibited.
 
 1 test, 0 passed, 0 warnings, 1 failure, 0 exceptions
 
 ```
-* :heavy_check_mark: To **approve** failing policies an authorized approver can comment:
+
+
+#### Policy Approval Status:
+```
+policy set: test_policy: requires: 1 approval(s), have: 0.
+```
+* :heavy_check_mark: To **approve** this project, comment:
+    * `atlantis approve_policies -d .`
+* :put_litter_in_its_place: To **delete** this plan click [here](lock-url)
+* :repeat: To re-run policies **plan** this project again by commenting:
+    * `atlantis plan -d .`
+
+---
+* :heavy_check_mark: To **approve** all unapplied plans from this pull request, comment:
     * `atlantis approve_policies`
-* :repeat: Or, address the policy failure by modifying the codebase and re-planning.
+* :put_litter_in_its_place: To delete all plans and locks for the PR, comment:
+    * `atlantis unlock`
+* :repeat: To re-run policies **plan** this project again by commenting:
+    * `atlantis plan`

server/controllers/events/testdata/test-repos/policy-checks-extra-args/exp-output-auto-policy-check.txt

@@ -1,15 +1,29 @@
 Ran Policy Check for dir: `.` workspace: `default`
 
-**Policy Check Error**
-```
-exit status 1
-Checking plan against the following policies: 
-  test_policy
+**Policy Check Failed**: Some policy sets did not pass.
+#### Policy Set: `test_policy`
+```diff
 FAIL - <redacted plan file> - null_resource_policy - WARNING: Null Resource creation is prohibited.
 
 1 test, 0 passed, 0 warnings, 1 failure, 0 exceptions
 
 ```
-* :heavy_check_mark: To **approve** failing policies an authorized approver can comment:
+
+
+#### Policy Approval Status:
+```
+policy set: test_policy: requires: 1 approval(s), have: 0.
+```
+* :heavy_check_mark: To **approve** this project, comment:
+    * `atlantis approve_policies -d .`
+* :put_litter_in_its_place: To **delete** this plan click [here](lock-url)
+* :repeat: To re-run policies **plan** this project again by commenting:
+    * `atlantis plan -d .`
+
+---
+* :heavy_check_mark: To **approve** all unapplied plans from this pull request, comment:
     * `atlantis approve_policies`
-* :repeat: Or, address the policy failure by modifying the codebase and re-planning.
+* :put_litter_in_its_place: To delete all plans and locks for the PR, comment:
+    * `atlantis unlock`
+* :repeat: To re-run policies **plan** this project again by commenting:
+    * `atlantis plan`

server/controllers/events/testdata/test-repos/policy-checks-multi-projects/exp-output-auto-policy-check.txt

@@ -4,13 +4,14 @@ Ran Policy Check for 2 projects:
 1. dir: `dir2` workspace: `default`
 
 ### 1. dir: `dir1` workspace: `default`
+#### Policy Set: `test_policy`
 ```diff
-Checking plan against the following policies: 
-  test_policy
 
 1 test, 1 passed, 0 warnings, 0 failures, 0 exceptions
+
 ```
 
+
 * :arrow_forward: To **apply** this plan, comment:
     * `atlantis apply -d dir1`
 * :put_litter_in_its_place: To **delete** this plan click [here](lock-url)
@@ -19,22 +20,30 @@ Checking plan against the following policies:
 
 ---
 ### 2. dir: `dir2` workspace: `default`
-**Policy Check Error**
-```
-exit status 1
-Checking plan against the following policies: 
-  test_policy
+**Policy Check Failed**: Some policy sets did not pass.
+#### Policy Set: `test_policy`
+```diff
 FAIL - <redacted plan file> - main - WARNING: Forbidden Resource creation is prohibited.
 
 1 test, 0 passed, 0 warnings, 1 failure, 0 exceptions
 
 ```
-* :heavy_check_mark: To **approve** failing policies an authorized approver can comment:
-    * `atlantis approve_policies`
-* :repeat: Or, address the policy failure by modifying the codebase and re-planning.
+
+
+#### Policy Approval Status:
+```
+policy set: test_policy: requires: 1 approval(s), have: 0.
+```
+* :heavy_check_mark: To **approve** this project, comment:
+    * `atlantis approve_policies -d dir2`
+* :put_litter_in_its_place: To **delete** this plan click [here](lock-url)
+* :repeat: To re-run policies **plan** this project again by commenting:
+    * `atlantis plan -d dir2`
 
 ---
-* :fast_forward: To **apply** all unapplied plans from this pull request, comment:
-    * `atlantis apply`
+* :heavy_check_mark: To **approve** all unapplied plans from this pull request, comment:
+    * `atlantis approve_policies`
 * :put_litter_in_its_place: To delete all plans and locks for the PR, comment:
-    * `atlantis unlock`
+    * `atlantis unlock`
+* :repeat: To re-run policies **plan** this project again by commenting:
+    * `atlantis plan`

server/controllers/events/testdata/test-repos/policy-checks/exp-output-auto-policy-check.txt

@@ -1,15 +1,29 @@
 Ran Policy Check for dir: `.` workspace: `default`
 
-**Policy Check Error**
-```
-exit status 1
-Checking plan against the following policies: 
-  test_policy
+**Policy Check Failed**: Some policy sets did not pass.
+#### Policy Set: `test_policy`
+```diff
 FAIL - <redacted plan file> - main - WARNING: Null Resource creation is prohibited.
 
 1 test, 0 passed, 0 warnings, 1 failure, 0 exceptions
 
 ```
-* :heavy_check_mark: To **approve** failing policies an authorized approver can comment:
+
+
+#### Policy Approval Status:
+```
+policy set: test_policy: requires: 1 approval(s), have: 0.
+```
+* :heavy_check_mark: To **approve** this project, comment:
+    * `atlantis approve_policies -d .`
+* :put_litter_in_its_place: To **delete** this plan click [here](lock-url)
+* :repeat: To re-run policies **plan** this project again by commenting:
+    * `atlantis plan -d .`
+
+---
+* :heavy_check_mark: To **approve** all unapplied plans from this pull request, comment:
     * `atlantis approve_policies`
-* :repeat: Or, address the policy failure by modifying the codebase and re-planning.
+* :put_litter_in_its_place: To delete all plans and locks for the PR, comment:
+    * `atlantis unlock`
+* :repeat: To re-run policies **plan** this project again by commenting:
+    * `atlantis plan`

server/core/config/parser_validator_test.go

@@ -1466,12 +1466,14 @@ policies:
 					"custom1": customWorkflow1,
 				},
 				PolicySets: valid.PolicySets{
-					Version: conftestVersion,
+					Version:      conftestVersion,
+					ApproveCount: 1,
 					PolicySets: []valid.PolicySet{
 						{
-							Name:   "good-policy",
-							Path:   "rel/path/to/policy",
-							Source: valid.LocalPolicySet,
+							Name:         "good-policy",
+							Path:         "rel/path/to/policy",
+							Source:       valid.LocalPolicySet,
+							ApproveCount: 1,
 						},
 					},
 				},
@@ -1801,12 +1803,14 @@ func TestParserValidator_ParseGlobalCfgJSON(t *testing.T) {
 					"custom": customWorkflow,
 				},
 				PolicySets: valid.PolicySets{
-					Version: conftestVersion,
+					Version:      conftestVersion,
+					ApproveCount: 1,
 					PolicySets: []valid.PolicySet{
 						{
-							Name:   "good-policy",
-							Path:   "rel/path/to/policy",
-							Source: valid.LocalPolicySet,
+							Name:         "good-policy",
+							Path:         "rel/path/to/policy",
+							Source:       valid.LocalPolicySet,
+							ApproveCount: 1,
 						},
 					},
 				},