Policy pack configuration using policy enable cmd #4127

sean1588 · 2020-03-20T04:34:57Z

Support adding policy pack configuration through CLI by specifying a policy configuration json file.

Example cmd:

pulumi policy enable {orgName}/{policyPackName} --config config.json --version={version}

Right now all policy pack config validation is being done server side. Maybe if we feel there would be an advantage to doing config validation on the client we can move to that at some point, but I believe we would have to request the schema from the server anyway, so either way we would need to make an additional api call. I feel we can revisit this at some point if need be. We can always do a follow up PR if we feel doing validation client side is necessary.

ekrengel · 2020-03-23T18:28:36Z

CHANGELOG.md

@@ -8,6 +8,8 @@ CHANGELOG
  [#4141](https://github.com/pulumi/pulumi/pull/4141)
 - Add missing builtin `MapArray` to Go SDK.
  [#4144](https://github.com/pulumi/pulumi/pull/4144)
+- Add support for policy pack configuration using CLI.


Since the changelog is actually used for our releases, we may want to clarify this. "Add support for enabling Policy Packs with configuration" ... or something to that effect.

justinvp · 2020-03-23T18:20:37Z

pkg/cmd/util.go

@@ -702,3 +703,20 @@ func updateFlagsToOptions(interactive, skipPreview, yes bool) (backend.UpdateOpt
 		SkipPreview: skipPreview,
 	}, nil
 }
+
+// loadJSONConfigFile loads file with json configuration data and unmarshals into a map[string]*json.RawMessage object.
+func loadJSONConfigFile(filePath string) (map[string]*json.RawMessage, error) {


Rather than having this function here, I am wondering if we should call the same function we use to load local config specified to the --policy-pack-config flag of pulumi up/preview/watch.

pulumi/pkg/resource/analyzer/config.go

Lines 31 to 37 in 61928f0

func LoadPolicyPackConfigFromFile(file string) (map[string]plugin.AnalyzerPolicyConfig, error) {

b, err := ioutil.ReadFile(file)

if err != nil {

return nil, err

}

return parsePolicyPackConfig(b)

}

That way we have consistent behavior of loading local policy config from a file.

This existing function does some validation of the EnforcementLevel values, and also, as a convenience, allows specifying enforcement level for a policy like:

{ "some-policy": "advisory" }

Rather than having to specify it like:

{ "some-policy": { "enforcementLevel": "advisory" } }

This would require adding some code to convert map[string]plugin.AnalyzerPolicyConfig to map[string]*json.RawMessage. Probably the easiest way to do that is to loop through the map and json.Marshal each plugin.AnalyzerPolicyConfig value to bytes. The bytes can then be cast to json.RawMessage. Similar to what's done here:

pulumi/pkg/backend/httpstate/client/client.go

Lines 633 to 641 in c28c602

properties := map[string]*json.RawMessage{}

for k, v := range schema.Properties {

bytes, err := json.Marshal(v)

if err != nil {

return nil, err

}

raw := json.RawMessage(bytes)

properties[k] = &raw

}

Ok. I will leverage that function so we can use the validations it has and then just convert the types.

ekrengel

Some comments.. ALSO thinking we should definitely add a pulumi policy validate-config command to allow people to validate config before enabling. You can open another issue if you dont want to do it as part of this?

ekrengel · 2020-03-23T18:29:57Z

tests/integration/policy/test_policy_pack/package.json

-        "@pulumi/policy": "^0.2.0"
+        "@pulumi/pulumi": "1.13.0",
+        "@pulumi/aws": "1.27.0",
+        "@pulumi/policy": "0.4.1-dev.1584625475"


I think we talked about have one Policy Pack using an older version of the policy repo to ensure backwards compatibility.. is there a reason you changed your mind?

nope, was actually just a miss on my part. the only way I could get this to work was just to add another test policy pack directory with the old version of the policy sdk. Not sure how far back in versions you want to test, so I just used the version that we were using for the tests before I made these changes. I also just made a completely different test case with all the old existing tests, since the set of commands we were using in the test I believe are only backwards compatible not forwards compatible with the changes, so I had to keep a lot of the old commands that we were testing with previously as well, for example the way we were specifying pp versions.

ekrengel · 2020-03-23T18:30:39Z

tests/integration/policy/test_policy_pack/index.ts

+                configSchema: {
+                    properties: {
+                        message: {
+                            "type": "string",


nit: I think type, minLength and maxLength do not need to be quotes & may make this look cleaner.

ekrengel · 2020-03-23T18:34:38Z

pkg/cmd/policy_enable.go

 		}),
 	}

 	cmd.PersistentFlags().StringVar(
 		&args.policyGroup, "policy-group", "",
 		"The Policy Group for which the Policy Pack will be enabled; if not specified, the default Policy Group is used")

+	cmd.PersistentFlags().StringVar(
+		&args.configFile, "config-file", "",


I think when running a policy-pack locally there is a different flag name used for the config file. Should we use the same here for consistency?

For pulumi preview/up/watch, the flag is --policy-pack-config. Since this command is already about policies, I think we can leave off the policy-pack prefix, and just use --config.

ekrengel · 2020-03-23T18:35:48Z

pkg/cmd/util.go

@@ -702,3 +703,20 @@ func updateFlagsToOptions(interactive, skipPreview, yes bool) (backend.UpdateOpt
 		SkipPreview: skipPreview,
 	}, nil
 }
+
+// loadJSONConfigFile loads file with json configuration data and unmarshals into a map[string]*json.RawMessage object.


Since this is in a generic util file, maybe be more specific about its use in the comment? Or just move it to the pkg/cmd/policy_enable.go file. I dont think its used anywhere else?

Ok, I went ahead and relocated to policy_enable.co and then leveraged the function justin is suggesting since it does some validations.

justinvp · 2020-03-23T18:48:59Z

tests/integration/policy/test_policy_pack/configs/invalid-config.json

@@ -0,0 +1,6 @@
+{
+    "s3-no-public-read":{


Nit:

Suggested change

"s3-no-public-read":{

"s3-no-public-read": {

justinvp · 2020-03-23T18:49:12Z

tests/integration/policy/test_policy_pack/configs/invalid-config.json

@@ -0,0 +1,6 @@
+{
+    "s3-no-public-read":{
+        "enforcementLevel":"mandatory",


Nit:

Suggested change

"enforcementLevel":"mandatory",

"enforcementLevel": "mandatory",

justinvp · 2020-03-23T18:49:23Z

tests/integration/policy/test_policy_pack/configs/invalid-config.json

+{
+    "s3-no-public-read":{
+        "enforcementLevel":"mandatory",
+        "message":"this message is invalid because it is way more than 10 characters......"


Nit:

Suggested change

"message":"this message is invalid because it is way more than 10 characters......"

"message": "this message is invalid because it is way more than 10 characters......"

justinvp · 2020-03-23T18:49:31Z

tests/integration/policy/test_policy_pack/configs/valid-config.json

@@ -0,0 +1,10 @@
+{
+    "all":{


Nit:

Suggested change

"all":{

"all": {

justinvp · 2020-03-23T18:49:43Z

tests/integration/policy/test_policy_pack/configs/valid-config.json

@@ -0,0 +1,10 @@
+{
+    "all":{
+        "enforcementLevel":"advisory"


Nit:

Suggested change

"enforcementLevel":"advisory"

"enforcementLevel": "advisory"

justinvp · 2020-03-23T18:49:55Z

tests/integration/policy/test_policy_pack/configs/valid-config.json

+    },
+    "s3-no-public-read":
+    {
+        "enforcementLevel":"mandatory",


Nit:

Suggested change

"enforcementLevel":"mandatory",

"enforcementLevel": "mandatory",

justinvp · 2020-03-23T18:50:07Z

tests/integration/policy/test_policy_pack/configs/valid-config.json

+    "s3-no-public-read":
+    {
+        "enforcementLevel":"mandatory",
+        "message":"test msg"


Nit:

Suggested change

"message":"test msg"

"message": "test msg"

justinvp · 2020-03-23T23:13:18Z

pkg/cmd/policy_enable.go

+	// Convert type map[string]plugin.AnalyzerPolicyConfig to map[string]*json.RawMessage.
+	config := map[string]*json.RawMessage{}
+	for k, v := range analyzerPolicyConfig {
+		bytes, err := json.Marshal(v)


I don't think we can just marshal v directly.

Doing so results in JSON like:

{"EnforcementLevel":"mandatory","Properties":{"foo":"bar"}}

The value needs to be converted into the format expected by the service. Something like:

func marshalAnalyzerPolicyConfig(c AnalyzerPolicyConfig) (*json.RawMessage, error) { m := make(map[string]interface{}) for k, v := range c.Properties { m[k] = v } if c.EnforcementLevel != "" { m["enforcementLevel"] = c.EnforcementLevel } bytes, err := json.Marshal(m) if err != nil { return nil, err } raw := json.RawMessage(bytes) return &raw, nil }

Which will result in:

{"enforcementLevel":"mandatory","foo":"bar"}

I created a Go Playground to demonstrate: https://play.golang.org/p/L8GYTzTAsOR

Correct, I noticed it structured the configurable values under the Properties key. Thanks for adding that code snippet btw. I updated to convert the values.

sean1588 · 2020-03-24T17:15:26Z

OK, this is ready for a re-review. I made a few changes since last review:

I updated the code to load the policy config to use the function in the resource analyzer package that Justin suggested so that we could leverage some of the validations it had. This also involved doing some conversion to convert the type being returned from that function to what the api service was expecting.
I also added the tests to test for backwards compatibility with other versions of the policy sdk. The only way I could reasonably do this is by adding another test policy pack directory with the old version. We could probably do this in a way that we could dynamically manipulate this through code at runtime and have only one test policy pack, but it ended up being a PITA and not worth the time because we need to also generate the index.ts file as well that is compatible with that particular version of the sdk. I also a specific test case with all the old tests for that specific version and just copied over all the old tests we were running prior to the change. I needed to retain the old commands because they are only backwards compatible not forwards. It looks like a lot of changes and the test code could be dry-ed up to some degree but would probably make them less readable and hard to follow, so I think its best not to IMO and just leave as is.

ekrengel · 2020-03-24T17:46:14Z

tests/integration/policy/policy_pack_v0-4-1-dev/package.json

@@ -0,0 +1,14 @@
+{


wondering if it makes more sense to call this dir policy_pack_w_config since the version is probably meaningless to most people?

Agreed, that's a more meaningful name. will update

ekrengel · 2020-03-24T17:49:56Z

tests/integration/policy/policy_test.go

@@ -14,9 +14,61 @@ import (
 )

 // TestPolicy tests policy related commands work.
-func TestPolicy(t *testing.T) {
-	t.Skip("Temporarily skipping test that is causing unrelated tests to fail - pulumi/pulumi#4149")
+func TestPolicyVersion0_4_1_dev(t *testing.T) {


Same with naming here.

ekrengel · 2020-03-24T17:52:16Z

tests/integration/policy/policy_test.go

+	e.RunCommand("pulumi", "policy", "rm", fmt.Sprintf("%s/%s", orgName, policyPackName), "all")
+}
+
+func TestPolicyVersion0_2_0(t *testing.T) {


Lets change naming here too

ekrengel · 2020-03-24T17:56:04Z

pkg/cmd/policy_enable.go

 		}),
 	}

 	cmd.PersistentFlags().StringVar(
 		&args.policyGroup, "policy-group", "",
 		"The Policy Group for which the Policy Pack will be enabled; if not specified, the default Policy Group is used")

+	cmd.PersistentFlags().StringVar(
+		&args.config, "config", "",
+		"The file path for the policy pack configuration file")


Policy Pack

ekrengel · 2020-03-24T17:57:22Z

pkg/cmd/policy_enable.go

+	}
+
+	// Convert type map[string]plugin.AnalyzerPolicyConfig to map[string]*json.RawMessage.
+	config := map[string]*json.RawMessage{}


Hmm not sure how this is working, but don't you normally have to use make for a map? 🤔

It is equivalent to initalizing an empty map using make. Only difference is not including the curly braces at the end when using the make function. make(map[string]*json.RawMessage).

Using make seems to be the more consistent pattern for us from a quick glance in our code base. So went and just changed to using make while I was in there. 😃

justinvp

One suggestion about deleting a commented-out line, and aside from Erin's feedback, LGTM.

justinvp · 2020-03-24T18:41:05Z

pkg/cmd/policy_enable.go

+			// Load the configuration from the user-specified JSON file into config object.
+			var config map[string]*json.RawMessage
+			if args.config != "" {
+				// config, err = loadJSONConfigFile(args.configFile)


Suggested change

// config, err = loadJSONConfigFile(args.configFile)

lol... sorry. completely missed that. will make change.

wip

sean1588 force-pushed the sholung/3756-support-policy-configuration branch 4 times, most recently from 07e78cf to d2e3291 Compare March 20, 2020 17:40

sean1588 linked an issue Mar 20, 2020 that may be closed by this pull request

Update Policy gestures to support configuration #3756

Closed

sean1588 removed a link to an issue Mar 20, 2020

Update Policy gestures to support configuration #3756

Closed

sean1588 force-pushed the sholung/3756-support-policy-configuration branch 9 times, most recently from 026f4b2 to 5a5a90f Compare March 23, 2020 17:25

sean1588 requested review from justinvp and ekrengel March 23, 2020 17:35

ekrengel reviewed Mar 23, 2020

View reviewed changes

justinvp reviewed Mar 23, 2020

View reviewed changes

ekrengel approved these changes Mar 23, 2020

View reviewed changes

justinvp reviewed Mar 23, 2020

View reviewed changes

sean1588 force-pushed the sholung/3756-support-policy-configuration branch 2 times, most recently from 329475c to 9cd2504 Compare March 23, 2020 21:37

justinvp reviewed Mar 23, 2020

View reviewed changes

sean1588 force-pushed the sholung/3756-support-policy-configuration branch 5 times, most recently from aeb3c92 to ac8fa40 Compare March 24, 2020 16:07

sean1588 requested review from justinvp and ekrengel March 24, 2020 17:15

sean1588 force-pushed the sholung/3756-support-policy-configuration branch from 9fdf93c to 8daef9b Compare March 24, 2020 17:17

ekrengel reviewed Mar 24, 2020

View reviewed changes

ekrengel approved these changes Mar 24, 2020

View reviewed changes

justinvp approved these changes Mar 24, 2020

View reviewed changes

sean1588 added 9 commits March 24, 2020 11:47

Support policy pack config using policy enable cmd.

7b47916

Add integration tests.

9e9152c

Make policy versions configurable for test

4f22647

Use loadPolicyConfigFromFile from resource analyzer.

e1f535a

convert the analyzerPolicyConfigFormat

7c5adf3

rename policy directory for verion

1a810b1

Add test for compatibility with older version of sdk

9773676

update changelog

bd3f10a

wip

PR feedback, cleaning up addressing some verbiage

2677fbc

sean1588 force-pushed the sholung/3756-support-policy-configuration branch from 8daef9b to 2677fbc Compare March 24, 2020 18:51

sean1588 merged commit d0f5e35 into master Mar 24, 2020

pulumi-bot deleted the sholung/3756-support-policy-configuration branch March 24, 2020 20:30

sean1588 linked an issue Mar 25, 2020 that may be closed by this pull request

Update Policy gestures to support configuration #3756

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Policy pack configuration using policy enable cmd #4127

Policy pack configuration using policy enable cmd #4127

sean1588 commented Mar 20, 2020 •

edited

Loading

ekrengel Mar 23, 2020

justinvp Mar 23, 2020

sean1588 Mar 23, 2020

ekrengel left a comment

ekrengel Mar 23, 2020

sean1588 Mar 24, 2020

ekrengel Mar 23, 2020

ekrengel Mar 23, 2020

justinvp Mar 23, 2020

ekrengel Mar 23, 2020

sean1588 Mar 23, 2020

justinvp Mar 23, 2020

justinvp Mar 23, 2020

justinvp Mar 23, 2020

justinvp Mar 23, 2020

justinvp Mar 23, 2020

justinvp Mar 23, 2020

justinvp Mar 23, 2020

justinvp Mar 23, 2020 •

edited

Loading

sean1588 Mar 24, 2020

sean1588 commented Mar 24, 2020

ekrengel Mar 24, 2020

sean1588 Mar 24, 2020

ekrengel Mar 24, 2020

ekrengel Mar 24, 2020

ekrengel Mar 24, 2020

ekrengel Mar 24, 2020

sean1588 Mar 24, 2020

sean1588 Mar 24, 2020

justinvp left a comment

justinvp Mar 24, 2020

sean1588 Mar 24, 2020

	func LoadPolicyPackConfigFromFile(file string) (map[string]plugin.AnalyzerPolicyConfig, error) {
	b, err := ioutil.ReadFile(file)
	if err != nil {
	return nil, err
	}
	return parsePolicyPackConfig(b)
	}

	properties := map[string]*json.RawMessage{}
	for k, v := range schema.Properties {
	bytes, err := json.Marshal(v)
	if err != nil {
	return nil, err
	}
	raw := json.RawMessage(bytes)
	properties[k] = &raw
	}

	"enforcementLevel":"mandatory",
	"enforcementLevel": "mandatory",

	"message":"this message is invalid because it is way more than 10 characters......"
	"message": "this message is invalid because it is way more than 10 characters......"

Policy pack configuration using policy enable cmd #4127

Policy pack configuration using policy enable cmd #4127

Conversation

sean1588 commented Mar 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ekrengel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justinvp Mar 23, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sean1588 commented Mar 24, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justinvp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sean1588 commented Mar 20, 2020 •

edited

Loading

justinvp Mar 23, 2020 •

edited

Loading