tools: add policy generation tool #8248

danmihai1 · 2023-10-18T21:23:17Z

Add application that infers K8s user's intentions based on user's K8s YAML file, and generates a Rego/OPA based policy for that YAML.

Fixes: #7673

danmihai1 · 2023-10-19T15:24:55Z

/test

danmihai1 · 2023-10-20T18:39:42Z

/test

stevenhorsman

Hey Dan, thanks so much for the submission. There is a lot of code here and I'm not an expert, so I've focussed on the documentation and through to get the tool build and running myself primarily, and left some comments about how we might be able to make this a bit more accessible as there were a few hurdles I bumped into and the tool has a lot of power, so letting people understand the processing and output more feels important. Thanks!

src/tools/genpolicy/README.md

danmihai1 · 2023-11-14T01:57:19Z

Hey Dan, thanks so much for the submission. There is a lot of code here and I'm not an expert, so I've focussed on the documentation and through to get the tool build and running myself primarily, and left some comments about how we might be able to make this a bit more accessible as there were a few hurdles I bumped into and the tool has a lot of power, so letting people understand the processing and output more feels important. Thanks!

Thanks for all the feedback @stevenhorsman. I have just pushed what I think are significant doc improvements in 2257a16

stevenhorsman

Thanks very much for the extra documentation Dan - it really helps. Again the code is too much for me to do a sensible review, but I've kicked the tyres with testing, so I think it's fine to go in as the initial drop of some tooling.

katexochen

I'm also if there are any requirements for the container images used. Not sure if that's written anywhere in the documentation and I've missed it. I've tested a bunch of images, some worked, some failed:

apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  containers:
    - name: test
      image: ${image}

image="ubuntu:latest"

Error: failed to generate policies: failed to generate policy for deployment/test.yml: genpolicy failed with exit code 101: thread 'main' panicked at src/registry.rs:111:17:
Failed to pull container image manifest and config - error: UnsupportedMediaTypeError(
    "application/vnd.oci.image.index.v1+json",
)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

image="nixery.dev/shell/socat"

Error: failed to generate policies: failed to generate policy for deployment/test.yml: genpolicy failed with exit code 101: thread 'main' panicked at src/registry.rs:91:61:
called `Result::unwrap()` on an `Err` value: Error("missing field `Env`", line: 1, column: 969)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

katexochen · 2023-12-01T18:35:10Z

src/tools/genpolicy/genpolicy-advanced-command-line-parameters.md

+
+`genpolicy` downloads the `quay.io/prometheus/busybox:latest` container image. 
+
+Depending on the size of the container images and the speed of the network connection to the container registry, downloading these images might take several minutes. For testing scenarios where `genpolicy` gets executed several times, it can be useful to cache the container images after downloading them, in order to avoid most of the time needed to download the same container images multiple times. If a container image layer was already cached locally, `genpolicy` uses the local copy of that container layer. The application caches the image information under the `./layers_cache` directory.


It would be great to have a flag for specifying the cache directory.

I agree that caching needs future improvements, but I don't think the initial merge of this code should wait for these improvements to become available. Please create github issues for such ideas for improvements.

katexochen · 2023-12-01T18:40:15Z

src/tools/genpolicy/genpolicy-advanced-command-line-parameters.md

+
+# Use a custom path to `genpolicy` input files
+
+By default, the `genpolicy` input files [`rules.rego`](rules.rego) and [`genpolicy-settings.json`](genpolicy-settings.json) must be present in the current directory - otherwise `genpolicy` returns an error. Users can specify a different path to these two files, using the `-i` command line parameter - e.g.,


I find the semantics of -i quite counter-intuitive. As a user, I would like to place these files independently from each other in the place I like. Would it be possible to implement this as -r flag instead to directly point to the rego policy?

Please log an issue for the future.

katexochen · 2023-12-01T18:50:23Z

src/tools/genpolicy/src/policy.rs

+        let mut resources = Vec::new();
+        let yaml_contents = yaml::get_input_yaml(&config.yaml_file)?;
+
+        for document in serde_yaml::Deserializer::from_str(&yaml_contents) {


Deployments can contain a mix of confidential and non-confidential resources. Parts that should deployed into confidential environment have a specific runtimeClassName. When testing these changes, the tool would just generate policies for all resources in the yaml files referenced, not respecting the (not) specified runtime class. I was wondering how we could solve this. I think we should either check for a set of well-known runtimeClassName values or let the user specify the runtimeClassName to target.

@katexochen , thanks for the feedback! This tool doesn't solve all the Confidential K8s Policy problems (yet?). For example, if you search for TODO in its source code, you'll find a bunch of other problem that were not solved yet. I think we should start somewhere and keep improving. After working on this for several months, I think that waiting for the ideal tool before the first merge is impractical.

danmihai1 · 2023-12-04T23:23:29Z

I'm also if there are any requirements for the container images used. Not sure if that's written anywhere in the documentation and I've missed it. I've tested a bunch of images, some worked, some failed:

apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  containers:
    - name: test
      image: ${image}

image="ubuntu:latest"

Error: failed to generate policies: failed to generate policy for deployment/test.yml: genpolicy failed with exit code 101: thread 'main' panicked at src/registry.rs:111:17:
Failed to pull container image manifest and config - error: UnsupportedMediaTypeError(
    "application/vnd.oci.image.index.v1+json",
)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

image="nixery.dev/shell/socat"

Error: failed to generate policies: failed to generate policy for deployment/test.yml: genpolicy failed with exit code 101: thread 'main' panicked at src/registry.rs:91:61:
called `Result::unwrap()` on an `Err` value: Error("missing field `Env`", line: 1, column: 969)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Please log github issues for YAML files that are not working and attach the problematic YAML files to those issues. For these particular two problems:

One of my colleagues at MSFT noticed the ubuntu:latest problem a few days ago, and we are working on a solution for it. I can share the gory details in case you are curious about the root cause. Meanwhile, creating your own container image on top of an Ubuntu base container image (but using a different manifest format) might be a good enough workaround.
Please attach your test.yaml in github - the Env problem you mentioned sounds more related to the YAML contents, not to the container image.

src/tools/genpolicy/Cargo.toml

src/tools/genpolicy/src/policy.rs

danmihai1 · 2023-12-06T01:13:46Z

No code or doc changes in my latest push. Just squashed the doc changes into a single commit and rebased on top of the latest main code.

Add application that infers K8s user's intentions based on user's K8s YAML file, and generates a Rego/OPA based policy for that YAML. Just Pod YAML files are supported as input using this initial source code. Support for other types of YAML files will come with upcoming commits. Fixes: kata-containers#7673 Signed-off-by: Dan Mihai <dmihai@microsoft.com>

Generate policy for K8s DaemonSet YAML. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

Generate policy for K8s Deployment YAML. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

Generate policy for K8s Job YAML. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

Generate policy for K8s List YAML. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

Generate policy for K8s ReplicaSet YAML. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

Generate policy for K8s ReplicationController YAML. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

Generate policy for K8s StatefulSet YAML. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

Add docs for the Agent Policy and for the genpolicy tool. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

Also support alternative media type and update samples Signed-off-by: Saul Paredes <saulparedes@microsoft.com> Signed-off-by: Dan Mihai <dmihai@microsoft.com>

Add SPDX license header to rules.rego. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

huangyingting · 2024-01-02T11:48:08Z

@danmihai1 I want to report an issue while I am testing confidential containers from AKS, there is no place from microsoft/kata-containers to let me open an issue, so I put it here

It seems we not able to correctly handle mountPropagation: Bidirectional or mountPropagation: HostToContainer on emptyDir volume, as a consequence, the container won't be created with error "Error: failed to create containerd task: failed to create shim task: "CreateContainerRequest is blocked by policy": unknown"

To work around the issue, for HostToContainer, we need to replace rprivate to rslave in the generated policy.

      {
        "destination": "/mnt/remote",
        "source": "^$(cpath)/$(sandbox-id)/local/remotemounts$",
        "type_": "local",
        "options": [
          "rbind",
          "rslave",
          "rw"
        ]
      }

For Bidirectional, we need to replace rprivate to rshared in the generated policy.

      {
        "destination": "/mnt/remote",
        "source": "^$(cpath)/$(sandbox-id)/local/remotemounts$",
        "type_": "local",
        "options": [
          "rbind",
          "rshared",
          "rw"
        ]
      }

To reproduce the issue, you can use https://github.com/huangyingting/sev-snp/blob/main/encfs.yaml (ignore io.katacontainers.config.agent.policy, it was tweaked already based on above findings to make it work)

danmihai1 · 2024-01-03T02:23:10Z

@danmihai1 I want to report an issue while I am testing confidential containers from AKS, there is no place from microsoft/kata-containers to let me open an issue, so I put it here

It seems we not able to correctly handle mountPropagation: Bidirectional or mountPropagation: HostToContainer on emptyDir volume, as a consequence, the container won't be created with error "Error: failed to create containerd task: failed to create shim task: "CreateContainerRequest is blocked by policy": unknown"

Thanks! We are tracking other bugs applicable to this app under https://github.com/kata-containers/kata-containers/issues .

fidencio

@danmihai1, lgtm, thanks!

I think this one should get in sooner than later, as we have been tracking the improvements in a different issue.

fidencio · 2024-01-11T12:42:42Z

/test

danmihai1 requested a review from a team as a code owner October 18, 2023 21:23

katacontainersbot added the size/huge Largest and most complex task (probably needs breaking into small pieces) label Oct 18, 2023

danmihai1 added no-forward-port-needed labels Oct 18, 2023

danmihai1 force-pushed the danmihai1/genpolicy-main branch from 3df07fb to 2ccfe7d Compare October 18, 2023 23:03

danmihai1 added the ok-to-test label Oct 18, 2023

danmihai1 mentioned this pull request Oct 19, 2023

How are we implementing policies in peerpods? confidential-containers/cloud-api-adaptor#1369

Open

danmihai1 force-pushed the danmihai1/genpolicy-main branch from 2ccfe7d to c5582b6 Compare October 20, 2023 14:49

danmihai1 requested review from fidencio and stevenhorsman October 20, 2023 20:05

stevenhorsman reviewed Nov 6, 2023

View reviewed changes

src/tools/genpolicy/README.md Outdated Show resolved Hide resolved

src/tools/genpolicy/README.md Show resolved Hide resolved

src/tools/genpolicy/README.md Outdated Show resolved Hide resolved

danmihai1 mentioned this pull request Nov 9, 2023

docs: add agent policy documentation #8406

Merged

danmihai1 force-pushed the danmihai1/genpolicy-main branch from c5582b6 to 2257a16 Compare November 14, 2023 01:53

stevenhorsman approved these changes Nov 14, 2023

View reviewed changes

danmihai1 mentioned this pull request Nov 20, 2023

[RFC] Initdata Specification for Confidential Containers confidential-containers/confidential-containers#171

Closed

danmihai1 requested a review from bpradipt November 30, 2023 14:17

katexochen suggested changes Dec 4, 2023

View reviewed changes

ChengyuZhu6 reviewed Dec 5, 2023

View reviewed changes

src/tools/genpolicy/Cargo.toml Outdated Show resolved Hide resolved

ChengyuZhu6 reviewed Dec 5, 2023

View reviewed changes

src/tools/genpolicy/src/policy.rs Show resolved Hide resolved

danmihai1 force-pushed the danmihai1/genpolicy-main branch from 0b3cb31 to 4e2fce2 Compare December 6, 2023 01:11

danmihai1 force-pushed the danmihai1/genpolicy-main branch from 4e2fce2 to 4197d7b Compare December 21, 2023 00:04

danmihai1 and others added 11 commits December 22, 2023 15:35

genpolicy: add support for DaemonSet YAML input

a40a600

Generate policy for K8s DaemonSet YAML. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

genpolicy: add support for Deployment YAML input

2dbd01c

Generate policy for K8s Deployment YAML. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

genpolicy: add support for Job YAML input

a034526

Generate policy for K8s Job YAML. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

genpolicy: add support for List YAML input

d84300f

Generate policy for K8s List YAML. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

genpolicy: add support for ReplicaSet YAML input

7da1709

Generate policy for K8s ReplicaSet YAML. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

genpolicy: add support for ReplicationController

35958ec

Generate policy for K8s ReplicationController YAML. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

genpolicy: add support for StatefulSet YAML input

57f9319

Generate policy for K8s StatefulSet YAML. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

docs: add agent policy and genpolicy docs

9eb6fd4

Add docs for the Agent Policy and for the genpolicy tool. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

genpolicy: Update oci_distribution to 0.10.0

7f126be

Also support alternative media type and update samples Signed-off-by: Saul Paredes <saulparedes@microsoft.com> Signed-off-by: Dan Mihai <dmihai@microsoft.com>

genpolicy: add SPDX license header

080541a

Add SPDX license header to rules.rego. Signed-off-by: Dan Mihai <dmihai@microsoft.com>

danmihai1 force-pushed the danmihai1/genpolicy-main branch from 6568389 to 080541a Compare December 22, 2023 15:35

fidencio approved these changes Jan 11, 2024

View reviewed changes

katexochen approved these changes Jan 11, 2024

View reviewed changes

fidencio merged commit 86a6d13 into kata-containers:main Jan 11, 2024
168 of 175 checks passed

danmihai1 deleted the danmihai1/genpolicy-main branch January 21, 2024 16:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tools: add policy generation tool #8248

tools: add policy generation tool #8248

danmihai1 commented Oct 18, 2023

danmihai1 commented Oct 19, 2023

danmihai1 commented Oct 20, 2023

stevenhorsman left a comment

danmihai1 commented Nov 14, 2023 •

edited

stevenhorsman left a comment

katexochen left a comment •

edited

katexochen Dec 1, 2023

danmihai1 Dec 4, 2023

katexochen Dec 1, 2023

danmihai1 Dec 4, 2023

katexochen Dec 1, 2023

danmihai1 Dec 4, 2023

danmihai1 commented Dec 4, 2023

danmihai1 commented Dec 6, 2023

huangyingting commented Jan 2, 2024

danmihai1 commented Jan 3, 2024

fidencio left a comment

fidencio commented Jan 11, 2024


		`genpolicy` downloads the `quay.io/prometheus/busybox:latest` container image.

		Depending on the size of the container images and the speed of the network connection to the container registry, downloading these images might take several minutes. For testing scenarios where `genpolicy` gets executed several times, it can be useful to cache the container images after downloading them, in order to avoid most of the time needed to download the same container images multiple times. If a container image layer was already cached locally, `genpolicy` uses the local copy of that container layer. The application caches the image information under the `./layers_cache` directory.


		# Use a custom path to `genpolicy` input files

		By default, the `genpolicy` input files [`rules.rego`](rules.rego) and [`genpolicy-settings.json`](genpolicy-settings.json) must be present in the current directory - otherwise `genpolicy` returns an error. Users can specify a different path to these two files, using the `-i` command line parameter - e.g.,

tools: add policy generation tool #8248

tools: add policy generation tool #8248

Conversation

danmihai1 commented Oct 18, 2023

danmihai1 commented Oct 19, 2023

danmihai1 commented Oct 20, 2023

stevenhorsman left a comment

Choose a reason for hiding this comment

danmihai1 commented Nov 14, 2023 • edited

stevenhorsman left a comment

Choose a reason for hiding this comment

katexochen left a comment • edited

Choose a reason for hiding this comment

katexochen Dec 1, 2023

Choose a reason for hiding this comment

danmihai1 Dec 4, 2023

Choose a reason for hiding this comment

katexochen Dec 1, 2023

Choose a reason for hiding this comment

danmihai1 Dec 4, 2023

Choose a reason for hiding this comment

katexochen Dec 1, 2023

Choose a reason for hiding this comment

danmihai1 Dec 4, 2023

Choose a reason for hiding this comment

danmihai1 commented Dec 4, 2023

danmihai1 commented Dec 6, 2023

huangyingting commented Jan 2, 2024

danmihai1 commented Jan 3, 2024

fidencio left a comment

Choose a reason for hiding this comment

fidencio commented Jan 11, 2024

danmihai1 commented Nov 14, 2023 •

edited

katexochen left a comment •

edited