Question: how to handle user input parameters (RawExtension) in custom resource

[hongchaodeng]

Hi, I'm working on the KubeVela project and trying to use Kustomize to handle multi-environment application deployment. We have a CRD that looks like below:

apiVersion: core.oam.dev/v1alpha2
kind: Application
metadata:
  name: website
spec:
  components:
    - name: backend
      type: worker
      settings:
        image: busybox
        cmd:
          - sleep
          - '1000'
    - name: frontend
      type: webservice
      settings:
        image: nginx
      traits:
        - name: autoscaler
          properties:
            min: 1
            max: 10
        - name: sidecar
          properties:
            name: "sidecar-test"
            image: "fluentd"

I'm super excited about the upcoming support for custom resource via OpenAPI schema. I have been hoping that once applying openapi schema we can use Kustomize native patchesStrategicMerge over our Application resources too.

I give it a try -- here is main.go and schema.json: https://gist.github.com/hongchaodeng/2a3fe0afcb7ab2e116d2533e407ce151

But something unexpected happens. The fields settings and properties are unstructured user input and of type runtime.RawExtension. The schema couldn't be know ahead of time and in openapi schema is described as object type:

"settings": {
  "type": "object",
  "x-kubernetes-preserve-unknown-fields": true
}

The problem is when applying patches it overrides instead of merging fields. Let me give an example below.

Here's the base:

apiVersion: core.oam.dev/v1alpha2
kind: Application
metadata:
  name: example
spec:
  components:
    - name: backend
      type: worker
      settings:
        change: change
        keep: keep

Here's the patch:

apiVersion: core.oam.dev/v1alpha2
kind: Application
metadata:
  name: example
spec:
  components:
    - name: backend
      settings:
        change: done
        new: new

Here will be the final output:

apiVersion: core.oam.dev/v1alpha2
kind: Application
metadata:
  name: example
spec:
  components:
    - name: backend
      settings:
        change: done
        new: new

As shown above, the keep field is gone since the entire settings is overriden.

Would appreciate any solutions for this problem. Thanks!

[Shell32-Natsu]

@natasha41575 Can you take a look at this? Looks like it's OpenAPI related.

[natasha41575]

Is the settings field completely unstructured? Are users allowed to input any keys and values they choose?

[natasha41575]

For the OpenAPI schema to merge correctly, it needs two variables in the OpenAPI schema that you would need to add manually (x-kubernetes-patch-merge-key and x-kubernetes-patch-strategy). E.g. for PodSpec containers, it looks like this:

       "containers": {
          "description": "List of containers belonging to the pod. Containers cannot currently be added or removed. There must be at least one container in a Pod. Cannot be updated.",
          "items": {
            "$ref": "#/definitions/io.k8s.api.core.v1.Container"
          },
          "type": "array",
          "x-kubernetes-patch-merge-key": "name",
          "x-kubernetes-patch-strategy": "merge"
        }

For your case, the OpenAPI for settings would need to look something like:

              "settings": {
                "type": "object",
                "x-kubernetes-preserve-unknown-fields": true,
                "x-kubernetes-patch-merge-key": <desired merge key here>,
                "x-kubernetes-patch-strategy": "merge"
              },

If you know the desired merge key then you can put it in your OpenAPI specification. (Side note: working on documentation to make this more clear.)

You may have to ask the user to specify their desired merge key.

[hongchaodeng]

Is the settings field completely unstructured? Are users allowed to input any keys and values they choose?

Right. Entirely unstructured.

For your case, the OpenAPI for settings would need to look something like: ... If you know the desired merge key then you can put it in your OpenAPI specification. (Side note: working on documentation to make this more clear.)

The example you gave looks awesome!
Just one question, since it is an object not an array, how could it have any merge key?
For instance, in the top example, we want to add the new field, merge the change field, but keep the keep field.

[natasha41575]

I did some investigation and I've discovered your problem. You also need to add the x-kubernetes-group-version-kind extension to your OpenAPI schema. This is unrelated to the merge keys.

I used the following schema here and it worked for your use case: https://pastebin.com/L3igtp78. Notice at the very bottom I added a x-kubernetes-group-version-kind field. This is necessary for kustomize to be able to properly understand your schema.

I am closing this issue, but feel free to reopen if for some reason this doesn't work for you or you have more questions.