Kustomize adds namespace and labels to Lists causing errors

[chrisghill]

Its entirely possible that this is user-error, but Kustomize is adding metadata.namespace and metadata.label to ConfigMapList, RoleBindingList, RoleList and List (and perhaps others) which is breaking kubectl apply. I saw this issue: #514 which seems related, and it sounds like the PR introducing mandatory namespaces was reverted, but the issue still arises with lists. I've tried version 1.0.6, 1.0.10 and 1.0.11 and it fails with all 3 (though slightly differently).

Here is an example:

apiVersion: rbac.authorization.k8s.io/v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    name: prometheus-k8s
    namespace: default
  rules:
  - apiGroups:
    - ""
    resources:
    - nodes
    - services
    - endpoints
    - pods
    verbs:
    - get
    - list
    - watch
- apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    name: prometheus-k8s
    namespace: kube-system
  rules:
  - apiGroups:
    - ""
    resources:
    - nodes
    - services
    - endpoints
    - pods
    verbs:
    - get
    - list
    - watch
- apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    name: prometheus-k8s
    namespace: monitoring
  rules:
  - apiGroups:
    - ""
    resources:
    - nodes
    - services
    - endpoints
    - pods
    verbs:
    - get
    - list
    - watch
kind: RoleList

In versions 1.0.6 and 1.0.10 kustomize build works, but here is an excerpt of the output where it adds the metadata.namespace and metadata.label:

...
    - get
    - list
    - watch
kind: RoleList
metadata:
  labels: {}
  namespace: monitoring

Then when piped to kubectl, it complains:
error: error validating "STDIN": error validating data: [ValidationError(RoleList.metadata): unknown field "labels" in io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta, ValidationError(RoleList.metadata): unknown field "namespace" in io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta]; if you choose to ignore these errors, turn validation off with --validate=false

Kustomize 1.0.11 seems to make matters worse by requiring a metadata.name field, then kubectl complains about that field as well. Kustomize build won't even work without a metadata.name filed on the RoleList:

Error: loadResMapFromBasesAndResources: SemiResources: loadResMapFromBasesAndResources: rawResources failed to read Resources: Missing metadata.name in object {map[apiVersion:rbac.authorization.k8s.io/v1 items:[map[apiVersion:rbac.authorization.k8s.io/v1 kind:Role metadata:map[name:prometheus-k8s namespace:default] rules:[map[apiGroups:[] resources:[nodes services endpoints pods] verbs:[get list watch]]]] map[apiVersion:rbac.authorization.k8s.io/v1 kind:Role metadata:map[name:prometheus-k8s namespace:kube-system] rules:[map[apiGroups:[] resources:[nodes services endpoints pods] verbs:[get list watch]]]] map[kind:Role metadata:map[name:prometheus-k8s namespace:monitoring] rules:[map[apiGroups:[] resources:[nodes services endpoints pods] verbs:[get list watch]]] apiVersion:rbac.authorization.k8s.io/v1]] kind:RoleList]}

So, I add the metadata.name field to the RoleList like so:

...
    - list
    - watch
kind: RoleList
metadata:
  name: role-list-name

And Kustomize build is happy, but kubectl apply isn't:
error: error validating "STDIN": error validating data: [ValidationError(RoleList.metadata): unknown field "labels" in io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta, ValidationError(RoleList.metadata): unknown field "name" in io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta, ValidationError(RoleList.metadata): unknown field "namespace" in io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta]; if you choose to ignore these errors, turn validation off with --validate=false

kubectl is complaining about the metadata.name field that Kustomize forced me to add, as well as the metadata.namespace and metadata.label field that it added automatically.

Perhaps its version mismatches between kubectl and Kustomize? As I mentioned I tried this with Kustomize 1.0.6, 1.0.10 and 1.0.11. My K8S cluster is 1.11.0 and my kubectl is also 1.11.0. Are there any documents specifying which Kustomize to use based on which kubectl/k8s version you are using?

[Liujingfang1]

@chrisghill First of all, this is not a user error. The problem is that Kustomize doesn't recognize RoleList as an aggregation of objects. It treats RoleList as a single object. The expected behavior should be that Kustomize build returns the role objects contained in the list.

The problem is in loading resources part in Kustomize. https://github.com/kubernetes-sigs/kustomize/blob/master/pkg/resource/factory.go#L85
We need to figure out a way to load resources from different list kinds: List, RoleList and so on.

[Liujingfang1]

@hyww Do you want to help with this issue?

[hyww]

I think I can take a look.

Latest master version (with #637 merged) of Kustomize recognizes List kind and flatten objects with List type. items in List of resources/patches will be treated as individual resources/patches and there will be no List in output of kustomize build.

I am not sure if other lists like ConfigMapList, RoleBindingList, RoleList should be handled as how Kustomize handles List. This behavior makes Kustomize handle items in *List easily but properties of *List, which I have no idea what they are, might be lost.

[hyww]

all lists that return objects with labels should support label filtering (see the labels documentation), and most lists should support filtering by fields.
https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#types-kinds

continue may be set if the user set a limit on the number of items returned, and indicates that the server has more data available
https://kubernetes.io/docs/reference/federation/v1/definitions/#_v1_listmeta

It seems that special properties of *List are used while communicating with server and maybe Kustomize can just ignore them.

Also, Kustomize might be able to validate *List with the following rules.

The name of a list kind must end with "List". Lists have a limited set of common metadata. All lists use the required "items" field to contain the array of objects they return. Any kind that has the "items" field must be a list kind.
https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#types-kinds

[Liujingfang1]

@hyww Sorry I missed your comments.

I am not sure if other lists like ConfigMapList, RoleBindingList, RoleList should be handled as how Kustomize handles List.

Yes, we need to handle all List types similar to List. When a List type object is given, Kustomize build need to load objects from items in it.

[laverya]

Is there any functional difference between a ConfigMapList and a list composed entirely of kind: ConfigMap objects? (Are you allowed to leave off the apiVersion/kind of the list items?) The "in-the-wild" examples of these that I've seen (RoleList, RoleBindingList, ConfigMapList) all specify the apiVersion and kind for each item, and it's tempting to change

kustomize/pkg/resource/factory.go

Line 97 in 02d7530

if u.GetKind() == "List" {

to

if strings.HasSuffix(u.GetKind(), "List") {

and be done with it.