kubectl apply does not update ThirdPartyResource object.

[ianlewis]

I have a ThirdPartyResource object. The resource is as follows:

resource.yaml

metadata:
  name: cron-tab.stable.example.com
apiVersion: extensions/v1beta1
kind: ThirdPartyResource
description: "A specification of a Pod to run on a cron style schedule"
versions:
  - name: v1
  - name: v2

crontab.yaml

apiVersion: "stable.example.com/v1"
kind: "CronTab"
metadata:
  name: hello-world
cronSpec: "*/1 * * * *"
image: hello-world

I create the resource and object

$ kubectl create -f resource.yaml
$ kubectl create -f crontab.yaml

However, when I run apply I fails to update the properties.

$ cat crontab.yaml 
apiVersion: "stable.example.com/v1"
kind: "CronTab"
metadata:
  name: hello-world
cronSpec: "*/5 * * * *"
image: hello-world

$ kubectl apply -f crontab.yaml
crontab "hello-world" configured

kubectl is sending a request to the API and if I watch the object I get a modified event but it is still the old value. Getting the object also returns the old value.

$ kubectl get crontab hello-world -o json | jq '.cronSpec'
"*/1 * * * *"

I suspect that kubectl is not diffing the object properly and neglects to send the updated object JSON data.

[adohe-zz]

@ianlewis what's your kubernetes version? kubectl version can give you this info.

[ianlewis]

Here ya go. Didn't realize the client wasn't 1.3.3 yet.

Client was pulled from https://storage.googleapis.com/kubernetes-release/release/v1.3.0/bin/darwin/amd64/kubectl

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"clean", BuildDate:"2016-07-01T19:26:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.2", GitCommit:"9bafa3400a77c14ee50782bb05f9efc5c91b3185", GitTreeState:"clean", BuildDate:"2016-07-17T18:23:58Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

[xiang90]

We just met the exact same issue. It does not work for current master either.

@pwittrock Should this be considered as a bug? Why this is a p2 issue?

[pwittrock]

@xiang90 Sounds like a bug. Re: the priority, it was a rough estimate. If you think things need a higher priority and attention, ping the thread and I will give it more attention. If you need this fixed immediately let me know and I will shuffle things around.

[xiang90]

@pwittrock Thank you!

We can workaround this by using cURL. But it is not ideal. :)

[pwittrock]

I plan to have this fixed in 1.5. Do you think that is sufficient?

[xiang90]

sure. thanks.

[pwittrock]

@adohe Do you have time to look into this?

[adohe-zz]

ok, will fix this asap.

[pwittrock]

@adohe Any updates?

[adohe-zz]

@ianlewis actually in the final patch, we have include the configuration change, something like below:

map[string]interface {}{"metadata":map[string]interface {}{"creationTimestamp":interface {}(nil), "annotations":map[string]interface {}{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"extensions/v1beta1\",\"metadata\":{\"name\":\"hello-world\",\"namespace\":\"default\",\"creationTimestamp\":null},\"data\":\"eyJhcGlWZXJzaW9uIjoic3RhYmxlLmV4YW1wbGUuY29tL3YxIiwiY3JvblNwZWMiOiIqLzUgKiAqICogKiIsImltYWdlIjoiaGVsbG8td29ybGQiLCJraW5kIjoiQ3JvblRhYiIsIm1ldGFkYXRhIjp7Im5hbWUiOiJoZWxsby13b3JsZCJ9fQ==\"}"}}, "data":"eyJhcGlWZXJzaW9uIjoic3RhYmxlLmV4YW1wbGUuY29tL3YxIiwiY3JvblNwZWMiOiIqLzUgKiAqICogKiIsImltYWdlIjoiaGVsbG8td29ybGQiLCJraW5kIjoiQ3JvblRhYiIsIm1ldGFkYXRhIjp7Im5hbWUiOiJoZWxsby13b3JsZCJ9fQ=="}

the reason why the cronSpec keeps the same is that in the server side when we apply this patch, we use strategic merge, this will just update the original configuration if the merge key exists in both maps, so it will still use the cronSpec in original document.

[adohe-zz]

@brendandburns could you please explain the data field in third party resource data? how we generate it and how to use it? I am taking a deep look at third party resource now.

[ianlewis]

@adohe The data field is basically arbitrary JSON that the user can specify per object.

[brendandburns]

@adohe the data field is the arbitrary JSON representation of the third party object.

We need to store the object as a runtime.Object inside the internals of the system, but of course the third party object doesn't exist as a golang object, so we use ThirdPartyResourceData and store the contents of the third party object in the data field.

[adohe-zz]

@brendandburns that's the problem, when we apply a patch, we just update the data field of ThirdPartyResourceData

[brendandburns]

@adohe are you working on this? If not I'll take it back over...

[ConnorDoyle]

There are a couple libraries in go to deal with the JSON-patch spec, here's one that can produce merge patches based on two arbitrary JSONs and also apply those patches: https://github.com/evanphx/json-patch

[adohe-zz]

@ConnorDoyle thanks for reference this, actually in kubernetes, we not only support standard JSON merge patch, but also support strategic merge patch, which is a custom implementation of merge patch. I have fixed this on my local, and will open a PR this weekend after I test it.

[foxish]

@adohe Any updates on this one? or has this been punted to 1.6?

[adohe-zz]

@foxish yes, I think we can put this on 1.6, we are going to remove ThirdPartyResourceData

[xiang90]

@foxish @adohe

This is going to affect anyone using TPR (we develop etcd operator for example) pretty hard. This is a bug, and I feel we should fix this in 1.5.

/cc @brendandburns

[foxish]

@xiang90 I agree, but I don't think we have enough bandwidth to fix this properly in 1.5.
We have a similar use case, and we're going to use PATCH, which is a little more convenient than the "PUT" that the etcd-operator example uses in documentation.

[tatsuhiro-t]

@foxish I did kubectl patch but it complains like this:

the provided version "example.com/v1alpha1" has no relevant versions: group example.com has not been registered

Could you tell us how to make this work?

I use kubectl v1.5.0-beta.2

[foxish]

@tatsuhiro-t, see my comment here foxish/spark#3 (comment)

[tatsuhiro-t]

@foxish thank you!

[hjacobs]

@xiang90 what's the workaround with cURL?

[JensRantil]

@hjacobs https://github.com/coreos/etcd-operator/blob/master/README.md#resize-an-etcd-cluster has an example of the workaround.

[paultiplady]

Note that this issue is breaking Helm for TPRs; see helm/helm#1468.

[pwittrock]

@adohe Are there any updates on the status of this issue? Would it help to have @ymqytw look into it as well?

[liggitt]

Isn't the core issue apply requiring a go struct to use as a schema?

[pwittrock]

@liggitt Yeah, the struct is used for the tags determining how to merge list fields. Is it used for anything else? WDYT of simply choosing defaults for the values we would get from the structs for TPRs? e.g. always assume replace strategy for lists in TPRs.

[liggitt]

WDYT of simply choosing defaults for the values we would get from the structs for TPRs? e.g. always assume replace strategy for lists in TPRs.

I wouldn't want to assume unknown types are TPRs, but having a default apply mode for types where we can't use the normal go struct logic (which could be TPRs or types from add-on API servers) makes sense to me. The same approach could work for fields of type Interface or runtime.RawExtension that currently don't support apply either

[liggitt]

runtime.RawExtension that currently don't support apply either

c.f. #14998

[liggitt]

fixed in #40666