Make kubernetes json serializer case sensitive #65034

caesarxuchao · 2018-06-12T22:43:56Z

This PR imported the latest jsoniterator library so that case sensitivity during unmarhsaling is optional. The PR also set Kubernetes json serializer to be case sensitive.

Kubernetes json serializer had been case sensitive for 1.1-1.7 as we were using ugorji. This PR restores the behavior.

Fix #64612.

Kubernetes json deserializer is now case-sensitive to restore compatibility with pre-1.8 servers.
If your config files contain fields with wrong case, the config files will be now invalid.

Action required: 
Fix your JSON config files that contain fields of wrong case, otherwise they will be invalid.
If the fields in the golang schema doesn't have json tags, e.g., missing `json:"podSpec,omitempty"`, then the field in the JSON config file should match the case of the golang field, e.g., CamelCase.

Known non camelCase fields:

v1.10.5, scheduler/v1#ExtenderConfig.BindVerb, in the JSON config file, it needs to be "BindVerb".

nikhita · 2018-06-13T03:06:30Z

I think some tests need fixing because of the ExcludeCIDRs change. And staging godeps needs to be fixed. But looks good otherwise. 👍

Should this be cherry-picked into 1.8+? I'm not sure if that is such a good idea since this does a dependency bump...

nikhita · 2018-06-13T03:08:14Z

/kind bug
/priority critical-urgent

Needs a milestone update too.

nikhita · 2018-06-13T03:14:37Z

cc @cblecker @sttts for godeps bump/approval

nikhita · 2018-06-13T03:16:15Z

/uncc @dchen1107

liggitt · 2018-06-13T04:47:59Z

staging/src/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go

@@ -157,7 +169,7 @@ func (s *Serializer) Decode(originalData []byte, gvk *schema.GroupVersionKind, i
 		types, _, err := s.typer.ObjectKinds(into)
 		switch {
 		case runtime.IsNotRegisteredError(err), isUnstructured:
-			if err := jsoniter.ConfigCompatibleWithStandardLibrary.Unmarshal(data, into); err != nil {
+			if err := caseSensitiveJsonIterator().Unmarshal(data, into); err != nil {


private package var rather than reconstructing the options every time?

liggitt · 2018-06-13T04:49:07Z

more godeps need fixing (apiextensions-apiserver/Godeps/Godeps.json)... should find/replace the json-iter revision in all of them

sttts · 2018-06-13T08:28:36Z

staging/src/k8s.io/apimachinery/pkg/runtime/serializer/json/json_test.go

+				Other: "test",
+			},
+		},
+		// Unmarshalling is case-sensitive for big strut.


nit: struct

sttts · 2018-06-13T08:29:17Z

/approve

liggitt · 2018-06-13T19:50:33Z

staging/src/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go

@@ -157,7 +167,7 @@ func (s *Serializer) Decode(originalData []byte, gvk *schema.GroupVersionKind, i
 		types, _, err := s.typer.ObjectKinds(into)
 		switch {
 		case runtime.IsNotRegisteredError(err), isUnstructured:
-			if err := jsoniter.ConfigCompatibleWithStandardLibrary.Unmarshal(data, into); err != nil {


looks like there are four other uses of jsoniter.ConfigCompatibleWithStandardLibrary in the codebase... should replace those as well (and as a follow-up think about how to ensure we only use a standard consistent config; possibly something like #63284)

Replaced the other references in the latest commit.

#63284 is a good follow-up.

caesarxuchao · 2018-06-13T22:17:11Z

/retest

@sttts

Automatic merge from submit-queue (batch tested with PRs 64838, 65138). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. use case-sensitive json-iter to decode custom resource ObjectMeta found one additional spot where case-insensitivity could bite us. uses the case-sensitive decode added in kubernetes/kubernetes#65034 /assign @sttts /cc @caesarxuchao ```release-note NONE ``` Kubernetes-commit: 4ba0dbde84ba54941d998fc93b863b7eb97b8d53

Automatic merge from submit-queue. Manually cherrypick #65034 to 1.10 Manually cherrypicking #65034. Using hack/cherry_pick_pull.sh to cherrypick is difficult because that requires cherrypicking #63059 first. This PR imported the latest jsoniterator library so that case sensitivity during unmarhsaling is optional. The PR also set Kubernetes json serializer to be case sensitive. Fix #64612. ```release-notes Kubernetes json deserializer is now case-sensitive to restore compatibility with pre-1.8 servers. If your config files contains fields with wrong case, the config files will be now invalid. ```

caesarxuchao · 2018-06-19T17:45:37Z

Cherrypicked to 1.10, still needs to be cherrypicked to 1.8 and 1.9.

Automatic merge from submit-queue (batch tested with PRs 65339, 65343, 65324, 65335, 65367). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Fix k8s json package import name Fixes nits found while cherry-picking #65034 in #65298. **Release note**: ```release-note NONE ``` /cc sttts liggitt caesarxuchao

Automatic merge from submit-queue (batch tested with PRs 65339, 65343, 65324, 65335, 65367). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Fix k8s json package import name Fixes nits found while cherry-picking kubernetes/kubernetes#65034 in kubernetes/kubernetes#65298. **Release note**: ```release-note NONE ``` /cc sttts liggitt caesarxuchao Kubernetes-commit: 456f85e736069398329226b9dff39d0304ee52d2

Automatic merge from submit-queue. Manual cherrypick of #65034 to 1.9: make json serializer case sensitive fixes partially #64612 This PR imports the latest jsoniterator library so that case sensitivity during unmarshalling is optional. The PR also sets Kubernetes json serializer to be case sensitive. **Release note**: ```release-note ACTION REQUIRED: Kubernetes JSON deserializer is now case-sensitive to restore compatibility with pre-1.8 servers. If your config files contains fields with wrong case, the config files will be now invalid. ``` /sig api-machinery /kind bug /assign caesarxuchao liggitt thockin sttts mbohlool

kfox1111 · 2018-06-28T00:38:28Z

Saw this in the release notes. Are there any tools to verify your cluster only has objects with the correct casing, or does each site need to write a tool to do so?

liggitt · 2018-06-28T00:57:37Z

object data persisted in etcd is correct, so no stored data is problematic.

kubectl create/apply performs case-sensitive client-side validation which complains about incorrect/unknown fields.

For your resource manifest files, I'm not aware of a stand-alone tool that only validates a resource manifest against the openapi schema

Automatic merge from submit-queue. Manual cherrypick of #65034 to 1.8: make json serializer case sensitive fixes partially #64612 This PR imports the latest jsoniterator library so that case sensitivity during unmarshalling is optional. The PR also sets Kubernetes json serializer to be case sensitive. **Release note**: ```release-note ACTION REQUIRED: Kubernetes JSON deserializer is now case-sensitive to restore compatibility with pre-1.8 servers. If your config files contains fields with wrong case, the config files will be now invalid. ``` /sig api-machinery /kind bug /assign caesarxuchao liggitt thockin sttts mbohlool

* Bump v1.11.0 * Bump vendor k8s 1.11.0 Also bump json-iterator as required per: kubernetes/kubernetes#65034

Using lessons learned from k8s changes: kubernetes/kubernetes#65034 Change-Id: Ia17a8f94ae6ed00c5af2595c2b48d3c9a0344427

…ugorji Using lessons learned from k8s changes: kubernetes/kubernetes#65034 Change-Id: Ia17a8f94ae6ed00c5af2595c2b48d3c9a0344427

Automatic merge from submit-queue (batch tested with PRs 65339, 65343, 65324, 65335, 65367). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Fix k8s json package import name Fixes nits found while cherry-picking kubernetes/kubernetes#65034 in kubernetes/kubernetes#65298. **Release note**: ```release-note NONE ``` /cc sttts liggitt caesarxuchao Kubernetes-commit: 456f85e736069398329226b9dff39d0304ee52d2

caesarxuchao assigned liggitt and nikhita Jun 12, 2018

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 12, 2018

k8s-ci-robot requested review from dchen1107 and deads2k June 12, 2018 22:44

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 12, 2018

caesarxuchao added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jun 12, 2018

caesarxuchao force-pushed the json-case-sensitive branch from 84be0b0 to c9efab0 Compare June 12, 2018 22:55

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 12, 2018

caesarxuchao force-pushed the json-case-sensitive branch from c9efab0 to f5e4ac6 Compare June 12, 2018 23:09

k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. labels Jun 13, 2018

k8s-ci-robot removed the request for review from dchen1107 June 13, 2018 03:16

liggitt reviewed Jun 13, 2018

View reviewed changes

sttts reviewed Jun 13, 2018

View reviewed changes

caesarxuchao force-pushed the json-case-sensitive branch from 67ff1fe to 0334021 Compare June 13, 2018 19:45

k8s-ci-robot added area/kubeadm sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. labels Jun 13, 2018

liggitt reviewed Jun 13, 2018

View reviewed changes

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 13, 2018

liggitt removed the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jun 15, 2018

caesarxuchao mentioned this pull request Jun 15, 2018

Manually cherrypick #65034 to 1.10 #65157

Merged

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. labels Jun 15, 2018

This was referenced Jun 21, 2018

JSON decoders should be case sensitive #64612

Closed

Fix k8s json package import name #65324

Merged

nikhita mentioned this pull request Jun 24, 2018

Manual cherrypick of #65034 to 1.8: make json serializer case sensitive #65410

Merged

k8s-ci-robot added release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. and removed release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Jun 24, 2018

caesarxuchao mentioned this pull request Jun 24, 2018

Scheduler extender broken by JSON decoder update #65413

Closed

rphillips pushed a commit to kubernetes-retired/bootkube that referenced this pull request Jul 11, 2018

Bump v1.11.0 (#990)

6618198

* Bump v1.11.0 * Bump vendor k8s 1.11.0 Also bump json-iterator as required per: kubernetes/kubernetes#65034

liggitt mentioned this pull request Apr 23, 2019

Use latest etcd from release-3.3 branch for dropping ugorji #76917

Merged

dims added a commit to dims/etcd that referenced this pull request Apr 23, 2019

Switch to case sensitive unmarshalling to be compatible with ugorji

fe3d7eb

Using lessons learned from k8s changes: kubernetes/kubernetes#65034 Change-Id: Ia17a8f94ae6ed00c5af2595c2b48d3c9a0344427

dims added a commit to dims/etcd that referenced this pull request Apr 23, 2019

client: Switch to case sensitive unmarshalling to be compatible with …

4cad27e

…ugorji Using lessons learned from k8s changes: kubernetes/kubernetes#65034 Change-Id: Ia17a8f94ae6ed00c5af2595c2b48d3c9a0344427

dims added a commit to dims/etcd that referenced this pull request Apr 23, 2019

client: Switch to case sensitive unmarshalling to be compatible with …

daee668

…ugorji Using lessons learned from k8s changes: kubernetes/kubernetes#65034 Change-Id: Ia17a8f94ae6ed00c5af2595c2b48d3c9a0344427

dims added a commit to dims/etcd that referenced this pull request Apr 23, 2019

client: Switch to case sensitive unmarshalling to be compatible with …

ada5a9d

…ugorji Using lessons learned from k8s changes: kubernetes/kubernetes#65034 Change-Id: Ia17a8f94ae6ed00c5af2595c2b48d3c9a0344427

dims added a commit to dims/etcd that referenced this pull request Apr 24, 2019

client: Switch to case sensitive unmarshalling to be compatible with …

022be3b

…ugorji Using lessons learned from k8s changes: kubernetes/kubernetes#65034 Change-Id: Ia17a8f94ae6ed00c5af2595c2b48d3c9a0344427

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make kubernetes json serializer case sensitive #65034

Make kubernetes json serializer case sensitive #65034

caesarxuchao commented Jun 12, 2018 •

edited

Loading

nikhita commented Jun 13, 2018

nikhita commented Jun 13, 2018

nikhita commented Jun 13, 2018

nikhita commented Jun 13, 2018

liggitt Jun 13, 2018

caesarxuchao Jun 13, 2018

liggitt commented Jun 13, 2018

sttts Jun 13, 2018

caesarxuchao Jun 13, 2018

sttts commented Jun 13, 2018

liggitt Jun 13, 2018 •

edited

Loading

caesarxuchao Jun 13, 2018

caesarxuchao commented Jun 13, 2018

caesarxuchao commented Jun 19, 2018

kfox1111 commented Jun 28, 2018

liggitt commented Jun 28, 2018

Make kubernetes json serializer case sensitive #65034

Make kubernetes json serializer case sensitive #65034

Conversation

caesarxuchao commented Jun 12, 2018 • edited Loading

nikhita commented Jun 13, 2018

nikhita commented Jun 13, 2018

nikhita commented Jun 13, 2018

nikhita commented Jun 13, 2018

liggitt Jun 13, 2018

Choose a reason for hiding this comment

caesarxuchao Jun 13, 2018

Choose a reason for hiding this comment

liggitt commented Jun 13, 2018

sttts Jun 13, 2018

Choose a reason for hiding this comment

caesarxuchao Jun 13, 2018

Choose a reason for hiding this comment

sttts commented Jun 13, 2018

liggitt Jun 13, 2018 • edited Loading

Choose a reason for hiding this comment

caesarxuchao Jun 13, 2018

Choose a reason for hiding this comment

caesarxuchao commented Jun 13, 2018

caesarxuchao commented Jun 19, 2018

kfox1111 commented Jun 28, 2018

liggitt commented Jun 28, 2018

caesarxuchao commented Jun 12, 2018 •

edited

Loading

liggitt Jun 13, 2018 •

edited

Loading