Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make kubernetes json serializer case sensitive #65034

Merged
merged 3 commits into from Jun 14, 2018

Conversation

@caesarxuchao
Copy link
Member

caesarxuchao commented Jun 12, 2018

This PR imported the latest jsoniterator library so that case sensitivity during unmarhsaling is optional. The PR also set Kubernetes json serializer to be case sensitive.

Kubernetes json serializer had been case sensitive for 1.1-1.7 as we were using ugorji. This PR restores the behavior.

Fix #64612.

Kubernetes json deserializer is now case-sensitive to restore compatibility with pre-1.8 servers.
If your config files contain fields with wrong case, the config files will be now invalid.

Action required: 
Fix your JSON config files that contain fields of wrong case, otherwise they will be invalid.
If the fields in the golang schema doesn't have json tags, e.g., missing `json:"podSpec,omitempty"`, then the field in the JSON config file should match the case of the golang field, e.g., CamelCase.

Known non camelCase fields:

v1.10.5, scheduler/v1#ExtenderConfig.BindVerb, in the JSON config file, it needs to be "BindVerb".

@k8s-ci-robot k8s-ci-robot requested review from dchen1107 and deads2k Jun 12, 2018

@caesarxuchao caesarxuchao force-pushed the caesarxuchao:json-case-sensitive branch from 84be0b0 to c9efab0 Jun 12, 2018

@caesarxuchao caesarxuchao force-pushed the caesarxuchao:json-case-sensitive branch from c9efab0 to f5e4ac6 Jun 12, 2018

@nikhita

This comment has been minimized.

Copy link
Member

nikhita commented Jun 13, 2018

I think some tests need fixing because of the ExcludeCIDRs change. And staging godeps needs to be fixed. But looks good otherwise. 👍

Should this be cherry-picked into 1.8+? I'm not sure if that is such a good idea since this does a dependency bump...

@nikhita

This comment has been minimized.

Copy link
Member

nikhita commented Jun 13, 2018

/kind bug
/priority critical-urgent

Needs a milestone update too.

@nikhita

This comment has been minimized.

Copy link
Member

nikhita commented Jun 13, 2018

cc @cblecker @sttts for godeps bump/approval

@nikhita

This comment has been minimized.

Copy link
Member

nikhita commented Jun 13, 2018

/uncc @dchen1107

@k8s-ci-robot k8s-ci-robot removed the request for review from dchen1107 Jun 13, 2018

@@ -157,7 +169,7 @@ func (s *Serializer) Decode(originalData []byte, gvk *schema.GroupVersionKind, i
types, _, err := s.typer.ObjectKinds(into)
switch {
case runtime.IsNotRegisteredError(err), isUnstructured:
if err := jsoniter.ConfigCompatibleWithStandardLibrary.Unmarshal(data, into); err != nil {
if err := caseSensitiveJsonIterator().Unmarshal(data, into); err != nil {

This comment has been minimized.

@liggitt

liggitt Jun 13, 2018

Member

private package var rather than reconstructing the options every time?

This comment has been minimized.

@caesarxuchao

caesarxuchao Jun 13, 2018

Author Member

Done.

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Jun 13, 2018

more godeps need fixing (apiextensions-apiserver/Godeps/Godeps.json)... should find/replace the json-iter revision in all of them

Other: "test",
},
},
// Unmarshalling is case-sensitive for big strut.

This comment has been minimized.

@sttts

sttts Jun 13, 2018

Contributor

nit: struct

This comment has been minimized.

@caesarxuchao

caesarxuchao Jun 13, 2018

Author Member

Done.

@sttts

This comment has been minimized.

Copy link
Contributor

sttts commented Jun 13, 2018

/approve

@caesarxuchao caesarxuchao force-pushed the caesarxuchao:json-case-sensitive branch from 67ff1fe to 0334021 Jun 13, 2018

@@ -157,7 +167,7 @@ func (s *Serializer) Decode(originalData []byte, gvk *schema.GroupVersionKind, i
types, _, err := s.typer.ObjectKinds(into)
switch {
case runtime.IsNotRegisteredError(err), isUnstructured:
if err := jsoniter.ConfigCompatibleWithStandardLibrary.Unmarshal(data, into); err != nil {

This comment has been minimized.

@liggitt

liggitt Jun 13, 2018

Member

looks like there are four other uses of jsoniter.ConfigCompatibleWithStandardLibrary in the codebase... should replace those as well (and as a follow-up think about how to ensure we only use a standard consistent config; possibly something like #63284)

This comment has been minimized.

@caesarxuchao

caesarxuchao Jun 13, 2018

Author Member

Replaced the other references in the latest commit.

#63284 is a good follow-up.

@k8s-ci-robot k8s-ci-robot added size/L and removed size/M labels Jun 13, 2018

@caesarxuchao

This comment has been minimized.

Copy link
Member Author

caesarxuchao commented Jun 13, 2018

/retest

@k8s-github-robot

This comment has been minimized.

Copy link
Contributor

k8s-github-robot commented Jun 14, 2018

/test all [submit-queue is verifying that this PR is safe to merge]

@k8s-github-robot

This comment has been minimized.

Copy link
Contributor

k8s-github-robot commented Jun 14, 2018

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-github-robot k8s-github-robot merged commit a2de139 into kubernetes:master Jun 14, 2018

17 of 18 checks passed

Submit Queue Required Github CI test is not green: pull-kubernetes-kubemark-e2e-gce-big
Details
cla/linuxfoundation caesarxuchao authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gke Skipped
pull-kubernetes-e2e-kops-aws Job succeeded.
Details
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped
pull-kubernetes-local-e2e-containerized Skipped
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details

k8s-github-robot pushed a commit that referenced this pull request Jun 15, 2018

Kubernetes Submit Queue
Merge pull request #65138 from liggitt/crd-objectmeta
Automatic merge from submit-queue (batch tested with PRs 64838, 65138). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

use case-sensitive json-iter to decode custom resource ObjectMeta

found one additional spot where case-insensitivity could bite us. uses the case-sensitive decode added in #65034


/assign @sttts
/cc @caesarxuchao 

```release-note
NONE
```

k8s-publishing-bot added a commit to kubernetes/apiextensions-apiserver that referenced this pull request Jun 15, 2018

Merge pull request #65138 from liggitt/crd-objectmeta
Automatic merge from submit-queue (batch tested with PRs 64838, 65138). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

use case-sensitive json-iter to decode custom resource ObjectMeta

found one additional spot where case-insensitivity could bite us. uses the case-sensitive decode added in kubernetes/kubernetes#65034


/assign @sttts
/cc @caesarxuchao 

```release-note
NONE
```

Kubernetes-commit: 4ba0dbde84ba54941d998fc93b863b7eb97b8d53

k8s-github-robot pushed a commit that referenced this pull request Jun 19, 2018

Kubernetes Submit Queue
Merge pull request #65157 from caesarxuchao/cherrypick-65034-1.10
Automatic merge from submit-queue.

Manually cherrypick #65034 to 1.10

Manually cherrypicking #65034. Using hack/cherry_pick_pull.sh to cherrypick is difficult because that requires cherrypicking #63059 first.

This PR imported the latest jsoniterator library so that case sensitivity during unmarhsaling is optional. The PR also set Kubernetes json serializer to be case sensitive.

Fix #64612.

```release-notes
Kubernetes json deserializer is now case-sensitive to restore compatibility with pre-1.8 servers.
If your config files contains fields with wrong case, the config files will be now invalid.
```
@caesarxuchao

This comment has been minimized.

Copy link
Member Author

caesarxuchao commented Jun 19, 2018

Cherrypicked to 1.10, still needs to be cherrypicked to 1.8 and 1.9.

k8s-github-robot pushed a commit that referenced this pull request Jun 22, 2018

Kubernetes Submit Queue
Merge pull request #65324 from nikhita/fix-nits-json-case-sensitive
Automatic merge from submit-queue (batch tested with PRs 65339, 65343, 65324, 65335, 65367). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix k8s json package import name

Fixes nits found while cherry-picking #65034 in #65298.

**Release note**:

```release-note
NONE
```

/cc sttts liggitt caesarxuchao

k8s-publishing-bot added a commit to kubernetes/apimachinery that referenced this pull request Jun 24, 2018

Merge pull request #65324 from nikhita/fix-nits-json-case-sensitive
Automatic merge from submit-queue (batch tested with PRs 65339, 65343, 65324, 65335, 65367). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix k8s json package import name

Fixes nits found while cherry-picking kubernetes/kubernetes#65034 in kubernetes/kubernetes#65298.

**Release note**:

```release-note
NONE
```

/cc sttts liggitt caesarxuchao

Kubernetes-commit: 456f85e736069398329226b9dff39d0304ee52d2

k8s-github-robot pushed a commit that referenced this pull request Jun 27, 2018

Kubernetes Submit Queue
Merge pull request #65298 from nikhita/cherrypick-jsoniter-bump-1.9
Automatic merge from submit-queue.

Manual cherrypick of #65034 to 1.9: make json serializer case sensitive

fixes partially #64612

This PR imports the latest jsoniterator library so that case sensitivity during unmarshalling is optional. The PR also sets Kubernetes json serializer to be case sensitive.

**Release note**:

```release-note
ACTION REQUIRED: Kubernetes JSON deserializer is now case-sensitive to restore compatibility with pre-1.8 servers. If your config files contains fields with wrong case, the config files will be now invalid.
```

/sig api-machinery
/kind bug
/assign caesarxuchao liggitt thockin sttts mbohlool
@kfox1111

This comment has been minimized.

Copy link

kfox1111 commented Jun 28, 2018

Saw this in the release notes. Are there any tools to verify your cluster only has objects with the correct casing, or does each site need to write a tool to do so?

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Jun 28, 2018

object data persisted in etcd is correct, so no stored data is problematic.

kubectl create/apply performs case-sensitive client-side validation which complains about incorrect/unknown fields.

For your resource manifest files, I'm not aware of a stand-alone tool that only validates a resource manifest against the openapi schema

k8s-github-robot pushed a commit that referenced this pull request Jun 28, 2018

Kubernetes Submit Queue
Merge pull request #65410 from nikhita/cherrypick-jsoniter-bump-1.8
Automatic merge from submit-queue.

Manual cherrypick of #65034 to 1.8: make json serializer case sensitive

fixes partially #64612

This PR imports the latest jsoniterator library so that case sensitivity during unmarshalling is optional. The PR also sets Kubernetes json serializer to be case sensitive.

**Release note**:

```release-note
ACTION REQUIRED: Kubernetes JSON deserializer is now case-sensitive to restore compatibility with pre-1.8 servers. If your config files contains fields with wrong case, the config files will be now invalid.
```

/sig api-machinery
/kind bug
/assign caesarxuchao liggitt thockin sttts mbohlool

klausenbusk added a commit to klausenbusk/bootkube that referenced this pull request Jul 10, 2018

Bump vendor k8s 1.11.0
Also bump json-iterator as required per: kubernetes/kubernetes#65034

klausenbusk added a commit to klausenbusk/bootkube that referenced this pull request Jul 11, 2018

Bump vendor k8s 1.11.0
Also bump json-iterator as required per: kubernetes/kubernetes#65034

rphillips added a commit to kubernetes-incubator/bootkube that referenced this pull request Jul 11, 2018

Bump v1.11.0 (#990)
* Bump v1.11.0

* Bump vendor k8s 1.11.0

Also bump json-iterator as required per: kubernetes/kubernetes#65034
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.