Invalid JSON received when watching events

[alex-ridge]

What happened?

While watching events, I received an invalid JSON.

I used the following command:

curl --cacert $ca_cert --header "Authorization: Bearer $token" "https://$KUBERNETES_SERVICE_HOST/api/v1/namespaces/kube-system/events?resourceVersion=$resource_version&watch=true"

And this is the invalid JSON I received:

{"type":"DELETED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"wireguard.16af72c4ed61d140","namespace":"kube-system","uid":"57d54486-bae8-4a7c-be28-1b12463ec987","resourceVersion":"1309353","creationTimestamp":"2021-10-21T08:32:34Z","managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-10-21T08:32:34Z","fieldsType":"FieldsV1","fieldsV1":{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{"f:apiVersion":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{}},"f:type":{}}}]},"involvedObject":{"kind":"Endpoints","namespace":"kube-system","name":"wireguard","uid":"4b1a9e7f-5ff1-4557-9f97-2ae4eed44439","apiVersion":"v1","resourceVersion":"1301666"},"reason":"FailedToUpdateEndpoint","message":"Failed to update endpoint kube-system/wireguard: Operation cannot be fulfilled on endpoints "wireguard": the object has been modified; please apply your changes to the latest version and try again","source":{"component":"endpoint-controller"},"firstTimestamp":"2021-10-19T13:49:39Z","lastTimestamp":"2021-10-21T08:32:34Z","count":6,"type":"Warning","eventTime":null,"reportingComponent":"","reportingInstance":""}}

The field .object.message contains quotes that are not escaped.

Here is the event pretty printed:

{
	"type": "DELETED",
	"object": {
		"kind": "Event",
		"apiVersion": "v1",
		"metadata": {
			"name": "wireguard.16af72c4ed61d140",
			"namespace": "kube-system",
			"uid": "57d54486-bae8-4a7c-be28-1b12463ec987",
			"resourceVersion": "1309353",
			"creationTimestamp": "2021-10-21T08:32:34Z",
		},
		"involvedObject": {
			"kind": "Endpoints",
			"namespace": "kube-system",
			"name": "wireguard",
			"uid": "4b1a9e7f-5ff1-4557-9f97-2ae4eed44439",
			"apiVersion": "v1",
			"resourceVersion": "1301666"
		},
		"reason": "FailedToUpdateEndpoint",
		"message": "Failed to update endpoint kube-system/wireguard: Operation cannot be fulfilled on endpoints "wireguard": the object has been modified; please apply your changes to the latest version and try again",
		"source": {
			"component": "endpoint-controller"
		},
		"firstTimestamp": "2021-10-19T13:49:39Z",
		"lastTimestamp": "2021-10-21T08:32:34Z",
		"count": 6,
		"type": "Warning",
		"eventTime": null,
		"reportingComponent": "",
		"reportingInstance": ""
	}
}

What did you expect to happen?

Receive a valid JSON (the endpoint name, here wireguard should be escaped)

How can we reproduce it (as minimally and precisely as possible)?

I don't know what lead to this error.

I have a ReplicaSet behind a Service. I killed the pod under the ReplicaSet to generate an event.

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"d94a81c724ea8e1ccc9002d89b7fe81d58f89ede", GitTreeState:"clean", BuildDate:"2020-03-12T21:08:59Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.8", GitCommit:"5575935422cc1cf5169dfc8847cb587aa47bac5a", GitTreeState:"clean", BuildDate:"2021-06-16T12:53:07Z", GoVersion:"go1.15.13", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

Ridge

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[sbarikdev]

@alex-ridge can you share all the steps you performed?

[alex-ridge]

These are the related objects I have on the cluster:

• a Service named wireguard with the selector k8s-app=wireguard
• a ReplicaSet named wireguard with the label and selector k8s-app=wireguard, with only one replica

I have a script running the curl above, inside a busybox pod, which watches for events related to the ReplicaSet (I have an additional parameter fieldSelector that filters on it). My goal is to catch the events where the pod restarts.

In order to simulate a pod restart, I killed the wireguard-xxx pod under the ReplicaSet, and that caused the event I quoted above.

[MadhavJivrajani]

/sig api-machinery

[fedebongio]

/cc @liggitt @jpbetz
/triage accepted

[liggitt]

what path is outputting unescaped json? that's ... very strange

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.