"unkeyable object" in sentry

[liggitt]

https://sentry.io/red-hat/openshift-3-dev-preview-int/issues/176887281/events/4583320669/

7k instances so far

*errors.errorString: unkeyable object: {ops-health-build-nodejs-ex-dev-preview-int-master-e5af0/nodejs-ex &TypeMeta{Kind:,APIVersion:,}}, object has no meta: object does not implement the Object interfaces
  File "/builddir/build/BUILD/atomic-openshift-git-0.ada983f/_output/local/go/src/github.com/openshift/origin/pkg/cmd/util/serviceability/panic.go", line 27, in CapturePanic-fm
  File "/builddir/build/BUILD/atomic-openshift-git-0.ada983f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go", line 49, in HandleCrash
  File "/usr/lib/golang/src/runtime/asm_amd64.s", line 479, in call32
  File "/usr/lib/golang/src/runtime/panic.go", line 458, in gopanic
  File "/builddir/build/BUILD/atomic-openshift-git-0.ada983f/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/eventqueue.go", line 187, in deltaCompressor
  File "/builddir/build/BUILD/atomic-openshift-git-0.ada983f/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/eventqueue.go", line 34, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.ada983f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go", line 573, in Compress
  File "/builddir/build/BUILD/atomic-openshift-git-0.ada983f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go", line 312, in queueActionLocked
  File "/builddir/build/BUILD/atomic-openshift-git-0.ada983f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go", line 490, in Replace
  File "/builddir/build/BUILD/atomic-openshift-git-0.ada983f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/cache/reflector.go", line 343, in syncWith
  File "/builddir/build/BUILD/atomic-openshift-git-0.ada983f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/cache/reflector.go", line 271, in ListAndWatch
  File "/builddir/build/BUILD/atomic-openshift-git-0.ada983f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/cache/reflector.go", line 202, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.ada983f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 88, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.ada983f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 89, in JitterUntil
  File "/builddir/build/BUILD/atomic-openshift-git-0.ada983f/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 49, in Until
  File "/usr/lib/golang/src/runtime/asm_amd64.s", line 2086, in goexit

not sure the source... maybe build based on some of the attempted key values

@deads2k @bparees

[liggitt]

p1 until triaged based on instance count

[bparees]

looks similar to this? #9830

which was fixed here:
#10581

possible online doesn't have this fix yet?

[liggitt]

dev-int is on 3.4.0.21, and git says that fix is in it:

git tag --contains c7d28dd
...
v3.4.0.21
v3.4.0.22
...

[bparees]

  File "/builddir/build/BUILD/atomic-openshift-git-0.ada983f/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/eventqueue.go", line 187, in deltaCompressor
  File "/builddir/build/BUILD/atomic-openshift-git-0.ada983f/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/eventqueue.go", line 34, in func1

would seem to point to the sdn?

[liggitt]

totally missed that, yup.

[bparees]

the fix i referenced above may be relevant to the issue in the sdn eventqueue implementation.

[knobunc]

This is bug 1389770 link

PR is open #11792