Failed to list *unstructured.Unstructured errors in kubemetrics #2577

miguelsorianod · 2020-02-18T13:18:15Z

Hi,

I'm trying to upgrade our existing operator to be based on operator-sdk v0.15.2 but I'm experiencing problems that seem related to the addition of metrics.

What did you do?
Run an operator.
I've also made sure that the changes implemented in 0.9.x related to metrics https://github.com/operator-framework/operator-sdk/blob/master/doc/migration/version-upgrade-guide.md#v09x and newer changes are incorporated into the cmd/manager/main.go file.

What did you expect to see?
The operator starts without errors

What did you see instead? Under which circumstances?
When starting the operator, lots of errors related to kube-metrics are reported.

For example:

"level":"info","ts":1582026342.3808215,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1582026343.9847555,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"}
{"level":"info","ts":1582026343.9851718,"logger":"cmd","msg":"Registering Components."}
{"level":"debug","ts":1582026343.9878285,"logger":"k8sutil","msg":"Found namespace","Namespace":"msoriano-test"}
{"level":"debug","ts":1582026343.9878504,"logger":"kubemetrics","msg":"Starting collecting operator types"}
{"level":"debug","ts":1582026343.9878607,"logger":"kubemetrics","msg":"Generating metric families","apiVersion":"extensions/v1beta1","kind":"ReplicaSet"}
{"level":"debug","ts":1582026345.5901196,"logger":"kubemetrics","msg":"Generating metric families","apiVersion":"image.openshift.io/pre012","kind":"DockerImage"}
E0218 11:45:45.591312       1 reflector.go:123] github.com/operator-framework/operator-sdk/pkg/kube-metrics/store.go:65: Failed to list *unstructured.Unstructured: replicasets.extensions is forbidden: User "system:serviceaccount:msoriano-test:3scale-operator" cannot list resource "replicasets" in API group "extensions" in the namespace "msoriano-test"
E0218 11:45:46.593154       1 reflector.go:123] github.com/operator-framework/operator-sdk/pkg/kube-metrics/store.go:65: Failed to list *unstructured.Unstructured: replicasets.extensions is forbidden: User "system:serviceaccount:msoriano-test:3scale-operator" cannot list resource "replicasets" in API group "extensions" in the namespace "msoriano-test"
{"level":"info","ts":1582026347.1919956,"logger":"cmd","msg":"Could not generate and serve custom resource metrics","error":"discovering resource information failed for DockerImage in image.openshift.io/pre012: apiVersion image.openshift.io/pre012 and kind DockerImage not found available in Kubernetes cluster"}
E0218 11:45:47.594086       1 reflector.go:123] github.com/operator-framework/operator-sdk/pkg/kube-metrics/store.go:65: Failed to list *unstructured.Unstructured: replicasets.extensions is forbidden: User "system:serviceaccount:msoriano-test:3scale-operator" cannot list resource "replicasets" in API group "extensions" in the namespace "msoriano-test"
E0218 11:45:48.595011       1 reflector.go:123] github.com/operator-framework/operator-sdk/pkg/kube-metrics/store.go:65: Failed to list *unstructured.Unstructured: replicasets.extensions is forbidden: User "system:serviceaccount:msoriano-test:3scale-operator" cannot list resource "replicasets" in API group "extensions" in the namespace "msoriano-test"
{"level":"debug","ts":1582026348.794077,"logger":"k8sutil","msg":"Found namespace","Namespace":"msoriano-test"}
{"level":"debug","ts":1582026348.794106,"logger":"k8sutil","msg":"Found podname","Pod.Name":"3scale-operator-745dd8479c-r7zpm"}
{"level":"debug","ts":1582026348.7974224,"logger":"k8sutil","msg":"Found Pod","Pod.Namespace":"msoriano-test","Pod.Name":"3scale-operator-745dd8479c-r7zpm"}
{"level":"debug","ts":1582026348.8036382,"logger":"metrics","msg":"Pods owner found","Kind":"Deployment","Name":"3scale-operator","Namespace":"msoriano-test"}
{"level":"info","ts":1582026348.8301811,"logger":"metrics","msg":"Metrics Service object updated","Service.Name":"threescale-operator-metrics","Service.Namespace":"msoriano-test"}
E0218 11:45:49.596072       1 reflector.go:123] github.com/operator-framework/operator-sdk/pkg/kube-metrics/store.go:65: Failed to list *unstructured.Unstructured: replicasets.extensions is forbidden: User "system:serviceaccount:msoriano-test:3scale-operator" cannot list resource "replicasets" in API group "extensions" in the namespace "msoriano-test"
{"level":"info","ts":1582026350.4440339,"logger":"cmd","msg":"Could not create ServiceMonitor object","error":"servicemonitors.monitoring.coreos.com \"threescale-operator-metrics\" already exists"}
{"level":"info","ts":1582026350.444059,"logger":"cmd","msg":"Starting the Cmd."}
...
{"level":"info","ts":1582026351.0462596,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"binding-controller","worker count":1}
E0218 11:45:51.598854       1 reflector.go:123] github.com/operator-framework/operator-sdk/pkg/kube-metrics/store.go:65: Failed to list *unstructured.Unstructured: replicasets.extensions is forbidden: User "system:serviceaccount:msoriano-test:3scale-operator" cannot list resource "replicasets" in API group "extensions" in the namespace "msoriano-test"
E0218 11:45:52.600073       1 reflector.go:123] github.com/operator-framework/operator-sdk/pkg/kube-metrics/store.go:65: Failed to list *unstructured.Unstructured: replicasets.extensions is forbidden: User "system:serviceaccount:msoriano-test:3scale-operator" cannot list resource "replicasets" in API group "extensions" in the namespace "msoriano-test"
E0218 11:45:53.601243       1 reflector.go:123] github.com/operator-framework/operator-sdk/pkg/kube-metrics/store.go:65: Failed to list *unstructured.Unstructured: replicasets.extensions is forbidden: User "system:serviceaccount:msoriano-test:3scale-operator" cannot list resource "replicasets" in API group "extensions" in the namespace "msoriano-test"
...

Other surprising errors that we've seen among the logs/executions are:

{"level":"debug","ts":1582026343.9878607,"logger":"kubemetrics","msg":"Generating metric families","apiVersion":"extensions/v1beta1","kind":"ReplicaSet"}
{"level":"debug","ts":1582026345.5901196,"logger":"kubemetrics","msg":"Generating metric families","apiVersion":"image.openshift.io/pre012","kind":"DockerImage"}

or:

{"level":"info","ts":1582025802.992184,"logger":"cmd","msg":"Could not generate and serve custom resource metrics","error":"discovering resource information failed for RangeAllocation in v1: apiVersion v1 and kind RangeAllocation not foun
d available in Kubernetes cluster"}

The errors keep being repeated constantly.
Additionally, each time the operator starts the failed to list errors are different each time apparently (the types that they refer to).
Some of the error messages are kind of surprising too. For example, this one:

E0218 11:12:23.353013       1 reflector.go:123] github.com/operator-framework/operator-sdk/pkg/kube-metrics/store.go:65: Failed to list *unstructured.Unstructured: daemonsets.extensions is forbidden: User "system:serviceaccount:msoriano-test:3scale-operator" cannot list resource "daemonsets" in API group "extensions" in the namespace "msoriano-test"

That refers to daemonsets. In our operator code we are not using any DaemonSet.

The role.yaml file is:


apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  creationTimestamp: null
  name: 3scale-operator
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - replicationcontrollers
  - services
  - services/finalizers
  - endpoints
  - persistentvolumeclaims
  - events
  - configmaps
  - secrets
  - serviceaccounts
  # For some reason there's an error creating serviceaccounts
  # if you do not include permissions to bindings/finalizers.
  # A related PR to this problem is:
  # https://github.com/openshift/origin/pull/16253
  - bindings/finalizers
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - apps
  resources:
  - deployments
  - daemonsets
  - replicasets
  - statefulsets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - monitoring.coreos.com
  resources:
  - servicemonitors
  verbs:
  - get
  - create
- apiGroups:
  - apps
  resourceNames:
  - 3scale-operator
  resources:
  - deployments/finalizers
  verbs:
  - update
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - apps
  resources:
  - replicasets
  - deployments
  verbs:
  - get
- apiGroups:
  - apps.3scale.net
  resources:
  - '*'
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - capabilities.3scale.net
  resources:
  - '*'
  - bindings
  - metrics
  - plans
  - limits
  - mappingrules
  - tenants
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - roles
  - rolebindings
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - image.openshift.io
  resources:
  - imagestreams
  - imagestreams/layers
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - route.openshift.io
  resources:
  - routes
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - route.openshift.io
  resources:
  - routes/custom-host
  verbs:
  - create
- apiGroups:
  - route.openshift.io
  resources:
  - routes/status
  verbs:
  - get
- apiGroups:
  - apps.openshift.io
  resources:
  - deploymentconfigs
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - get
  - list
  - create
  - update
  - watch
  - delete

To provide some more information that might be relevant, we are deploying OpenShift objects too, and on our addtoscheme_* file in the pkg/apis package we add some OpenShift schemes into the "AddToSchemes" variable:

// File pkg/apis/addtoscheme_apps_v1alpha1.go
func init() {
	// Register the types with the Scheme so the components can map objects to GroupVersionKinds and back
	AddToSchemes = append(AddToSchemes, v1alpha1.SchemeBuilder.AddToScheme)
	registerOpenShiftAPIGroups()
}

func registerOpenShiftAPIGroups() {
	AddToSchemes = append(AddToSchemes,
		appsv1.Install,
		imagev1.Install,
		routev1.Install,
	)
}

Do you know what might be happening?

Environment

operator-sdk version:

operator-sdk version: "v0.15.2", commit: "ffaf278993c8fcb00c6f527c9f20091eb8dd3352", go version: "go1.13.3 linux/amd64"

go version:

go version go1.13.7 linux/amd64

Kubernetes version information:

Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.34-202002031224+5cdd27b-dirty", GitCommit:"5cdd27b", GitTreeState:"dirty", BuildDate:"2020-02-03T13:59:07Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+09f0e83", GitCommit:"09f0e83", GitTreeState:"clean", BuildDate:"2019-11-12T09:34:07Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes cluster kind:
OpenShift 4.1.24 installation
Are you writing your operator in ansible, helm, or go?
The operator is written in Go

The text was updated successfully, but these errors were encountered:

varshaprasad96 · 2020-02-24T21:35:17Z

Hello @miguelsorianod
I tried replicating this issue on an example operator, but I am not getting any errors while collecting custom metrics. Could you please provide a link to your operator, so that we can check if this is operator-specific. It would be helpful to debug this issue and find the cause.

Thank you!

camilamacedo86 · 2020-02-24T22:52:43Z

Hi @miguelsorianod,

See that your error is related to permissions.

replicasets.extensions is forbidden: User "system:serviceaccount:msoriano-test:3scale-operator" cannot list resource "replicasets" in API group "extensions" in the namespace "msoriano-test"

Shows that you are missing the permission to list the replicasets. I'd recommend you create a new project just to compare the current role permissions file scaffolded with yours project. You can also use the Memcached sample to do this check. See here

Could you please ensure that you have all permissions which are scaffolds by default such here and let us know if your problem was solved?

miguelsorianod · 2020-02-25T10:33:26Z

Hi @camilamacedo86,

Our operator does not make use of replicasets.
However, the file role.yaml has permissions to manage them anyway because we maintained the role permissions that are scaffolded when creating a new operator-sdk project.
You can see the content of our role.yaml file here:

https://github.com/3scale/3scale-operator/blob/needed-changes-operator-sdk-upgrade/deploy/role.yaml

A commit that can be used to see/use the operator code is:

https://github.com/3scale/3scale-operator/commit/61d3d730556cb19c2e11c7b8180b293d8bf0a54f

camilamacedo86 · 2020-02-25T10:44:29Z

HI @miguelsorianod,

The permission in the file 3scale/3scale-operator@61d3d73 apparently shows correctly. However, note that you are creating a Role and RoleBinding.

Are the ALL resources applied in the same namespace as your operator? If not, then, you need to set up it as cluster-scope. If yes, ensure that you are applying all resources indeed the role, service account and etc in the same namespace used by your operator.

Please, let us know if it worked.

miguelsorianod · 2020-02-25T10:48:38Z

Hi,

I can confirm we ensure that we are applying/creating all the resources (role, service account and rolebinding) in the same namespace used by our operator.
The errors are shown just starting the operator (without any CR being created).

camilamacedo86 · 2020-02-25T12:09:54Z

Hi @miguelsorianod,

See that the error faced by you is:

cannot list resource "replicasets" in API group "extensions" in the namespace "msoriano-test"`
And then,
Could not generate and serve custom resource metrics"

So, it means that an RBCA permission is missing and because of this, you face an issue in serveCRMetrics in your code implementation.

However, by looking at your code you are NOT using the SDK implementation to generate the metrics for the custom resources. See that your serveCRMetrics is not the SDK implementation to do it. Also, note that you have other customizations in place as well to achieve it.

In this way, as you can check in the logs and in your code, it is NOT related to SDK features and implementations provided by it.

Then, I am closing it as sorted out since we could confirm that the error faced came from a code which is NOT the same provided by SDK to export the metrics as we could ensure that this error cannot be reproduced with the implementations provided by SDK as well.

I hope that the above info helps you solve the problem in your code customizations used to achieve this goal. Please, let us know if you think it should be re-opened because something is missing to be addressed here.

miguelsorianod · 2020-02-25T13:20:44Z

Hi @camilamacedo86,

I think there's been some confusion. The implementation you linked to belongs to a new commit that we added to try to workaround the issue that we reported.
That's why instead of telling the branch "changes-operator-sdk-upgrade" could be used we reported that the following commit id could be used:

https://github.com/3scale/3scale-operator/commit/61d3d730556cb19c2e11c7b8180b293d8bf0a54f

This commit is the second one in that branch and does not include the workaround customizations that we applied.

If you checkout that commit (instead of the "changes-operator-sdk-upgrade" branch directly) you will see that those modifications do not exist and that we are using the SDK implementation for serveCRMetrics:
https://github.com/3scale/3scale-operator/blob/61d3d730556cb19c2e11c7b8180b293d8bf0a54f/cmd/manager/main.go#L188

Sorry for the confusion.

camilamacedo86 · 2020-02-25T13:47:20Z

Hi @miguelsorianod,

Note that it still having customizations. See for example https://github.com/3scale/3scale-operator/blob/61d3d730556cb19c2e11c7b8180b293d8bf0a54f/cmd/manager/main.go#L134. Also, note your imports. And if I check the open PRs, if you are using this one https://github.com/3scale/3scale-operator/pull/327/files#diff-9e63f7b0791edcb4a53c944552b88970R199 to test it is not using the SDK code impl to create thee metrics.

Also, we do not really know how your project works, however, we are unable to reproduce your scenario by using any project scaffolded with SDK which is using the implementation provided by it without any customization. Could you provide the steps for we face the same by using the Memcached Samples, for example, or by following the Getting or Quick Started?

miguelsorianod · 2020-02-25T14:43:58Z

Hi,

I'm unable to reproduce it with an empty operator-sdk project either. An empty operator project called "exampleproject" has been created in the following way:

operator-sdk new exampleproject
operator-sdk add api --api-version app.example.com/v1alpha1 --kind AppService
operator-sdk add controller --api-version app.example.com/v1alpha1 --kind AppService
operator-sdk generate k8s
operator-sdk generate openapi

Regarding to still having customizations: Those are not related to serveCRMetrics. In any case, to avoid confusions when referring to files and just in case to discard any problem related to that I created a new branch called "remove-maincustomization" available here: https://github.com/3scale/3scale-operator/tree/remove-maincustomization .
There the main.go file does not contain any customization or imports.

By diffing locally the main.go file in the project where we have the issue with the main.go file of the new empty operator project (named exampleproject) it can be verified that there's not customization:

msoriano@localhost:~/go/src/github.com/3scale/3scale-operator (remove-maincustomization)$ diff -u cmd/manager/main.go ../exampleproject/cmd/manager/main.go 
--- cmd/manager/main.go	2020-02-25 15:29:14.893149430 +0100
+++ ../exampleproject/cmd/manager/main.go	2020-02-25 14:50:40.583333593 +0100
@@ -12,9 +12,9 @@
 	_ "k8s.io/client-go/plugin/pkg/client/auth"
 	"k8s.io/client-go/rest"
 
-	"github.com/3scale/3scale-operator/pkg/apis"
-	"github.com/3scale/3scale-operator/pkg/controller"
-	"github.com/3scale/3scale-operator/version"
+	"github.com/3scale/exampleproject/pkg/apis"
+	"github.com/3scale/exampleproject/pkg/controller"
+	"github.com/3scale/exampleproject/version"
 
 	"github.com/operator-framework/operator-sdk/pkg/k8sutil"
 	kubemetrics "github.com/operator-framework/operator-sdk/pkg/kube-metrics"
@@ -84,7 +84,7 @@
 
 	ctx := context.TODO()
 	// Become the leader before proceeding
-	err = leader.Become(ctx, "3scale-operator-lock")
+	err = leader.Become(ctx, "exampleproject-lock")
 	if err != nil {
 		log.Error(err, "")
 		os.Exit(1)

Notice how the only differences detected are due to changes in the operator project name.

Then we deploy the operator with:

operator-sdk build quay.io/miguelsorianod/operator-dev:issue-2577
docker push quay.io/miguelsorianod/operator-dev:issue-2577
oc new-project extraproject
for i in `ls -1 | grep -i crd`; do oc create -f $i; done # create CRDs associated to the operator
oc create -f deploy/role.yaml
oc create -f deploy/service_account.yaml
oc create -f deploy/role_binding.yaml
oc create -f operator.yaml

and looking at the operator pod logs we see all those RBAC related problems

camilamacedo86 · 2020-02-25T16:09:44Z

Hi @miguelsorianod,

Note that it is really more specific to the process to upgrade your project and its code than SDK features. So, I am not sure how we can help you more with. I'd probably recommend you do it step by step to ensure all and has fewer chances to see what may be wrong or missing.

Please, let us know if you could solve it.

camilamacedo86 · 2020-02-26T01:52:21Z

Hi @miguelsorianod,

All information provided so far still valid. However, we could reproduce a similar issue which may also be valid in your case. See #1858 (comment)

Then, I thought that the above scenario can be helpful for you. Also, if you are not using the default metrics provided by the SDK then you may also can just not use the addMetrics implementation for now until all your project be upgraded successfully since it can be added later.

c/c @varshaprasad96

miguelsorianod · 2020-02-26T10:13:59Z

Hi @camilamacedo86,

We have been doing more tests and this is what we have found:

The issue we experienced is not related to the operator-sdk upgrade process for our project.
The issue is related to the addition of schemas to the operator's runtime.Schema and how kube-metrics handle them. In our specific case, it is due to the addition of OpenShift specific schemas.

For what we have seen, kube-metrics tries to list all the GVKs defined in the GKVs it receives, whether they are being used or not.
As we reported at the beginning of the issue, we have the following code that adds OpenShift-specific schemas to the runtime.Schema of the operator so we are able to use OpenShift Kinds:

// File pkg/apis/addtoscheme_apps_v1alpha1.go
func init() {
	// Register the types with the Scheme so the components can map objects to GroupVersionKinds and back
	AddToSchemes = append(AddToSchemes, v1alpha1.SchemeBuilder.AddToScheme)
	registerOpenShiftAPIGroups()
}

func registerOpenShiftAPIGroups() {
	AddToSchemes = append(AddToSchemes,
		appsv1.Install,
		imagev1.Install,
		routev1.Install,
	)
}

Creating a new empty operator-sdk project v0.15.2 and just setting that modification we are able to reproduce the problem.

If we focus for example in the line imagev1.Install, "imagev1" is the import:

	imagev1 "github.com/openshift/api/image/v1"

And imagev1.Install corresponds to:
https://github.com/openshift/api/blob/c99157bc6492218400b18cde7f66510f9ee56a9a/image/v1/register.go#L18

Which is an addToScheme function. That addToScheme function (https://github.com/openshift/api/blob/c99157bc6492218400b18cde7f66510f9ee56a9a/image/v1/register.go#L16) does:

	schemeBuilder = runtime.NewSchemeBuilder(addKnownTypes, docker10.AddToScheme, dockerpre012.AddToScheme, corev1.AddToScheme)

Notice for example the dockerpre012.AddToScheme, which is another package in OpenShift
https://github.com/openshift/api/blob/master/image/dockerpre012/register.go that adds another kinds to the scheme (https://github.com/openshift/api/blob/master/image/dockerpre012/register.go#L25)

func addKnownTypes(scheme *runtime.Scheme) error {
	scheme.AddKnownTypes(SchemeGroupVersion,
		&DockerImage{},
	)
	return nil
}

Our operator does not make use of image.openshift.io/pre012 defined Kinds. Consequently, we don't have permissions for those kinds or API Group on the role.yaml as we don't use them in our operator nor we desire to give unnecessary permissions to it. We just use a subset of the registered Kinds. However kube-metrics tries to performs actions with all of them (List). This happens with all registered Kinds that are automatically added when you add them to the Scheme. The observed log errors are due to those Kinds that we don't have permissions for, as we don't use them, but they are added to the list of GKVs anyway and kube-metrics list them all.

In order to avoid having to add permissions in our role.yaml for Kinds or API groups that our operator doesn't use at all we have tried to apply a workaround to further filter the GKVs before sending them to kube-metrics. This can be seen here:
3scale/3scale-operator@df49604

camilamacedo86 · 2020-02-26T10:55:07Z

Hi @miguelsorianod,

Thank you for your input and reply. Also, really thank you for providing the steps and information for we are able to check it. We will be checking this one and I will keep you updated.

c/c @varshaprasad96

camilamacedo86 · 2020-02-27T23:54:10Z

Hi @miguelsorianod,

Really thank you for your inputs. They were very helpful. Besides, I understand it is solved for you already let's add a summary here for you check the taken actions as well.

The error Failed to list *unstructured.Unstructured which is faced means that the operator has not the permission to List some GroupVersionKind(GVK). However, more than one scenario can cause it, follow some examples:

The operator is not with the required permissions (missing RBCA/roles). See a POC in the comment Metrics of the Cluster scoped operator causes can't list unstructured object errors #1858 (comment)
The cache.manager does not receive the namespace where the operator is deployed in. See the issue: Bug creating metric service monitors when watching a different namespace #2494 and the PR to address it: Scaffold code in cmd/manager/main.go for Go operators and add logic to Ansible/Helm operators to handle [multinamespace caching] #2522
Or when the WATCH_NAMESPACE is empty. See the issue: Unable to create ServiceMonitor when WATCH_NAMESPACE has not the namespace used to deploy the operator #2602 and the PR to address it: fix: creation of monitor service when operator is cluster-scoped #2601
And then, when third party APIs schemas were added which shows your scenario. So, the solution is to customize the implementation. See the PR to address it: Update the docs and scaffold to make clear how works the metrics and what is required to do for third party API's. #2606

Please, let us know if you have anything more that you believe that should be addressed here. Otherwise, this issue will be closed when the PR #2606 get merged.

operator-framework/operator-sdk#2577

miguelsorianod changed the title ~~Failed to list *unstructured.Unstructured errors being reported by kubemetrics on operator start~~ Failed to list *unstructured.Unstructured errors in kubemetrics Feb 18, 2020

miguelsorianod mentioned this issue Feb 18, 2020

Needed changes operator sdk upgrade 3scale/3scale-operator#327

Merged

jmccormick2001 self-assigned this Feb 18, 2020

jmccormick2001 added the kind/bug Categorizes issue or PR as related to a bug. label Feb 18, 2020

varshaprasad96 self-assigned this Feb 21, 2020

ron1 mentioned this issue Feb 24, 2020

Resolve or work-around potential operator-sdk issue 2577 openshift/tektoncd-pipeline-operator#243

Closed

camilamacedo86 closed this as completed Feb 25, 2020

camilamacedo86 self-assigned this Feb 25, 2020

camilamacedo86 added triage/support Indicates an issue that is a support question. and removed kind/bug Categorizes issue or PR as related to a bug. labels Feb 25, 2020

camilamacedo86 reopened this Feb 25, 2020

camilamacedo86 closed this as completed Feb 25, 2020

camilamacedo86 reopened this Feb 25, 2020

estroz unassigned jmccormick2001 and camilamacedo86 Feb 25, 2020

camilamacedo86 self-assigned this Feb 26, 2020

camilamacedo86 mentioned this issue Feb 27, 2020

Update the docs and scaffold to make clear how works the metrics and what is required to do for third party API's. #2606

Merged

camilamacedo86 added kind/documentation Categorizes issue or PR as related to documentation. kind/bug Categorizes issue or PR as related to a bug. and removed triage/support Indicates an issue that is a support question. labels Feb 27, 2020

camilamacedo86 closed this as completed in #2606 Mar 9, 2020

joelanford mentioned this issue Apr 7, 2020

Added variadic string param to GetGVKsFromAddToScheme for customkinds #2794

Closed

tomaszsek added a commit to jenkinsci/kubernetes-operator that referenced this issue Jun 7, 2020

Remove unwanted schemes

445cf2e

operator-framework/operator-sdk#2577

tomaszsek mentioned this issue Jun 19, 2020

Fix GVK Parsng and restore Types in Schema jenkinsci/kubernetes-operator#413

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to list *unstructured.Unstructured errors in kubemetrics #2577

Failed to list *unstructured.Unstructured errors in kubemetrics #2577

miguelsorianod commented Feb 18, 2020

varshaprasad96 commented Feb 24, 2020

camilamacedo86 commented Feb 24, 2020 •

edited

Loading

miguelsorianod commented Feb 25, 2020

camilamacedo86 commented Feb 25, 2020 •

edited

Loading

miguelsorianod commented Feb 25, 2020

camilamacedo86 commented Feb 25, 2020 •

edited

Loading

miguelsorianod commented Feb 25, 2020 •

edited

Loading

camilamacedo86 commented Feb 25, 2020 •

edited

Loading

miguelsorianod commented Feb 25, 2020

camilamacedo86 commented Feb 25, 2020 •

edited

Loading

camilamacedo86 commented Feb 26, 2020 •

edited

Loading

miguelsorianod commented Feb 26, 2020 •

edited

Loading

camilamacedo86 commented Feb 26, 2020 •

edited

Loading

camilamacedo86 commented Feb 27, 2020 •

edited

Loading

Failed to list *unstructured.Unstructured errors in kubemetrics #2577

Failed to list *unstructured.Unstructured errors in kubemetrics #2577

Comments

miguelsorianod commented Feb 18, 2020

varshaprasad96 commented Feb 24, 2020

camilamacedo86 commented Feb 24, 2020 • edited Loading

miguelsorianod commented Feb 25, 2020

camilamacedo86 commented Feb 25, 2020 • edited Loading

miguelsorianod commented Feb 25, 2020

camilamacedo86 commented Feb 25, 2020 • edited Loading

miguelsorianod commented Feb 25, 2020 • edited Loading

camilamacedo86 commented Feb 25, 2020 • edited Loading

miguelsorianod commented Feb 25, 2020

camilamacedo86 commented Feb 25, 2020 • edited Loading

camilamacedo86 commented Feb 26, 2020 • edited Loading

miguelsorianod commented Feb 26, 2020 • edited Loading

camilamacedo86 commented Feb 26, 2020 • edited Loading

camilamacedo86 commented Feb 27, 2020 • edited Loading

camilamacedo86 commented Feb 24, 2020 •

edited

Loading

camilamacedo86 commented Feb 25, 2020 •

edited

Loading

camilamacedo86 commented Feb 25, 2020 •

edited

Loading

miguelsorianod commented Feb 25, 2020 •

edited

Loading

camilamacedo86 commented Feb 25, 2020 •

edited

Loading

camilamacedo86 commented Feb 25, 2020 •

edited

Loading

camilamacedo86 commented Feb 26, 2020 •

edited

Loading

miguelsorianod commented Feb 26, 2020 •

edited

Loading

camilamacedo86 commented Feb 26, 2020 •

edited

Loading

camilamacedo86 commented Feb 27, 2020 •

edited

Loading