Operator not getting any requests for related CRD

[zonnie]

Bug Report

What did you do?

Followed the tutorial here operator-sdk tutorial

When running locally with
make run ENABLE_WEBHOOKS=false

It seems fine:

go: creating new go.mod: module tmp
go: finding sigs.k8s.io/controller-tools/cmd v0.3.0
go: finding sigs.k8s.io v0.3.0
go: finding sigs.k8s.io/controller-tools/cmd/controller-gen v0.3.0
/Users/yonis/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/Users/yonis/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go run ./main.go
2020-10-18T16:24:20.808+0300	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2020-10-18T16:24:20.809+0300	INFO	setup	starting manager
2020-10-18T16:24:20.809+0300	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2020-10-18T16:24:20.810+0300	INFO	controller	Starting EventSource	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "source": "kind source: /, Kind="}
2020-10-18T16:24:21.715+0300	INFO	controller	Starting EventSource	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "source": "kind source: /, Kind="}
2020-10-18T16:24:22.716+0300	INFO	controller	Starting Controller	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp"}
2020-10-18T16:24:22.716+0300	INFO	controller	Starting workers	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "worker count": 1}
2020-10-18T16:24:33.242+0300	INFO	controllers.MgmtApp	Creating a new Deployment	{"mgmtapp": "default/gc-mgmt-app-test", "Deployment.Namespace": "default", "Deployment.Name": "gc-mgmt-app-test"}
2020-10-18T16:24:36.339+0300	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}
2020-10-18T16:24:36.339+0300	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}
2020-10-18T16:24:36.470+0300	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}
2020-10-18T16:24:36.470+0300	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}
2020-10-18T16:25:27.081+0300	INFO	controllers.MgmtApp	MgmtApp resource not found. Ignoring since object must be deleted	{"mgmtapp": "default/gc-mgmt-app-test"}
2020-10-18T16:25:27.081+0300	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}
2020-10-18T16:25:27.796+0300	INFO	controllers.MgmtApp	MgmtApp resource not found. Ignoring since object must be deleted	{"mgmtapp": "default/gc-mgmt-app-test"}
2020-10-18T16:25:27.796+0300	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "name": "gc-mgmt-app-test", "namespace": "default"}

The deployments and pods are created successfully.

When running remotely on a GKE cluster (the same one the above uses):

2020-10-18T13:26:41.140Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": "127.0.0.1:8080"}
2020-10-18T13:26:41.228Z	INFO	setup	starting manager
I1018 13:26:41.233994       1 leaderelection.go:242] attempting to acquire leader lease  default/c17466cf....
2020-10-18T13:26:41.234Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
I1018 13:26:58.638603       1 leaderelection.go:252] successfully acquired lease default/c17466cf.
2020-10-18T13:26:58.638Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "source": "kind source: /, Kind="}
2020-10-18T13:26:58.638Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ConfigMap","namespace":"default","name":"c17466cf.","uid":"733ade09-f2ac-4d17-9168-936d439eed99","apiVersion":"v1","resourceVersion":"2278867"}, "reason": "LeaderElection", "message": "mgmt-app-operator-controller-manager-7d6df57d64-p4rt6_4cbff890-d6e7-4319-8a6d-7e650c313569 became leader"}
2020-10-18T13:26:58.739Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "mgmt-app.", "reconcilerKind": "MgmtApp", "controller": "mgmtapp", "source": "kind source: /, Kind="}
E1018 13:26:58.743329       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:26:59.786673       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:27:02.213555       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:27:08.019872       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:27:15.795812       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:28:07.215738       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:28:51.286639       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:29:29.780657       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:30:08.576422       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:30:58.953973       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:31:35.513270       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:32:11.611564       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:32:52.440143       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:33:48.317201       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:34:27.113410       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:35:06.029765       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:35:58.609352       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope
E1018 13:36:34.809457       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope

Obviously the permission issue above seems suspicious

Both are created in the default namespace

What did you expect to see?

I expect the operator to work on the GKE cluster as it works when being ran locally (against the same GKE cluster)

What did you see instead? Under which circumstances?

The above logs and the deployments/pods are not created

Environment

Operator type:

/language go

Kubernetes cluster type:

GKE

$ operator-sdk version

operator-sdk version: "v1.0.1", commit: "4169b318b578156ed56530f373d328276d040a1b", kubernetes version: "v1.18.2", go version: "go1.15.2 darwin/amd64", GOOS: "darwin", GOARCH: "amd64"

$ go version (if language is Go)

go version go1.13.15 darwin/amd64

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-gke.1504", GitCommit:"fb367de18820fe57bac5d97ad60b7a2cef8765e9", GitTreeState:"clean", BuildDate:"2020-09-09T01:00:23Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}

Possible Solution

Additional context

The markups added on the Reconsile function are:

// +kubebuilder:rbac:groups=mgmt-app,resources=mgmtapps,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=mgmt-app,resources=mgmtapps/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;

It seems that the operator should be able to list the delpoyment API

I also did kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user <my-user-name>
As mentioned in the tutorial to make sure my user is privileged

[zonnie]

Seems like I found the problem, can't really explain it though.
I followed the guild and modified the namespace in config/default/kustomization.yaml using

cd config/default/ && kustomize edit set namespace "default" && cd ../..

Before that it seems that running make manifests didn't change the role.yaml at all.
I have no clue how (or if) these are related but not the Deployments are "fetchable" by the operator.

Now however I'm spammed with these in the operator's log

E1019 08:39:15.130195       1 reflector.go:383] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to watch *v1.Pod: unknown (get pods)
E1019 08:40:11.103031       1 reflector.go:383] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to watch *v1.Pod: unknown (get pods)
E1019 08:40:56.830486       1 reflector.go:383] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to watch *v1.Pod: unknown (get pods)
E1019 08:41:45.059471       1 reflector.go:383] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to watch *v1.Pod: unknown (get pods)

My current role.yaml is

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: manager-role
rules:
- apiGroups:
  - apps
  resources:
  - deployments
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
- apiGroups:
  - mgmt-app
  resources:
  - mgmtapps
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - mgmt-app
  resources:
  - mgmtapps/status
  verbs:
  - get
  - patch
  - update

Which looks fine as far as Pods are concerned

[camilamacedo86]

The error faced is:

E1018 13:26:58.743329 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:default:default" cannot list resource "deployments" in API group "apps" at the cluster scope

which means that the user used to deploy/run the project has not the required permissions. You said that you did kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user <my-user-name>

However, you said as well:

When running locally with make run ENABLE_WEBHOOKS=false

See that the tutorial has no webhooks and you do not need to use ENABLE_WEBHOOKS=false. We have a PR to clarifies that, see: #4021.

Then, see that to run locally you need to:

• run make to call all targets to ensure that the manifests will be updated/generated properly
• runmake install to install the CRD's
• and then, run make run which will run the manager outside of the cluster.
• to check the project doing the reconcile, run kubectl apply -f config/samples/ to apply the samples.

Could you please re-check the steps and let us know?

[zonnie]

@camilamacedo86 As I said, the Deployment issue was some kind of "quirk" with the make manifest that just didn't re-create the role.yaml for some reason.
I managed to make it happen but now it has some kind of issue watching Pods

E1019 14:16:15.443229       1 reflector.go:383] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to watch *v1.Pod: unknown (get pods)
E1019 14:17:01.788900       1 reflector.go:383] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to watch *v1.Pod: unknown (get pods)
E1019 14:17:44.545198       1 reflector.go:383] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to watch *v1.Pod: unknown (get pods)
E1019 14:18:22.243119       1 reflector.go:383] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to watch *v1.Pod: unknown (get pods)

The role.yaml looks like this now:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: manager-role
rules:
- apiGroups:
  - apps
  resources:
  - deployments
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
- apiGroups:
  - mgmt-app
  resources:
  - mgmtapps
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - mgmt-app
  resources:
  - mgmtapps/status
  verbs:
  - get
  - patch
  - update

I re-built and re-deployed the image and still get the above log messages

[camilamacedo86]

Could you please try to add the following in your main.go and let us know if it solves your problem?

operator-sdk/cmd/operator-sdk/main.go

Lines 18 to 20 in 4054b5f

	// Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.)
	// to ensure that `exec-entrypoint` and `run` can make use of them.
	_ "k8s.io/client-go/plugin/pkg/client/auth"

[zonnie]

@camilamacedo86 I think there's a misunderstanding...my problems are when the operator is running inside the cluster.
I mentioned the local run only to say that I have no problems there.
In any case I tried your suggestion anyway and I'm still having the same issues watching Pods.

[camilamacedo86]

Hi @zonnie,

Could you share a link for your project or the controller implementation for we check what is missing on that? Are you using the make deploy IMG=<image> to deploy the project? Did you setup the watch such as the following example?

func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&cachev1alpha1.Memcached{}).
		Owns(&appsv1.Deployment{}).
		Complete(r)
}

[zonnie]

@camilamacedo86 - I can't share the repo but the controller implementation is this:

package controllers

import (
  "context"
  "github.com/go-logr/logr"
  mgmtappv1alpha1 "github.com.test.com/mgmt-app-operator/api/v1alpha1"
  "github.com.test.com/mgmt-app-operator/controllers/logic"
  appsv1 "k8s.io/api/apps/v1"
  corev1 "k8s.io/api/core/v1"
  "k8s.io/apimachinery/pkg/api/errors"
  "k8s.io/apimachinery/pkg/runtime"
  "k8s.io/apimachinery/pkg/types"
  "reflect"
  ctrl "sigs.k8s.io/controller-runtime"
  "sigs.k8s.io/controller-runtime/pkg/client"
)

// MgmtAppReconciler reconciles a MgmtApp object
type MgmtAppReconciler struct {
  client.Client
  Log    logr.Logger
  Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=mgmt-app.test.com,resources=mgmtapps,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=mgmt-app.test.com,resources=mgmtapps/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;

func (r *MgmtAppReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
  ctx := context.Background()
  log := r.Log.WithValues("mgmtapp", req.NamespacedName)

  // Fetch the MgmtApp instance
  mgmtapp := &mgmtappv1alpha1.MgmtApp{}
  err := r.Get(ctx, req.NamespacedName, mgmtapp)
  if err != nil {
    if errors.IsNotFound(err) {
      // Request object not found, could have been deleted after reconcile request.
      // Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
      // Return and don't requeue
      log.Info("MgmtApp resource not found. Ignoring since object must be deleted")
      return ctrl.Result{}, nil
    }
    // Error reading the object - requeue the request.
    log.Error(err, "Failed to get MgmtApp")
    return ctrl.Result{}, err
  }

  log.Info("Creating Mgmt App workers", "Amount of workers", len(mgmtapp.Spec.Workers))
  for _, worker := range mgmtapp.Spec.Workers {
    // Check if the deployment already exists, if not create a new one
    found := &appsv1.Deployment{}
    err = r.Get(ctx, types.NamespacedName{Name: worker.Name, Namespace: mgmtapp.Namespace}, found)

    // ************ CREATE ************ //

    if err != nil && errors.IsNotFound(err) {
      // Define a new deployment
      deployment, err := r.buildDeploymentForMgmtAppWorker(mgmtapp, &worker, req)
      if err != nil {
        log.Error(err, "Failed to create deployment for worker", "Worker.Name", worker.Name)
        continue
      }

      log.Info("Creating a new Deployment",
        "Deployment.Namespace", deployment.Namespace,
        "Deployment.Name", deployment.Name)

      err = r.Create(ctx, deployment)
      if err != nil {
        log.Error(err,
          "Failed to create new Deployment",
          "Deployment.Namespace", deployment.Namespace,
          "Deployment.Name", deployment.Name)
        return ctrl.Result{}, err
      }
      // Deployment created successfully - return and requeue
      return ctrl.Result{Requeue: true}, nil
    } else if err != nil {
      log.Error(err, "Failed to get Deployment")
      return ctrl.Result{}, err
    }

    // ************ UPDATE ************ //

    // Ensure the deployment size is the same as the spec
    size := worker.Size
    if *found.Spec.Replicas != size {
      found.Spec.Replicas = &size
      err = r.Update(ctx, found)
      if err != nil {
        log.Error(err,
          "Failed to update Deployment",
          "Deployment.Namespace", found.Namespace,
          "Deployment.Name", found.Name)
        return ctrl.Result{}, err
      }
      // Spec updated - return and requeue
      log.Info("Worker instance count updated",
        "Deployment.Name",
        found.Name, "existing",
        *found.Spec.Replicas, "new", size)
      return ctrl.Result{Requeue: true}, nil
    }

    // Update the MgmtApp status with the pod names
    // List the pods for this mgmtapp's deployment
    podList := &corev1.PodList{}
    matchingLabels := logic.GetMgmtAppLabels(mgmtapp.Name, mgmtapp.Name)
    listOpts := []client.ListOption{
      client.InNamespace(mgmtapp.Namespace),
      client.MatchingLabels(matchingLabels),
    }
    if err = r.List(ctx, podList, listOpts...); err != nil {
      log.Error(err,
        "Failed to list pods",
        "MgmtApp.Namespace",
        mgmtapp.Namespace, "MgmtApp.Name",
        mgmtapp.Name)
      return ctrl.Result{}, err
    }
    podNames := getPodNames(podList.Items)
    log.Info("Found following pods",
      "Deployment.Name", found.Name,
      "podNames", podNames,
      "labels", matchingLabels)

    // Update status.Nodes if needed
    if !reflect.DeepEqual(podNames, mgmtapp.Status.MgmtAppWorkerNames) {
      log.Info("Update deployment pod names",
        "existing pod names", mgmtapp.Status.MgmtAppWorkerNames,
        "updated pod names", podNames)
      mgmtapp.Status.MgmtAppWorkerNames = podNames
      err := r.Status().Update(ctx, mgmtapp)
      if err != nil {
        log.Error(err, "Failed to update MgmtApp status")
        // Reconcile failed due to error - requeue
        return ctrl.Result{}, err
      }
    }

  }

  // Reconcile successful - don't requeue
  return ctrl.Result{}, nil
}

// buildDeploymentForMgmtAppWorker returns a mgmtapp Deployment object
func (r *MgmtAppReconciler) buildDeploymentForMgmtAppWorker(
  m *mgmtappv1alpha1.MgmtApp,
  worker *mgmtappv1alpha1.MgmtAppWorker,
  req ctrl.Request) (*appsv1.Deployment, error) {
  log := r.Log.WithValues("mgmtapp", req.NamespacedName)

  deployment, err := logic.CreateAppDeployment(worker, m)
  if err != nil {
    log.Error(err, "Failed to create deployment", "Worker.Name", worker.Name)
    return nil, err
  }

  // Set MgmtApp instance as the owner and controller
  err = ctrl.SetControllerReference(m, deployment, r.Scheme)
  if err != nil {
    log.Error(err,
      "Failed to set ownership over controller",
      "MgmtApp.Namespace", m.Name)
    return nil, err
  }
  return deployment, nil
}

// getPodNames returns the pod names of the array of pods passed in
func getPodNames(pods []corev1.Pod) []string {
  var podNames []string
  for _, pod := range pods {
    podNames = append(podNames, pod.Name)
  }
  return podNames
}

func (r *MgmtAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
  return ctrl.NewControllerManagedBy(mgr).
    For(&mgmtappv1alpha1.MgmtApp{}).
    Owns(&appsv1.Deployment{}).
    Complete(r)
}

This is the logic package that is referred to above:

package logic

import (
  "fmt"
  mgmtappv1alpha1 "github.com/test.com/mgmt-app-operator/api/v1alpha1"
  appsv1 "k8s.io/api/apps/v1"
  corev1 "k8s.io/api/core/v1"
  "k8s.io/apimachinery/pkg/api/resource"
  metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
  "strings"
)

// GetMgmtAppLabels returns the labels for selecting the resources
// belonging to the given mgmtapp CR name.
func GetMgmtAppLabels(name string, CRDName string) map[string]string {
  return map[string]string{"app": name, "gc-mgmt-app_cr": CRDName}
}

// CreateAppDeployment returns a Guardicore application deployment
// it is responsible for building the k8s deployment from the given MgmtApp CRD
func CreateAppDeployment(
  worker *mgmtappv1alpha1.MgmtAppWorker,
  mgmtApp *mgmtappv1alpha1.MgmtApp) (*appsv1.Deployment, error) {

  labels := GetMgmtAppLabels(mgmtApp.Name, mgmtApp.Name)
  replicas := worker.Size
  resources, err := generateResources(worker)
  if err != nil {
    return nil, err
  }

  // Set MgmtApp instance as the owner and controller
  deployment := &appsv1.Deployment{
    ObjectMeta: metav1.ObjectMeta{
      Name:      worker.Name,
      Namespace: mgmtApp.Namespace,
    },
    Spec: appsv1.DeploymentSpec{
      Replicas: &replicas,
      Selector: &metav1.LabelSelector{
        MatchLabels: labels,
      },
      Template: corev1.PodTemplateSpec{
        ObjectMeta: metav1.ObjectMeta{
          Labels: labels,
        },
        Spec: corev1.PodSpec{
          Containers: []corev1.Container{{
            Image:     mgmtApp.Spec.GuardicoreTag,
            Name:      worker.Name,
            Command:   worker.Command,
            Args:      worker.Args,
            Env:       generateEnvVars(mgmtApp),
            Resources: resources},
          },
        },
      },
    },
  }

  return deployment, nil
}

func generateResources(worker *mgmtappv1alpha1.MgmtAppWorker) (corev1.ResourceRequirements, error) {
  CPULimit, err := resource.ParseQuantity(worker.Limits.CPU)
  if err != nil {
    return corev1.ResourceRequirements{}, err
  }
  CPURequest, err := resource.ParseQuantity(worker.Requests.CPU)
  if err != nil {
    return corev1.ResourceRequirements{}, err
  }

  MemoryMBLimit, err := resource.ParseQuantity(worker.Limits.MemoryMB)
  if err != nil {
    return corev1.ResourceRequirements{}, err
  }
  MemoryMBRequest, err := resource.ParseQuantity(worker.Requests.MemoryMB)
  if err != nil {
    return corev1.ResourceRequirements{}, err
  }

  limits := corev1.ResourceList{corev1.ResourceCPU: CPULimit, corev1.ResourceMemory: MemoryMBLimit}
  requests := corev1.ResourceList{corev1.ResourceCPU: CPURequest, corev1.ResourceMemory: MemoryMBRequest}
  return corev1.ResourceRequirements{Limits: limits, Requests: requests}, nil
}

func generateEnvVars(mgmtApp *mgmtappv1alpha1.MgmtApp) []corev1.EnvVar {
  envVars := []corev1.EnvVar{
    {
      Name:  "SAAS",
      Value: "true",
    },
    {
      Name: "NAMESPACE_NAME",
      ValueFrom: &corev1.EnvVarSource{
        FieldRef: &corev1.ObjectFieldSelector{
          FieldPath: "metadata.namespace",
        },
      },
    },
    {
      Name: "HOST_IP",
      ValueFrom: &corev1.EnvVarSource{
        FieldRef: &corev1.ObjectFieldSelector{
          FieldPath: "status.hostIP",
        },
      },
    },
    {
      Name: "SERVICE_POD_ID",
      ValueFrom: &corev1.EnvVarSource{
        FieldRef: &corev1.ObjectFieldSelector{
          FieldPath: "status.podIP",
        },
      },
    },
  }

  for _, redisServerName := range mgmtApp.Spec.RedisServerNames {
    redisEnvVarName := strings.ToUpper(redisServerName)
    redisEnvVarName = strings.Replace(redisEnvVarName, "-", "_", -1)
    envVars = append(envVars, corev1.EnvVar{
      Name:      fmt.Sprintf("GC_%s_DNS_URL", redisEnvVarName),
      Value:     fmt.Sprintf("%s-headless.%s.svc.cluster.local", redisServerName, mgmtApp.Namespace),
      ValueFrom: nil,
    })
  }

  envVars = append(envVars, corev1.EnvVar{
    Name:  "CONSUL_DNS",
    Value: fmt.Sprintf("%s.%s.svc.cluster.local", mgmtApp.Spec.ConsulName, mgmtApp.Spec.ConsulNamespace),
  })
  envVars = append(envVars, corev1.EnvVar{
    Name: "GC_ELASTICSEARCH_SERVICE_DNS",
    Value: fmt.Sprintf("%s-master-headless.%s.svc.cluster.local",
      mgmtApp.Spec.ElasticsearchName, mgmtApp.Namespace),
  })
  envVars = append(envVars, corev1.EnvVar{
    Name:  "GC_MONITORING_SERVICE_DNS",
    Value: "",
  })
  envVars = append(envVars, corev1.EnvVar{
    Name:  "GC_MONGODB_SERVICE_DNS",
    Value: fmt.Sprintf("%s.%s.svc.cluster.local", mgmtApp.Spec.MongoDBName, mgmtApp.Namespace),
  })
  envVars = append(envVars, corev1.EnvVar{
    Name:  "GC_RABBITMQ_SERVICE_DNS",
    Value: fmt.Sprintf("%s-headless.%s.svc.cluster.local", mgmtApp.Spec.RabbitMQName, mgmtApp.Namespace),
  })
  envVars = append(envVars, corev1.EnvVar{
    Name:  "GC_INFLUXDB_SERVICE_DNS",
    Value: "",
  })
  envVars = append(envVars, corev1.EnvVar{
    Name:  "GC_POSTGRES_SERVICE_DNS",
    Value: "",
  })

  return envVars
}

[camilamacedo86]

Hi @zonnie,

E1019 14:16:15.443229 1 reflector.go:383] pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125: Failed to watch *v1.Pod: unknown (get pods)

By looking at your roles it shows missing the Pod rules. See the https://github.com/operator-framework/operator-sdk/blob/master/testdata/go/memcached-operator/config/rbac/role.yaml#L41-L47 which is generated by the tool as well. I mean with make manifests.

Also, could you check your go.mod and Makefile files to see if they are using the same versions and have the same targets and etc? I am unable to reproduce your issue. Also, another try can be to bump your controller-runtime version to v0.6.3 since it has a bugfix for the watches feature. See: https://github.com/kubernetes-sigs/controller-runtime/releases/tag/v0.6.3.

[zonnie]

@camilamacedo86 my role.yaml (same one attached above) has these roles as far as I understand:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: manager-role
rules:
- apiGroups:
  - apps
  resources:
  - deployments
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
- apiGroups:
  - mgmt-app.guardicore.com
  resources:
  - mgmtapps
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - mgmt-app.guardicore.com
  resources:
  - mgmtapps/status
  verbs:
  - get
  - patch
  - update

I'm not really sure what versions you are referring to in the Makefile - this is my Makefile:

# Current Operator version
VERSION ?= 0.0.1
# Default bundle image tag
BUNDLE_IMG ?= controller-bundle:$(VERSION)
# Options for 'bundle-build'
ifneq ($(origin CHANNELS), undefined)
BUNDLE_CHANNELS := --channels=$(CHANNELS)
endif
ifneq ($(origin DEFAULT_CHANNEL), undefined)
BUNDLE_DEFAULT_CHANNEL := --default-channel=$(DEFAULT_CHANNEL)
endif
BUNDLE_METADATA_OPTS ?= $(BUNDLE_CHANNELS) $(BUNDLE_DEFAULT_CHANNEL)

# Image URL to use all building/pushing image targets
IMG ?= controller:latest
# Produce CRDs that work back to Kubernetes 1.11 (no version conversion)
CRD_OPTIONS ?= "crd:trivialVersions=true"

# Get the currently used golang install path (in GOPATH/bin, unless GOBIN is set)
ifeq (,$(shell go env GOBIN))
GOBIN=$(shell go env GOPATH)/bin
else
GOBIN=$(shell go env GOBIN)
endif

all: manager

# Run tests
ENVTEST_ASSETS_DIR=$(shell pwd)/testbin
test: generate fmt vet manifests
	mkdir -p ${ENVTEST_ASSETS_DIR}
	test -f ${ENVTEST_ASSETS_DIR}/setup-envtest.sh || curl -sSLo ${ENVTEST_ASSETS_DIR}/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/master/hack/setup-envtest.sh
	source ${ENVTEST_ASSETS_DIR}/setup-envtest.sh; fetch_envtest_tools $(ENVTEST_ASSETS_DIR); setup_envtest_env $(ENVTEST_ASSETS_DIR); go test ./... -coverprofile cover.out

# Build manager binary
manager: generate fmt vet
	go build -o bin/manager main.go

# Run against the configured Kubernetes cluster in ~/.kube/config
run: generate fmt vet manifests
	go run ./main.go

# Install CRDs into a cluster
install: manifests kustomize
	$(KUSTOMIZE) build config/crd | kubectl apply -f -

# Uninstall CRDs from a cluster
uninstall: manifests kustomize
	$(KUSTOMIZE) build config/crd | kubectl delete -f -

# Deploy controller in the configured Kubernetes cluster in ~/.kube/config
deploy: manifests kustomize
	cd config/manager && $(KUSTOMIZE) edit set image controller=${IMG}
	$(KUSTOMIZE) build config/default | kubectl apply -f -

# Generate manifests e.g. CRD, RBAC etc.
manifests: controller-gen
	$(CONTROLLER_GEN) $(CRD_OPTIONS) rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases

# Run go fmt against code
fmt:
	go fmt ./...

# Run go vet against code
vet:
	go vet ./...

# Generate code
generate: controller-gen
	$(CONTROLLER_GEN) object:headerFile="hack/boilerplate.go.txt" paths="./..."

# Build the docker image
docker-build: test
	docker build . -t ${IMG}

# Push the docker image
docker-push:
	docker push ${IMG}

# find or download controller-gen
# download controller-gen if necessary
controller-gen:
ifeq (, $(shell which controller-gen))
	@{ \
	set -e ;\
	CONTROLLER_GEN_TMP_DIR=$$(mktemp -d) ;\
	cd $$CONTROLLER_GEN_TMP_DIR ;\
	go mod init tmp ;\
	go get sigs.k8s.io/controller-tools/cmd/controller-gen@v0.3.0 ;\
	rm -rf $$CONTROLLER_GEN_TMP_DIR ;\
	}
CONTROLLER_GEN=$(GOBIN)/controller-gen
else
CONTROLLER_GEN=$(shell which controller-gen)
endif

kustomize:
ifeq (, $(shell which kustomize))
	@{ \
	set -e ;\
	KUSTOMIZE_GEN_TMP_DIR=$$(mktemp -d) ;\
	cd $$KUSTOMIZE_GEN_TMP_DIR ;\
	go mod init tmp ;\
	go get sigs.k8s.io/kustomize/kustomize/v3@v3.5.4 ;\
	rm -rf $$KUSTOMIZE_GEN_TMP_DIR ;\
	}
KUSTOMIZE=$(GOBIN)/kustomize
else
KUSTOMIZE=$(shell which kustomize)
endif

# Generate bundle manifests and metadata, then validate generated files.
.PHONY: bundle
bundle: manifests
	operator-sdk generate kustomize manifests -q
	cd config/manager && $(KUSTOMIZE) edit set image controller=$(IMG)
	$(KUSTOMIZE) build config/manifests | operator-sdk generate bundle -q --overwrite --version $(VERSION) $(BUNDLE_METADATA_OPTS)
	operator-sdk bundle validate ./bundle

# Build the bundle image.
.PHONY: bundle-build
bundle-build:
	docker build -f bundle.Dockerfile -t $(BUNDLE_IMG) .

This is my go.mod:

module github.com/guardicore.com/mgmt-app-operator

go 1.13

require (
	github.com/go-logr/logr v0.1.0
	github.com/onsi/ginkgo v1.14.2
	github.com/onsi/gomega v1.10.3
	k8s.io/api v0.18.6
	k8s.io/apimachinery v0.18.6
	k8s.io/client-go v0.18.6
	sigs.k8s.io/controller-runtime v0.6.2
)

I'll try to bump the controller-runtime as you suggested anyway