Multiple Reconcile() invocations on single object creation

[champak]

Hi folks,
I am experimenting with an operator that creates Pods via the client .Create() api (based on processing YAML on a CRD) and listens on Pod resources. I am getting more than 1 Reconcile() invocation for a single pod instantiation. Based on examples I used the Owns api to "subscribe" to CRUD updates on the PODs that are generated

func (r *reconciler_foo) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&foo.bar{}).
		Owns(&corev1.Pod{}).
		Complete(r)
}

Also I had used the .SetControllerReference() for associating the pod definition to the scheme based on the pattern in the example.

All of the Reconcile invocations have a stack that indicates they are triggered from

....
#3  0x0000000001510d7d in sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker (c=0xc000340000)
    at <...>go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0-beta.4/pkg/internal/controller/controller.go:171

which is the c.processNextWorkItem() loop. Not sure what causes > 1 work item to get queued on a single pod creation and if that is the expected behavior. I was expecting to see just 1 invocation after the new Pod object is minted. Would love to hear from anyone that may have seen this type of behavior. Thanks.

[DirectXMan12]

Are you sure it's actually for a single instantiation? Many things in kubernetes could be touching a pod -- it's likely that you're seeing the effects of your own pod creation, the scheduler, and the kubelet in rapid succession.

Can you confirm that they're all the same? For instance, if you just run against pkg/envtest.Environment in CR, which only sets up an API server, do you see the same behavior?

[champak]

@DirectXMan12 thanks for the input. The updates that show up all seem to be the result of a single Pod instantiation. I know for sure there was a single Pod instantiation but like you mentioned there are probably updates from the k8s infra that I do not have visibility into. I was wondering whether I need to cache the prior "Spec" and diff with what shows up in order to filter updates I do not care about. Is there an established pattern for this type of filtering ? Also is there any trace in K8 on what all entities touch a given resource ? I am yet to check with the pkg/envtest.Environment. Thanks !

[DirectXMan12]

Also is there any trace in K8 on what all entities touch a given resource

Not really. You can check the API server logs to see this, to a certain extent, or use the audit logs.

know for sure there was a single Pod instantiation but like you mentioned there are probably updates from the k8s infra that I do not have visibility into

Yeah, a "pod instantiation" isn't really a single thing in kubernetes -- it's a whole cluster of updates.

I was wondering whether I need to cache the prior "Spec" and diff with what shows up in order to filter updates I do not care about. Is there an established pattern for this type of filtering

It's not entirely clear what you're looking to accomplish here. At best, you'll get different behavior on controller startup vs updates, which is generally a recipe for bugs. Can you clarify why you're interested in doing this?

[DirectXMan12]

(P.S. I'm closing this for bookkeeping purposes, but feel free to continue replying)

[champak]

Thanks for the pointers. I am specifying types K8 resources such as Pods and Services inside a custom resource definition and then instantiating the applicable resources via .Create(context, resource-def). This is so that most of the attributes to be specified for the application can be located under the custom resource definition. I set reference for the operator with the .SetControllerReference() API ; am hoping this approach is not an anti-pattern.

Resource Creation on this path resulted in the Reconcile() getting invoked more than once for a single resource (e.g. Pod) creation. That is what I was asking about. It sounds like I am seeing expected behavior with a cluster of updates from the various infra components that update the Pod object and I will just need to ignore the updates I do not care about.

[DirectXMan12]

Yep, what your seeing is more-or-less expected behavior, afaict

[akashjain971]

@champak you can leverage Predicates and specifically GenerationChangePredicate.

[rufreakde]

@champak you can leverage Predicates and specifically GenerationChangePredicate.

@akashjain971 Any example how to configure this correctly with kubebuilder? To not overwrite this when rebuilding.

Okay I found the section:

// SetupWithManager sets up the controller with the Manager.
func (r *<YOURRECONCILER>) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&v1alpha1.YOURRECONCILER{}).
		WithEventFilter(predicate.GenerationChangedPredicate{}).
		Complete(r)
}