Logging the reason for a reconcile

[austince]

Hey there, is there any mechanism to log which event caused a reconcile request, or generally some other tooling to figure this out?

We have a controller that owns some resources directly, references others it does not own, and has a custom source for external triggers. All these different sources make it difficult to understand why a given reconcile request was triggered, especially in cases when trying to debug over-active reconcilers.

[alvaroaleman]

Not really, the problem is that the reconcile.Request get put into a workqueue that does de-duplication, so custom information like that can not be preserved (or at least not with the current workqueue implementation).

The best you can do is add logging to your eventsources and/or handlers.

[austince]

That makes sense, thanks for the reply. I think adding logging to the handlers seems like a good approach.

One complication I could see is that the builder doesn't expose ways to construct the handlers unless you use the lower-level Watches(..) method (i.e., not Owns(..) or For(..)).

For

controller-runtime/pkg/builder/controller.go

Lines 222 to 227 in b93b5f9

	typeForSrc, err := blder.project(blder.forInput.object, blder.forInput.objectProjection)
	if err != nil {
	return err
	}
	src := &source.Kind{Type: typeForSrc}
	hdler := &handler.EnqueueRequestForObject{}

Owns

controller-runtime/pkg/builder/controller.go

Lines 240 to 243 in b93b5f9

	hdler := &handler.EnqueueRequestForOwner{
	OwnerType: blder.forInput.object,
	IsController: true,
	}

What do you think about adding an option to modify the handler that is used?

I could see something like:

func (b *Builder) WithHandlerWrapper(func(handler.EventHandler) handler.EventHandler) *Builder { 
// ...
}

var _ ForOption = &HandlerWrapper{}
var _ OwnsOption = &HandlerWrapper{}

Then we could wrap the handler with a handler that just logs and delegates.

What do you think?

[austince]

Hey @alvaroaleman , hope you don't mind the ping. Just wondering if you have thoughts on this proposal / would be able to review a PR in that direction.

[alvaroaleman]

Just use watches and wrap the handler there, no reason to change the builder

[austince]

I think that's a decent workaround but forces us to reimplement the special setup for Owns and For. Do you have more concerns other than it's more to maintain?

[anderssonw]

Hey @austince, have you gotten a chance to look more into this? We are currently in the same boat and want to see why some of our reconciliation loops start in the first place, due to some odd inconsistencies.

If you have any insight I would be glad to take it :)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[anderssonw]

Bump @austince :)

[anderssonw]

/remove-lifecycle stale

[austince]

This is the recommended approach: #1997 (comment)

[anderssonw]

Hmm, I'll peep once i get time to prioritise it :)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale