Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

logging: do not swallow subsystem logs #19188

Merged

Conversation

ldelossa
Copy link
Contributor

commit #16861 introduced a normalization of error handling into the
daemon/cmd package.

by doing so we swallowed useful error logs.

this commit adds the error logs back and adds a few additional
fmt.Errorf wrappers where logging is not adequate

Signed-off-by: Louis DeLosSantos louis.delos@isovalent.com

Removes any log swallowing that was occuring on daemon/cmd init

@ldelossa ldelossa requested review from a team and nathanjsweet March 18, 2022 13:05
@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Mar 18, 2022
@ldelossa ldelossa added the release-note/misc This PR makes changes that have no direct user impact. label Mar 18, 2022
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Mar 18, 2022
Copy link
Member

@joestringer joestringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High level, if we can catch some of those missing log cases, this is in an improvement and I'm all for it. I've dropped a few notes below just since I wasn't entirely clear on which cases should be using error wrapping vs. logging directly and whether you had an idea in mind about which to use depending on where in the calltree the code is.

daemon/cmd/daemon.go Show resolved Hide resolved
daemon/cmd/daemon.go Show resolved Hide resolved
daemon/cmd/daemon.go Outdated Show resolved Hide resolved
@maintainer-s-little-helper maintainer-s-little-helper bot added this to Needs backport from master in 1.11.3 Mar 18, 2022
@joestringer
Copy link
Member

Given the potential improvements to debuggability, I'm nominating this for 1.11 backport.

@@ -242,7 +242,7 @@ func (d *Daemon) init() error {
}

if err := d.Datapath().Loader().Reinitialize(d.ctx, d, d.mtuConfig.GetDeviceMTU(), d.Datapath(), d.l7Proxy); err != nil {
return err
return fmt.Errorf("failed while reinitializing datapath: %w", err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^^ This is good, though for something like Reinitialize we may want to do another round of review inside there to evaluate the logging for each failure case and potentially disambiguate them, maybe even push the logging down if that allows the child function here to provide additional context for debugging.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joestringer it appears that the Reinitialize method handles logging internally.

	// Lock so that endpoints cannot be built while we are compile base programs.
	o.GetCompilationLock().Lock()
	defer o.GetCompilationLock().Unlock()
	defer func() { firstInitialization = false }()

	l.init(o.Datapath(), o.LocalConfig())

	if err := l.writeNetdevHeader("./", o); err != nil {
		log.WithError(err).Warn("Unable to write netdev header")
		return err
	}

for example.

So in our logs, Reinitialize will display a fair amount of log information before we pop out the stacked error here on line 245.

@aanm aanm added this to Needs backport from master in 1.11.4 Mar 26, 2022
@aanm aanm removed this from Needs backport from master in 1.11.3 Mar 26, 2022
commit cilium#16861 introduced a normalization of error handling into the
daemon/cmd package.

by doing so we swallowed useful error logs.

this commit adds the error logs back and adds a few additional
fmt.Errorf wrappers where logging is not adequate

Signed-off-by: Louis DeLosSantos <louis.delos@isovalent.com>
@ldelossa ldelossa force-pushed the ldelossa/dont-swallow-init-errors branch from d7eec02 to 0abd290 Compare March 30, 2022 13:36
@ldelossa
Copy link
Contributor Author

alright @joestringer I think this one is ready to go.

@ldelossa
Copy link
Contributor Author

/test

@joestringer joestringer added this to Needs backport from master in 1.11.5 Apr 15, 2022
@joestringer joestringer removed this from Needs backport from master in 1.11.4 Apr 15, 2022
@github-actions
Copy link

This pull request has been automatically marked as stale because it
has not had recent activity. It will be closed if no further activity
occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Apr 30, 2022
Copy link
Member

@joestringer joestringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, seems like I completely dropped the ball on this one. LGTM.

@joestringer joestringer removed the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Apr 30, 2022
@joestringer joestringer merged commit 6154322 into cilium:master Apr 30, 2022
@aditighag aditighag added backport-pending/1.11 backport-done/1.11 The backport for Cilium 1.11.x for this PR is done. and removed needs-backport/1.11 labels May 2, 2022
@aanm aanm moved this from Needs backport from master to Backport done to v1.11 in 1.11.5 May 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-done/1.11 The backport for Cilium 1.11.x for this PR is done. release-note/misc This PR makes changes that have no direct user impact.
Projects
No open projects
1.11.5
Backport done to v1.11
Development

Successfully merging this pull request may close these issues.

None yet

4 participants