Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Remove time.After from any Loops #14265

Merged
merged 1 commit into from
Dec 11, 2020

Conversation

nathanjsweet
Copy link
Member

@nathanjsweet nathanjsweet commented Dec 3, 2020

time.After is not garbage collected until after
its channel fires once. In situations where it is called
repeatedely this can lead to significant memory build up.
Instead of creating a new timer object for every iteration,
a single timer, with a hard-reset mechanism, is used in
situations where time.After is being called repeatedely.

Signed-off-by: Nate Sweet nathanjsweet@pm.me

Fixes: #14219

@nathanjsweet nathanjsweet added area/daemon Impacts operation of the Cilium daemon. area/operator Impacts the cilium-operator component release-note/misc This PR makes changes that have no direct user impact. sig/hubble Impacts hubble server or relay labels Dec 3, 2020
@maintainer-s-little-helper maintainer-s-little-helper bot added this to In progress in 1.10.0 Dec 3, 2020
@nathanjsweet nathanjsweet force-pushed the pr/nathanjsweet/remove-time-after branch from 49f394d to d751950 Compare December 3, 2020 20:27
@brb
Copy link
Member

brb commented Dec 4, 2020

Could we add a link to the article in the commit msg about time.After not being GC'd? It will help the future's readers.

@nathanjsweet
Copy link
Member Author

test-me-please

Comment on lines 25 to 64
func New() (*IncTimer, func() bool) {
t := time.NewTimer(time.Nanosecond)
return &IncTimer{
t: t,
}, t.Stop
}

// After returns a channel that will fire after
// the specified duration.
func (it *IncTimer) After(d time.Duration) <-chan time.Time {
// We cannot call reset on an expired timer,
// so we need to stop it and drain it first.
// See https://golang.org/pkg/time/#Timer.Reset for more details.
if !it.t.Stop() {
// It could be that the channel was read already
select {
case <-it.t.C:
default:
}
}
it.t.Reset(d)
return it.t.C
}
Copy link
Member

@aditighag aditighag Dec 4, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why we need a wrapper. Why isn't the in-place fix mentioned in the medium article sufficient?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The solution suggested in the article is actually partially incorrect. In order to reset a timer safely (as suggested in the article), you have to also stop the timer and clear the channel.
I think this wrapper is useful because it basically boils the following code:

t := time.NewTimer(time.Nanosecond)
defer t.Stop()
for {
...
        if !t.Stop() {
		// It could be that the channel was read already
		select {
		case <-t.C:
		default:
		}
	}
        t.Reset(someInterval)
        select {
        ...
        case <- t.C:
        }
}

to:

t, stop := inctimer.New()
defer stop()
for {
...
        select {
        ...
        case <- t.After(someInterval):
        }
}

In one case I believe the After function is called even more than once, so that would have either required the creation of two separate timers or the same hard reset logic repeated twice above the select.
I find this wrapper approach to be a lot more elegant and easier to use.

Copy link
Member

@christarazi christarazi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks good, but I'm also wondering the same as Aditi. Do we need a wrapper? If so, that would be helpful context in the commit msg and in the godoc of the inctimer package itself, for why it's necessary.

Comment on lines 25 to 64
func New() (*IncTimer, func() bool) {
t := time.NewTimer(time.Nanosecond)
return &IncTimer{
t: t,
}, t.Stop
}

// After returns a channel that will fire after
// the specified duration.
func (it *IncTimer) After(d time.Duration) <-chan time.Time {
// We cannot call reset on an expired timer,
// so we need to stop it and drain it first.
// See https://golang.org/pkg/time/#Timer.Reset for more details.
if !it.t.Stop() {
// It could be that the channel was read already
select {
case <-it.t.C:
default:
}
}
it.t.Reset(d)
return it.t.C
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@nathanjsweet nathanjsweet force-pushed the pr/nathanjsweet/remove-time-after branch from ac98143 to f2b0cc7 Compare December 7, 2020 18:50
Copy link
Member

@christarazi christarazi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, looks great! Thanks for adding the extra context. I think it makes sense to have the wrapper.

Thinking a bit into the future, I wonder how we can prevent uses of time.After without requiring code reviewers to remember. Help me understand, should we be replacing all uses of time.After with IncTimer going forward? If so, then we could do similar to what we have for lock.Mutex where we don't allow the use of the upstream sync.Mutex via a script (contrib/scripts/lock-check.sh) and this is enforced via our Golang GH Actions job.

@nathanjsweet nathanjsweet force-pushed the pr/nathanjsweet/remove-time-after branch 2 times, most recently from fc9d0db to ed93de6 Compare December 9, 2020 04:05
@nathanjsweet nathanjsweet marked this pull request as draft December 9, 2020 04:05
@nathanjsweet nathanjsweet force-pushed the pr/nathanjsweet/remove-time-after branch 3 times, most recently from 53eb47a to afccab3 Compare December 9, 2020 04:33
Copy link
Member

@fristonio fristonio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks 🚀 Just one question/suggestion, does not need a second review.
Really cool work on the customvet check 🎉

clustermesh-apiserver/main.go Outdated Show resolved Hide resolved
@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label Dec 9, 2020
@cilium cilium deleted a comment from maintainer-s-little-helper bot Dec 9, 2020
@nathanjsweet nathanjsweet force-pushed the pr/nathanjsweet/remove-time-after branch from 32b4dc2 to af80505 Compare December 9, 2020 15:52
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label Dec 9, 2020
`time.After` is not garbage collected until after
its channel fires once. In situations where it is called
repeatedely this *can* lead to significant memory build up.
Instead of creating a new timer object for every iteration,
a single timer, with a hard-reset mechanism, is used in
situations where `time.After` is being called repeatedely.

See the following article for details:
https://medium.com/@oboturov/golang-time-after-is-not-garbage-collected-4cbc94740082

pkg: Add inctimer

The new inctimer package is useful for boiling down the monotonous
taks of "hard" resetting the reused timer. In order to reuse a timer
to solve the `time.After` garbage collection problem the timer must
be stopped, have its channel cleared (or not, if it was already read),
and then reset. Duplicating this code all over the cilium code base
seems superfluous, a small package that mimics the more terse
`time.After` method seems more desireable.

Signed-off-by: Nate Sweet <nathanjsweet@pm.me>
@nathanjsweet nathanjsweet force-pushed the pr/nathanjsweet/remove-time-after branch from af80505 to 16d810e Compare December 9, 2020 22:59
@nathanjsweet
Copy link
Member Author

test-me-please

@nathanjsweet nathanjsweet marked this pull request as ready for review December 10, 2020 02:53
@nathanjsweet nathanjsweet requested a review from a team as a code owner December 10, 2020 02:53
Copy link
Contributor

@kkourt kkourt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@joestringer joestringer removed their assignment Dec 10, 2020
@jrajahalme jrajahalme merged commit 7d39243 into master Dec 11, 2020
@jrajahalme jrajahalme deleted the pr/nathanjsweet/remove-time-after branch December 11, 2020 04:39
@tklauser
Copy link
Member

This seems to have broken CI, see e.g. https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/6273

10:58:36      runtime: contrib/scripts/custom-vet-check.sh
10:58:36      runtime: # cd .; git clone -- https://github.com/cilium/customvet /home/vagrant/go/src/github.com/cilium/customvet
10:58:36      runtime: fatal: could not create work tree dir '/home/vagrant/go/src/github.com/cilium/customvet': Permission denied
10:58:36      runtime: package github.com/cilium/customvet: exit status 128
10:58:41      runtime: flag provided but not defined: -timeafter.ignore
10:58:41      runtime: usage: go vet [-n] [-x] [-vettool prog] [build flags] [vet flags] [packages]
10:58:41      runtime: Run 'go help vet' for details.
10:58:41      runtime: Run 'go tool vet -help' for the vet tool's flags.
10:58:41      runtime: Makefile:506: recipe for target 'precheck' failed
10:58:41      runtime: make: *** [precheck] Error 2

Looks like the CI runs failed on this PR already 😞

tklauser added a commit that referenced this pull request Dec 11, 2020
This reverts commit 7d39243 / PR #14265

Reason for revert: This broke the build in the master pipeline, e.g.
https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/6273/

  10:58:36      runtime: contrib/scripts/custom-vet-check.sh
  10:58:36      runtime: # cd .; git clone -- https://github.com/cilium/customvet /home/vagrant/go/src/github.com/cilium/customvet
  10:58:36      runtime: fatal: could not create work tree dir '/home/vagrant/go/src/github.com/cilium/customvet': Permission denied
  10:58:36      runtime: package github.com/cilium/customvet: exit status 128
  10:58:41      runtime: flag provided but not defined: -timeafter.ignore
  10:58:41      runtime: usage: go vet [-n] [-x] [-vettool prog] [build flags] [vet flags] [packages]
  10:58:41      runtime: Run 'go help vet' for details.
  10:58:41      runtime: Run 'go tool vet -help' for the vet tool's flags.
  10:58:41      runtime: Makefile:506: recipe for target 'precheck' failed
  10:58:41      runtime: make: *** [precheck] Error 2

Apparently this failure was already present on the CI runs on PR #14265
itself.

Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
@tklauser
Copy link
Member

I've sent a revert for this to unbreak CI: #14371

aanm pushed a commit that referenced this pull request Dec 11, 2020
This reverts commit 7d39243 / PR #14265

Reason for revert: This broke the build in the master pipeline, e.g.
https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/6273/

  10:58:36      runtime: contrib/scripts/custom-vet-check.sh
  10:58:36      runtime: # cd .; git clone -- https://github.com/cilium/customvet /home/vagrant/go/src/github.com/cilium/customvet
  10:58:36      runtime: fatal: could not create work tree dir '/home/vagrant/go/src/github.com/cilium/customvet': Permission denied
  10:58:36      runtime: package github.com/cilium/customvet: exit status 128
  10:58:41      runtime: flag provided but not defined: -timeafter.ignore
  10:58:41      runtime: usage: go vet [-n] [-x] [-vettool prog] [build flags] [vet flags] [packages]
  10:58:41      runtime: Run 'go help vet' for details.
  10:58:41      runtime: Run 'go tool vet -help' for the vet tool's flags.
  10:58:41      runtime: Makefile:506: recipe for target 'precheck' failed
  10:58:41      runtime: make: *** [precheck] Error 2

Apparently this failure was already present on the CI runs on PR #14265
itself.

Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
@nathanjsweet nathanjsweet restored the pr/nathanjsweet/remove-time-after branch December 11, 2020 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/daemon Impacts operation of the Cilium daemon. area/operator Impacts the cilium-operator component release-note/misc This PR makes changes that have no direct user impact. sig/hubble Impacts hubble server or relay
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

Remove time.After usages