Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove caching objects to prevent leak #2966

Merged
merged 1 commit into from
Jun 7, 2022
Merged

Remove caching objects to prevent leak #2966

merged 1 commit into from
Jun 7, 2022

Conversation

jbreiding
Copy link
Contributor

What changed?
Remove caching recording functions

Why?
Fixing memory leak

How did you test it?
locally, pprof heap dump with running canary

Potential risks
None

Is hotfix candidate?
No

@jbreiding jbreiding requested a review from a team as a code owner June 7, 2022 18:20
common/metrics/otel_event_handler.go Outdated Show resolved Hide resolved
excludeTags: configExcludeTags(cfg),
}
}

func (m *OtelMetricHandler) Event(ctx context.Context, e *event.Event) context.Context {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you provide some context on why the pointer is not necessary here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nothing is changing on the receiver and does not need to be a pointer.

func (m *OtelMetricHandler) getRecordFunc(em event.Metric) recordFunc {
m.mu.Lock()
defer m.mu.Unlock()
if f, ok := m.recordFuncs[em]; ok {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so event.Metric is unbounded?

And instead of caching recordFuns, we create a new one every time? Will it impact latency or gc?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The difference is the GC is now collecting what was leaking before.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure I understand the unbounded, the objects needed in the function are captured, em is only needed for logging before returning recording function

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the events coming as parameters to the Event function are pooled, it is not unbounded.

@jbreiding jbreiding merged commit d3dc17e into temporalio:master Jun 7, 2022
@jbreiding jbreiding deleted the removing-cache branch June 7, 2022 21:03
Sushisource pushed a commit to Sushisource/temporal that referenced this pull request Jun 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants