Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

trace: unbounded trace channel can cause out-of-memory crashes #197

Closed
kevinlebrun opened this issue Mar 8, 2018 · 2 comments
Closed

trace: unbounded trace channel can cause out-of-memory crashes #197

kevinlebrun opened this issue Mar 8, 2018 · 2 comments
Assignees
Labels
core enhancement quick change/addition that does not need full team approval
Milestone

Comments

@kevinlebrun
Copy link

It was actually seen in production, with large traces (around 2 MiB but it is actually an issue too) and heavy load, the channel got filled until the host memory became exhausted.

The channel can take 1000 traces and force flushes when it is half-filled.

https://github.com/DataDog/dd-trace-go/blob/master/tracer/channels.go#L8

We need to ensure two properties:

  • the tracer memory usage should remain low (defining a SLA on this one would be great)
  • the flush payload should not reach 10 MiB which is the maximum input size accepted by the agent
@gbbr
Copy link
Contributor

gbbr commented Mar 8, 2018

This sounds good, we need to think about a good approach to this and see if it's reasonable from a performance point of view too.

@gbbr gbbr self-assigned this Apr 3, 2018
@gbbr gbbr added core bug unintended behavior that has to be fixed enhancement quick change/addition that does not need full team approval and removed bug unintended behavior that has to be fixed labels Apr 3, 2018
@palazzem palazzem added this to the 0.7.0 milestone Apr 9, 2018
@gbbr
Copy link
Contributor

gbbr commented Apr 16, 2018

Resolved in #206

@gbbr gbbr closed this as completed Apr 16, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core enhancement quick change/addition that does not need full team approval
Projects
None yet
Development

No branches or pull requests

3 participants