pkg/symbolizer: Improved caching in symbolization #2311
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Instead of indefinitely or caching liners via TTL, we take a few different approaches, that ultimately lower both the used memory as well as CPU. 1) The first time we see a piece of debuginfo we record whether it has `DWARF`, `.gopclntab`, `.symtab` and `.dynsym`. When the same piece of debuginfo is seen again the cached information is used. For introspection, we also save this as part of the debuginfo metadata. 2) The first time we see a piece of debuginfo we record its valid PC ranges. Using this we can filter out whether a location can even be possible to be symbolized based on the known total range of program counters in the debuginfo. 3) Liners are closed in subsequent symbolization rounds if their cached value is unused. Therefore liners that are constantly used are also kept alive and caches stay warm, but liners that are only used once in a while are opened and then closed again to not occupy unnecessary amounts of memory.
|
|
||
| // Fetch the debug info for the build ID. | ||
| rc, err := s.debuginfo.FetchDebuginfo(ctx, m.BuildId) | ||
| func (s *Symbolizer) symbolizeLocationsForMapping(ctx context.Context, m *pb.Mapping, locations []*pb.Location) ([][]profile.LocationLine, liner, error) { |
There was a problem hiding this comment.
Is it possible to split this method into two? One that returns a liner and another one consumes that an returns location lines?
| delete(s.linerCache, k) | ||
| } | ||
| for _, liner := range s.linerCache { | ||
| // These are liners that didn't show up in the latest iteration. |
There was a problem hiding this comment.
How do you feel about adding a metric to track the number of times we see the same build id? It could help us to fine-tune the lifecycle of a cached liner. I guess the cache is to correlate it with cache hits/misses and symbolization rounds.
There was a problem hiding this comment.
LGTM. 🎉 Glad to have .dynsym back 😄
Not a blocker at all but I'll be curious to see if some of the correctness problems we were seeing with elf symbolization before #1930 are also handled by this while keeping .dynsym. Will do some manual testing later this week.
For the moment it's just a check to see if it exists, using it for symbolization is not implemented yet. Thanks for the reviews! I'm going to go ahead and merge this as is and deploy it on the demo cluster to see if it has the effect that we want to achieve. If so I'll follow up by addressing the refactoring and adding more metrics. |
Instead of indefinitely or caching liners via TTL, we take a few different approaches, that ultimately lower both the used memory as well as CPU.
DWARF,.gopclntab,.symtaband.dynsym. When the same pieceof debuginfo is seen again the cached information is used. For
introspection, we also save this as part of the debuginfo metadata.
ranges. Using this we can filter out whether a location can even be
possible to be symbolized based on the known total range of program
counters in the debuginfo.
value is unused. Therefore liners that are constantly used are also
kept alive and caches stay warm, but liners that are only used once
in a while are opened and then closed again to not occupy unnecessary
amounts of memory.