Skip to content
This repository has been archived by the owner on Jul 19, 2023. It is now read-only.

StacktraceResolver returns a non-requested stacktraceID #794

Closed
cyriltovena opened this issue Jun 26, 2023 · 1 comment
Closed

StacktraceResolver returns a non-requested stacktraceID #794

cyriltovena opened this issue Jun 26, 2023 · 1 comment

Comments

@cyriltovena
Copy link
Collaborator

We're running #767 which yeld awesome result but I found one issue which seems very hard to repro



2023-06-26 06:01:45.206	/Users/kolesnikovae/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0xa52023-06-26 06:01:45.206created by golang.org/x/sync/errgroup.(*Group).Go2023-06-26 06:01:45.206	/Users/kolesnikovae/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x642023-06-26 06:01:45.206golang.org/x/sync/errgroup.(*Group).Go.func1()2023-06-26 06:01:45.206	/Users/kolesnikovae/src/grafana/phlare/pkg/util/recovery.go:56 +0x632023-06-26 06:01:45.206github.com/grafana/phlare/pkg/util.RecoverPanic.func1()2023-06-26 06:01:45.206	/Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/block_querier.go:663 +0xdf2023-06-26 06:01:45.206github.com/grafana/phlare/pkg/phlaredb.MergeProfilesStacktraces.func1()2023-06-26 06:01:45.206	/Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/head_queriers.go:120 +0x1b42023-06-26 06:01:45.206github.com/grafana/phlare/pkg/phlaredb.(*headOnDiskQuerier).MergeByStacktraces(0xc106b8f940, {0x4d5fd58?, 0xc0594ceff0?}, {0x4d60ed8, 0xc0fd43f1d0})2023-06-26 06:01:45.206	/Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/head.go:617 +0x3672023-06-26 06:01:45.206github.com/grafana/phlare/pkg/phlaredb.(*Head).resolveStacktraces(0xc08ba62000, {0x4d5fe00, 0xc0fd43f260}, 0xc053c5e6f0?)2023-06-26 06:01:45.206	/Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/sample_merge.go:432 +0x822023-06-26 06:01:45.206github.com/grafana/phlare/pkg/phlaredb.stacktracesByMapping.ForEach(0xc018cec000?, 0xc122bf7cb8)2023-06-26 06:01:45.206	/Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/head.go:626 +0x2fe2023-06-26 06:01:45.206github.com/grafana/phlare/pkg/phlaredb.(*Head).resolveStacktraces.func2(0xc122bf7b48?, 0xc0fc64e3c0)2023-06-26 06:01:45.206	/Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/symdb/mapping_memory.go:200 +0xa52023-06-26 06:01:45.206github.com/grafana/phlare/pkg/phlaredb/symdb.(*stacktraceResolverMemory).ResolveStacktraces(0xc1f7e37258, {0x0?, 0xc122bf7a58?}, {0x4d34940, 0xc0e2b548d0}, {0xc250c54000?, 0x1?, 0x0?})2023-06-26 06:01:45.206	/Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/symdb/mapping_memory.go:236 +0x1dc2023-06-26 06:01:45.206github.com/grafana/phlare/pkg/phlaredb/symdb.(*stacktraceResolverMemory).ResolveStacktracesChunk(0xc250c54000?, {0x4d34940, 0xc0e2b548d0}, {{0xc250c54000?, 0xc0fc64e3c0?, 0xc2fe79b0c8?}, 0x3947240?, 0x0?})2023-06-26 06:01:45.206	/Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/symdb/interfaces.go:69 +0x2d2023-06-26 06:01:45.206github.com/grafana/phlare/pkg/phlaredb/symdb.StacktraceInserterFn.InsertStacktrace(0x6e997c0?, 0x423001?, {0xc33878a000?, 0x3111267?, 0xc122bf7930?})2023-06-26 06:01:45.206	/Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/head.go:644 +0x1372023-06-26 06:01:45.206github.com/grafana/phlare/pkg/phlaredb.(*Head).resolveStacktraces.func2.1(0x42f5b800?, {0xc33878a000, 0x17, 0xc1c369cba0?})2023-06-26 06:01:45.206panic({0x39b8160, 0x6e8c480})2023-06-26 06:01:45.206	/Users/kolesnikovae/src/grafana/phlare/pkg/util/recovery.go:53 +0x312023-06-26 06:01:45.206github.com/grafana/phlare/pkg/util.RecoverPanic.func1.1()2023-06-26 06:01:45.206	/Users/kolesnikovae/src/grafana/phlare/pkg/util/recovery.go:41 +0x532023-06-26 06:01:45.206github.com/grafana/phlare/pkg/util.panicError({0x39b8160, 0x6e8c480})2023-06-26 06:01:45.206goroutine 286285938 [running]:2023-06-26 06:01:45.206panic: runtime error: invalid memory address or nil pointer dereference | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0xa5 |   |   |   | 2023-06-26 06:01:45.206 | created by golang.org/x/sync/errgroup.(*Group).Go |   |   |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x64 |   |   |   | 2023-06-26 06:01:45.206 | golang.org/x/sync/errgroup.(*Group).Go.func1() |   |   |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/util/recovery.go:56 +0x63 |   |   |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/util.RecoverPanic.func1() |   |   |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/block_querier.go:663 +0xdf |   |   |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb.MergeProfilesStacktraces.func1() |   |   |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/head_queriers.go:120 +0x1b4 |   |   |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb.(*headOnDiskQuerier).MergeByStacktraces(0xc106b8f940, {0x4d5fd58?, 0xc0594ceff0?}, {0x4d60ed8, 0xc0fd43f1d0}) |   |   |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/head.go:617 +0x367 |   |   |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb.(*Head).resolveStacktraces(0xc08ba62000, {0x4d5fe00, 0xc0fd43f260}, 0xc053c5e6f0?) |   |   |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/sample_merge.go:432 +0x82 |   |   |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb.stacktracesByMapping.ForEach(0xc018cec000?, 0xc122bf7cb8) |   |   |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/head.go:626 +0x2fe |   |   |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb.(*Head).resolveStacktraces.func2(0xc122bf7b48?, 0xc0fc64e3c0) |   |   |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/symdb/mapping_memory.go:200 +0xa5 |   |   |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb/symdb.(*stacktraceResolverMemory).ResolveStacktraces(0xc1f7e37258, {0x0?, 0xc122bf7a58?}, {0x4d34940, 0xc0e2b548d0}, {0xc250c54000?, 0x1?, 0x0?}) |   |   |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/symdb/mapping_memory.go:236 +0x1dc |   |   |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb/symdb.(*stacktraceResolverMemory).ResolveStacktracesChunk(0xc250c54000?, {0x4d34940, 0xc0e2b548d0}, {{0xc250c54000?, 0xc0fc64e3c0?, 0xc2fe79b0c8?}, 0x3947240?, 0x0?}) |   |   |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/symdb/interfaces.go:69 +0x2d |   |   |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb/symdb.StacktraceInserterFn.InsertStacktrace(0x6e997c0?, 0x423001?, {0xc33878a000?, 0x3111267?, 0xc122bf7930?}) |   |   |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/head.go:644 +0x137 |   |   |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb.(*Head).resolveStacktraces.func2.1(0x42f5b800?, {0xc33878a000, 0x17, 0xc1c369cba0?}) |   |   |   | 2023-06-26 06:01:45.206 | panic({0x39b8160, 0x6e8c480}) |   |   |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/util/recovery.go:53 +0x31 |   |   |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/util.RecoverPanic.func1.1() |   |   |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/util/recovery.go:41 +0x53 |   |   |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/util.panicError({0x39b8160, 0x6e8c480}) |   |   |   | 2023-06-26 06:01:45.206 | goroutine 286285938 [running]: |   |   |   | 2023-06-26 06:01:45.206 | panic: runtime error: invalid memory address or nil pointer dereference |  
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
2023-06-26 06:01:45.206 | /Users/kolesnikovae/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0xa5 |  
  |   | 2023-06-26 06:01:45.206 | created by golang.org/x/sync/errgroup.(*Group).Go |  
  |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x64 |  
  |   | 2023-06-26 06:01:45.206 | golang.org/x/sync/errgroup.(*Group).Go.func1() |  
  |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/util/recovery.go:56 +0x63 |  
  |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/util.RecoverPanic.func1() |  
  |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/block_querier.go:663 +0xdf |  
  |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb.MergeProfilesStacktraces.func1() |  
  |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/head_queriers.go:120 +0x1b4 |  
  |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb.(*headOnDiskQuerier).MergeByStacktraces(0xc106b8f940, {0x4d5fd58?, 0xc0594ceff0?}, {0x4d60ed8, 0xc0fd43f1d0}) |  
  |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/head.go:617 +0x367 |  
  |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb.(*Head).resolveStacktraces(0xc08ba62000, {0x4d5fe00, 0xc0fd43f260}, 0xc053c5e6f0?) |  
  |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/sample_merge.go:432 +0x82 |  
  |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb.stacktracesByMapping.ForEach(0xc018cec000?, 0xc122bf7cb8) |  
  |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/head.go:626 +0x2fe |  
  |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb.(*Head).resolveStacktraces.func2(0xc122bf7b48?, 0xc0fc64e3c0) |  
  |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/symdb/mapping_memory.go:200 +0xa5 |  
  |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb/symdb.(*stacktraceResolverMemory).ResolveStacktraces(0xc1f7e37258, {0x0?, 0xc122bf7a58?}, {0x4d34940, 0xc0e2b548d0}, {0xc250c54000?, 0x1?, 0x0?}) |  
  |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/symdb/mapping_memory.go:236 +0x1dc |  
  |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb/symdb.(*stacktraceResolverMemory).ResolveStacktracesChunk(0xc250c54000?, {0x4d34940, 0xc0e2b548d0}, {{0xc250c54000?, 0xc0fc64e3c0?, 0xc2fe79b0c8?}, 0x3947240?, 0x0?}) |  
  |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/symdb/interfaces.go:69 +0x2d |  
  |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb/symdb.StacktraceInserterFn.InsertStacktrace(0x6e997c0?, 0x423001?, {0xc33878a000?, 0x3111267?, 0xc122bf7930?}) |  
  |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/phlaredb/head.go:644 +0x137 |  
  |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/phlaredb.(*Head).resolveStacktraces.func2.1(0x42f5b800?, {0xc33878a000, 0x17, 0xc1c369cba0?}) |  
  |   | 2023-06-26 06:01:45.206 | panic({0x39b8160, 0x6e8c480}) |  
  |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/util/recovery.go:53 +0x31 |  
  |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/util.RecoverPanic.func1.1() |  
  |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/src/grafana/phlare/pkg/util/recovery.go:41 +0x53 |  
  |   | 2023-06-26 06:01:45.206 | github.com/grafana/phlare/pkg/util.panicError({0x39b8160, 0x6e8c480}) |  
  |   | 2023-06-26 06:01:45.206 | goroutine 286285938 [running]: |  
  |   | 2023-06-26 06:01:45.206 | panic: runtime error: invalid memory address or nil pointer dereference |  

2023-06-26 06:01:45.206	/Users/kolesnikovae/.gvm/gos/go1.20rc3/src/runtime/panic.go:884 +0x213 |   |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/.gvm/gos/go1.20rc3/src/runtime/panic.go:884 +0x213
  |   | 2023-06-26 06:01:45.206 | /Users/kolesnikovae/.gvm/gos/go1.20rc3/src/runtime/panic.go:884 +0x213

This seems to happens when resolving stacktraces ID in memory, we return a unexpected stacktraceID causing a nil map entry here https://github.com/grafana/phlare/blob/feat/symdb/pkg/phlaredb/head.go#L644
@cyriltovena
Copy link
Collaborator Author

I believe the issue was because we were not sorting stacktraces IDs in the head before using the resolver. See 819f6e9

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cyriltovena