Minidump stack walking does not use CFI data from debug symbols

[cpfair]

Minidumps are parsed at upload time, including extraction of the stack traces - however, this stack walking does not use any CFI records that may be present in the modules' previously uploaded symbol files. This causes problems on platforms like x86, where breakpad's naive stack-walking routine tries its best but ultimately falls short, producing inaccurate stack traces.

This appears to be a known issue, with a suggestion that minidumps could be reprocessed later, once CFI data was available. But, this isn't currently feasible given that the minidump is thrown away as soon as possible. It seems like the next best option would be to opportunistically check for any uploaded symbols w/ CFI at minidump upload time, using them if present, but otherwise continuing without.

[jan-auer]

Hi @cpfair, you're right. At the moment, we do not use CFI for stack walking and also throw minidumps away immediately. Using CFI at upload time is not feasible, as we cannot fetch the required symbols in our web process.

Instead, the plan is to move minidumps to a worker, where we can download symbols and compute CFI caches on the fly. The initial decision to throw away minidumps was made for both data protection and performance reasons. In the meanwhile, we've also received feature requests to persist minidumps - so this will likely become a project setting in the future. Most likely, the entire process will be:

1. Initially process the minidump without symbols and create the raw event
2. Reprocess the minidump with CFI on a worker and update stack traces
3. Optionally store the minidump for later downloads for the default event retention period
4. Symbolicate all frames with full debug information

Also, we currently do not create events when processing the minidump fails. We plan to fix this along the way, by creating a raw event without stack traces in this case and adding processing errors.

[asgoel]

@jan-auer any update on this?

[jan-auer]

@asgoel We’ve recently started working on using CFI and expect to deliver this feature soon. I will update this GH issue once it is available.

[rrueth]

Thanks for all of the hard work,@jan-auer! Now that all of the changes for this issue have been merged, when should we expect the changes to be deployed to sentry.io?

[jan-auer]

They are deploying at the moment. We will be updating documentation around CFI in the upcoming days.