ES crash on Sentry

[gergelycsernak]

I have a sentry event related to vSMR:
It happens when the user is closing an ASR:

• PublishAsrToBeSaved is called,
• the save changes modal dialog is displayed
• ES ticks the next second to update the screen and calls PublishUpdateScreen
• Then vSMR crashes with the following callstack
  vSMR +0x00a607
  vSMR +0x0092a3
  vSMR +0x038ee2
  EuroScope +0x0ffeda CPlugInData::PublishUpdateScreen (PlugInHandler.cpp:504)

[Epse]

@gergelycsernak you wouldn't happen to have any more information that could nail down what vSMR version and possibly what fork that is?

I'll see if I can reproduce on the fork I maintain at least

[gergelycsernak]

Unfortunately not. Sentry does support downloading the minidump file only in paying version :(.

[pierr3]

Let me know if you can reproduce Stef, indeed we need to figure out which version of vSMR this relates to as I understand many people use a different fork nowadays. Thank you for the report Gergely.

…

On Sun, 18 Feb 2024 at 12:50, gergelycsernak ***@***.***> wrote: Unfortunately not. Sentry does support downloading the minidump file only in paying version :(. — Reply to this email directly, view it on GitHub <#111 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAN7UK57I6XVDY5HA3YRYJDYUI5OJAVCNFSM6AAAAABDOJTNO6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJRGM4TONBSGI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Epse]

I've just finished reversing that callstack into something useful, based upon the latest release build of my fork. The results are sensible, but also points to code that has not been touched and is identical in this repo, so it may be a common issue.

Decoded, I see the callstack originating from SMRRadar.cpp:1898 which in my fork is attempting to fetch a color. Next up we end up in Config.cpp:76, which should be an identical line number in this repo, attempting to load a sid colour. The final address points to some hard to understand c++ stringstream manipulating inside rapidjson.

I will attempt to re-run this analysis on a pierr3/vSMR build, but my current working theory is some kind of invalid config document being loaded which happens to fail at the closing of the ASR.

Edited to add the procedure below for those interested

Tracing procedure

Ensure you have ntsd installed. It comes with the Windows SDK.
You need a build of vSMR, I used a release build because presumably the reported crashes happen on release builds. Make sure your build process also produces a pdb file.

In the folder containing your dll and pdb file, launch the debugger using: ntsd -u vSMR.dll -y .
This will open a new window.
In this new window, I suggest running .lines to get line numbers printed and ld vSMR to ensure it loaded the debug symbols.
Then, you can look up what's at an address using u vSMR+0x92a3 for example.

[Epse]

Re-running the analysis on a build of upstream vSMR master perhaps gives even more sensible, if different results. Disclaimer is that both analysis had to be run on fresh builds as neither pierr3 nor I kept the symbols files for our releases.

On this repo, the first recognisable codepoint is in Config.cpp:152, specifically in one of the many rapidjson operator[] calls used on that line. From there, through various rapidjson internals we end up in rapidjson/reader.h:250 which is in fact an error handler, raising "Nothing should follow the root object or array."

Either analysis seems to point to invalid JSON configuration somewhere in the chain, with poor error handling. I see no clear bug or issue, other than that maybe the configuration handling across versions and forks could be improved.