Add support for macOS __crash_info data

[steven-michaud]

I'm creating this pull request to follow up https://bugzilla.mozilla.org/show_bug.cgi?id=1577886, a fix for which landed a few days ago. This fix needs a followup fix, but the patch in my pull request has already been updated along these lines.

I've tested this patch using make build and make shell. It seems to work fine. But I do have one question about the output of stackwalker <MINIDUMPFILE> or stackwalker --pretty <MINIDUMPFILE>. As my patch currently stands, the output looks like this:

    "mac_crash_info" : {
       "num_records" : 2,
       "records" : [
          {
             "message" : "CryptKit fatal error: Raise test exception from _pthread_cond_wait(2)",
             "module" : "/System/Library/Frameworks/Security.framework/Versions/A/Security"
          },
          {
             "message" : "abort() called",
             "module" : "/usr/lib/system/libsystem_c.dylib"
          }
       ]
    },

But I could also make it look like this:

    "mac_crash_info" : {
       "num_records" : 2,
       {
           "message" : "CryptKit fatal error: Raise test exception from _pthread_cond_wait(2)",
           "module" : "/System/Library/Frameworks/Security.framework/Versions/A/Security"
       },
       {
           "message" : "abort() called",
           "module" : "/usr/lib/system/libsystem_c.dylib"
       }
    },

Or this:

    "mac_crash_info" : {
       {
           "message" : "CryptKit fatal error: Raise test exception from _pthread_cond_wait(2)",
           "module" : "/System/Library/Frameworks/Security.framework/Versions/A/Security"
       },
       {
           "message" : "abort() called",
           "module" : "/usr/lib/system/libsystem_c.dylib"
       }
    },

Do you have a preference?

[willkg]

I was thinking about this and how the data could get used in Socorro and I don't think I have a preference. Having a "results" intermediary key allows us to expand in the future with additional bits. I don't know what other bits could be interesting.

I think I'm fine with what's implemented.

I'll try to find some time to test this out this week. Do we have any crash reports that have been submitted to Crash Stats that I can test with?

[steven-michaud]

Do we have any crash reports that have been submitted to Crash Stats that I can test with?

Here are two that I just made, using today's mozilla-central nightly and my HookCase hook library from https://bugzilla.mozilla.org/show_bug.cgi?id=1577886#c12. Both should contain some __crash_info data:

https://crash-stats.mozilla.org/report/index/1bc9e790-d009-4419-a87a-e4af50210503
https://crash-stats.mozilla.org/report/index/366161a6-34a0-4411-b016-0fad30210503

[steven-michaud]

Also, any crash with a signature containing "gpusGenerateCrashLog" should have __crash_info data, if its minidump was generated by a build containing my patch for bug 1577886:

https://crash-stats.mozilla.org/search/?build_id=%3E%3D20210501093251&signature=~gpusGenerateCrashLog&version=90.0a1&platform=Mac%20OS%20X&date=%3E%3D2021-04-27T14%3A21%3A00.000Z&date=%3C2021-05-04T14%3A21%3A00.000Z&_facets=signature&_facets=platform_version&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

[willkg]

Awesome--that helps a lot!

[willkg]

This looks good to me!

[willkg]

@gabrielesvelto Any objections to this?

[gabrielesvelto]

No objections. I asked Steven to get the patch into Socorro after having deployed it in m-c.

[steven-michaud]

As best I can tell, I can't merge this myself. So someone please do it for me :-)

[steven-michaud]

And once this has merged, there's still another step (or steps) to take:

We need to get a summary of "mac_crash_info" onto the "details" page of crash reports at https://crash-stats.mozilla.org/. And we need to make this information searchable. I have ideas about both of these. Should I open a bmo bug to discuss them?

[gabrielesvelto]

And once this has merged, there's still another step (or steps) to take:

We need to get a summary of "mac_crash_info" onto the "details" page of crash reports at https://crash-stats.mozilla.org/. And we need to make this information searchable. I have ideas about both of these. Should I open a bmo bug to discuss them?

Yes, open a bug under the Socorro component. Since we don't know exactly what these strings might contain we cannot make them publicly visible right away. We first have to make sure that they do not contain arbitrary data about the user because if they do they'll have to be protected content. Also we might need a data review before adding new fields to Socorro.

[steven-michaud]

I've opened https://bugzilla.mozilla.org/show_bug.cgi?id=1709658.

[willkg]

Now that this is merged, I need to do a release and then pull it into Socorro and then do a prod deploy of that. Typically that takes me a couple of days to work through.

[steven-michaud]

Now that this is merged, I need to do a release and then pull it into Socorro and then do a prod deploy of that. Typically that takes me a couple of days to work through.

I just triggered another crash with __crash_info data. It's still not showing up on the "raw data" page of the crash report. Have you done this yet?

https://crash-stats.mozilla.org/report/index/2e7cb5a1-832d-4fea-8fa7-67fc20210510

[willkg]

No, I haven't. I'm going to update minidump-stackwalk in Socorro along with the rest of the work in bug 1709658.

[steven-michaud]

OK, that makes sense.