Crash rendering the map — mbgl::FillBucket::drawElements, glrAGXRenderVertexArray #5731
Comments
|
We’ve seen some reports of this exact crash as well, on iOS 9.2.1 on an iPhone 6, several minutes after launch. It looks vaguely similar to #1792. |
1ec5
changed the title
|
Also, does anything appear in the console around the time of the crash, as suggested in this Apple Developer Forums thread? |
|
Based on other reports we’ve received, as well as some automated reports via an internal testing tool, this issue appears to be triggered or exacerbated by continuous animation of the viewport for long stretches of time. Does that describe your situation, @MikeKasperlik? |
|
We animate the viewport, but not for a long time. Based on what the user does, it can happen that the viewport gets animated often, but that are separate animations. I checked the logs and in some cases we do had similar entries as described in the developer thread. |
|
@MikeKasperlik Any chance you can share a minimal piece of code that triggers (always or frequently) this problem? Going to start looking into this. |
|
Going to try to repro myself and investigate. |
|
Also @MikeKasperlik more info welcome on device and/or version. Haven't documented this on devices newer than iPhone 6 or iOS later than 9.2.1 yet. |
|
Actually, we’re seeing some reports internally of similar crashes on iOS 9.3.3 on iPhone 6 Plus. |
|
@incanus I can't say whats triggering this unfortunately. I do can provide you some more information: Device stats: Operating system: Device status: What's strange about this is, that we received quite a lot crashes until August 18th. Since then not a single crash without updating our app - maybe something with the delivered tiles ? |
|
Trying varying things (mostly a modified "world tour" functionality) to repro this, but no luck yet. |
|
I've been running through this using a database of 1,250 of the world's largest cities and a zoom level appropriate for heavy fill drawing, but still no luck reproducing. @MikeKasperlik any way you can provide more info on at least one crash location (geographically) so that I can get an idea of what might be drawing when this occurs? |
|
Possible lead: https://forums.developer.apple.com/thread/9589 |
|
I have no records of the exact crash locations, but most of them happened in Germany (I think that's because most of our users are located there). The most recent crash was somewhere around here: Lat: |
|
I just had same kind of crash in my app! Device: iPad mini 2, iOS 9.3.4. Location: http://www.openstreetmap.org/#map=16/24.8033/121.0368 |
|
Really not getting anywhere on repro of this, but still working on it. |
|
Probably same crash |
|
I'm having more and more MapBox related crashes...recently: With the thread: And With the thread: |
|
@YBthebest, based on the stack trace, #5731 (comment) looks more like #1509, which was originally reported long before this issue. ( |
|
Based on #5731 (comment), one lead we’re investigating is that the number of OpenGL drawing calls is indeed very high. We’re looking into whether there was a significant increase recently, and whether turning off complex layers such as landcover alleviates the issue. We did make some major changes to map rendering in v3.3.0, including switching to a new tesselation library (#2444), but it remains to be seen whether those changes had any impact on the volume of drawing calls. |
|
Looked a bit into this without luck yet, but I'm wondering if we can put in some live-running counts of draw calls per frame (along with FPS) as a debug mechanism here post-mortem? Not finding anything in OpenGL for this yet—all we've seen is using Xcode's GPU frame capture, which is one time and then outputs the draw call count (among many other statistics). |
Really weird, but simple testing here removing the four default landcover layers takes me from 6,270 draw calls down to 6,266 in a quick test. Negligible. |
|
Oddly, I'm still seeing draw calls for landcover layers even without them in the style—about 337 draw calls though.
|
|
I guess it has something to do with memory warning handling. |
|
Could this be a concrete solution? |
|
I tried to replicate this crash, in particular focusing on |
|
We've been experiencing a similar crash through our app (reported through Fabric) Here's the main thread stacktrace: Another similar crash: |
|
@cellininicholas what device/OS version are you using? are you referring to free storage space or free RAM? |
|
The crash I mentioned above has come in numerous times (48 in a short space of time, during a period where we don't have many users) through Crashlytics. 52% on iOS 9, 46% on iOS 10. Of all the iPhone crashes, 33% were iPhone 6, 19% where iPhone 6s Plus, 19% were iPhone 5s, 29% were Other... All of iPad crashes were on an iPad Air. I was referring to the free RAM when the crashes occurred. All seem to have occurred when there was <8% RAM free on the device. Hope some of this might be helpful? |
|
I see a similar crash in our app. Last occured on an iPhone 5 (iOS 9.3.5) MapBox Version 3.3.1. According to Crashlytics, 62 MB free RAM. |
|
This is by far one of the most frequent crashes we're also seeing within our app since upgrading to v3.3.x of the Mapbox iOS SDK, among many other OpenGL-related crashes. Others have provided similar stacktraces related to this same issue, but here's another from our app which is using v3.3.4 of the SDK: Haven't really seen any of these crashes come in for iOS 10 yet as most are from iOS 9.0. Also, it doesn't appear as if it's memory-related as several crashes were reporting free RAM > 15%. |
|
A common thread in all the stack traces is that the crash occurs when rendering the opaque pass for a fill layer. (You can tell it's the opaque pass because I saw a crash like this when working on #6468. It happened when I introduced a bug that resulted in calling As a test, I reintroduced the bug in #6468 and I get almost exactly the same stack trace as seen here. In particular, the top of the stack trace is How would this happen? A VAO will have an unbound element buffer if:
I think this list is exhaustive. @kkaefer @ansis Can you think of anything I'm missing? I've combed through the code, and I can't see where any of these issues are present. In particular:
Another question that might be the key to unravelling this is: why do we only see crashes in fill layers, and not any other layer type? If it was really a bug with VAO state, I'd expect at least some crashes on other layer types, since our use of VAOs isn't specific to fill layers. |
|
Other hypotheses explored, none of which seem plausible:
Perhaps if a build that calls |
|
Hi Mapbox, Any immediate fix? |
|
As a data-gathering mechanism, 596f9c5 enables error checking for all GL calls. We're not going to apply it in mainline releases because there may be usage scenarios where it would have a performance impact, but we can make targeted test builds that include it and see if they reveal anything interesting. /cc @frederoni |
|
For us it is happening very often in different devices, we use a map with a custom style. |
|
I was finally able to reproduce the bug locally and determine the cause. The crash is triggered when an application which uses multiple map views encounters a low-memory situation. In such a situation, However, The fix is simple: #6972. We'll need to cherry pick this to the iOS and Android release branches, and should also consider an iOS 3.3.6 release that includes it -- I'm not sure that 057b7b7 will end up helping much. |
|
Yep. |
|
Published ios-v3.3.6 |
|
Upgraded to 3.3.6 and still crashing |
|
@volonbolon That is a different crash signature, one that I don't believe we've seen before. Can you please open a separate issue? |
|
Sure |
Platform: iOS
Mapbox SDK version: 3.3.1
We see a high crash rate for this on fabric.
I can't reproduce this crash but it happens randomly when the app is in foreground and doing something with the map.
We use a custom style & draw a single polyline on the map
Other Threads:
And a lot of worker threads that all look the same
The text was updated successfully, but these errors were encountered: