Map repaint is slow / CPU and GPU intensive

[JannikGM]

(This is a meta-issue, I hope we can create more performance related issues and documentation over time)

As graphics programmer, I have the task of adding animations to our maps to make our web-products feel more alive.
Originally I modified mapbox shaders to integrate animations, but we are currently migrating to custom-layers with a custom WebGL renderer.

However, I ran into a lot of performance issues with mapbox (and how we use it). In this issue I share my experience.

(Note: We use mapbox 1.x - basically what maplibre is, at the time of writing)

---

On most machines, we can't achieve stable 60FPS with a fullscreen map, if the camera is animated or anything else triggers map repaints regularily.
Rendering the map consumes a lot of CPU and GPU time, so the fans will spin up.

Our takeaway from this:

maplibre-gl-js / mapbox-gl-js are slow and seemingly not meant for animations of any kind; even if the mapbox samples suggest this.

These performance issues can be a distraction while our customers work with our webapps, so we try to combat this issue.
We do this, by optimizing our own maplibre / mapbox fork, but also by optimizing how we use our map.

The core issue is that the entire map re-renders (slowly!), even if only a single features in a single layer changes.

Our products mostly have a static camera, so short-term we optimize in 3 ways:

• Use raster tiles for the basemap, and only draw symbol layers locally.
  • Symbol layers are mostly calculated on the CPU, so we also try to keep these to a minimum.
  • In our fork, we'll likely move some of the label placement into the shaders (in the future).
• Use 2 canvases:
  1. Canvas for mapbox (only updated when camera moves).
  2. Canvas for our custom-layer overlays (updated at 30 or 60 FPS).
• Avoiding large GeoJSON sources and reducing the source-count.
  • Each source has high overhead during rendering.
  • GeoJSON is internally converted to VTF in a worker; if a new tile becomes available, map rendering is triggered.

We basically attempt to reduce the map render time and also try to avoid map-rendering altogether (if the base map didn't have any changes).

We are about to deploy these optimizations in our products.
However, we are already aware of different browser-performance issues with mixing those 2 canvases on high-DPI displays.
So it's not ideal either.

Long-term, we'll have to do a lot more optimizations to the maplibre or mapbox rendering code until we find an alternative product that we can use (if the issues can't be resolved in the near future).

---

The performance issues don't seem to be as severe with the native versions (mapbox-gl-native / maplibre-gl-native).

However, there are other issues with their custom-layer APIs (or lack thereof), which make it very hard, if not impossible, to render in RTC mode (namely, I believe the matrix you get is single-precision floats, and the inputs for the matrix calculation are rounded, too).
So custom-layers don't seem to be a cross-platform feature (or well supported anywhere) in maplibre / mapbox.

[CharcoalStyles]

In my experience, Raster has quite a few quirks on Mapbox GL JS. But it's generally really good at rendering vector and specifically vector tiles.

My suggestion would be to either:

• Render your data out into static vector tiles using Tippecanoe.
• Store the data in a PostGIS DB and use Tegola to serve them

For my day job, we have multiple Australia-wide datasets and using Tippecaone takes a while to build. But using Tegola is quite fast, even for complex or dense data sets. And for us, we can serve Tegola out of AWS Lambda and cache the output vector tiles in AWS S3, which is a quite cheap way of doing this.

We've also vectorised raster data and have that serving out of Tegola. If you'd like to have a look, the Surface Cover and Tree Heights on the map on this page shows that off.

[JannikGM]

In my experience, Raster has quite a few quirks on Mapbox GL JS. But it's generally really good at rendering vector and specifically vector tiles.

Can you elaborate on that?

My suggestion would be to either:

• Render your data out into static vector tiles using Tippecanoe.
• Store the data in a PostGIS DB and use Tegola to serve them

We do have a custom-made vector-tile-server (also written in Go) which generates some of our own data layers (such as traffic-flow).

On the client-side (webbrowser running mapbox-gl-js / probably maplibre soon) we have line-layers and symbol-layers which access these vector-tile-sources. We have about 5 of these layers which are not part of our default map style (which is generated by mapbox GL Studio; 1 composite source, ~100 layers, based on an older version of mapbox-streets).

---

In addition to those tile-server generated vector-tiles, our webapp receive realtime data-updates from another server which runs the business logic.

These data-updates are processed, aggregated and filtered on the client-side and end up in GeoJSON sources, which we update every couple of seconds (or on user-interaction). It's impossible to turn this into static vector-tiles, because this data is changing in realtime; when the source changes, mapbox has to repaint.
→ So every couple of seconds mapbox did a map.triggerRepaint() internally - performance was still okay.

We already had additional camera animations with flyTo and rotateTo, which caused a repaint once per frame, while the animation was active.
We already didn't maintain stable 60 (or even 30) FPS in those instances, but it was still okay, because animation durations were short.
→ So performance wasn't always great, but acceptable.

If you'd like to have a look, the Surface Cover and Tree Heights on the map on this page shows that off.

Your map mostly repaints while loading the map and while new tiles are coming in every couple of frames.
The other stressful case is when the user moves the camera - there's a noticable stutter, and the CPU usage goes up if the user "shakes" the map for long enough, but it's so short that it's acceptable.
These are rare or constructed, theoretical events.

Therefore, your use case is pretty much what I described above for our old use-case: Your map paints once / occasionally, and then sits idle for most of the time.
→ This issue does not affect you.

---

So to make it more clear why I created this issue, I'll elaborate on what I already hinted at in the issue description.

Recently, we added animations to our products map; similar to what's done in this sample: https://docs.mapbox.com/mapbox-gl-js/example/animate-a-line/
- except we did it more sophisticated with custom shaders / layer types in our fork of mapbox.

However, to support this buttery-smooth 60Hz animations, we now have to use map.triggerRepaint() every ~16 milliseconds.

So instead of redrawing the map once every couple of seconds, we now do it once per frame!
So any performance bottleneck in mapbox is amplified a lot.

When map performance breaks down, this is something you can feel because map interactions (dragging / rotating) suffer.
On this MacBook Pro (15-inch, 2017), the CPU usage also goes to almost 100%, the GPU usage goes to around 80%.
The fans spin to maximum and the whole machine feels slow.

And it turns out that this is caused by a number of different bottlenecks in mapbox (maplibre):

• 10 sources are insanely slow, with each GeoJSON source occasionally taking up about 1 millisecond (coveringTiles shows up in profiler) - so, with 10 sources, our source processing is already taking 10ms of our 16ms CPU budget.
  To combat this, we aggressively reduce the number of GeoJSON sources right now - but we still require them for data that we simply can't generate server-side / until we can generate VTFs locally. We are also in the process of moving some of our animations into custom-layers and rendering them independently from mapbox, so we can avoid map.triggerRepaint() altogether.
• updatePlacement is another huge bottleneck (during camera movement), because all the symbol/label placement is done on the CPU side and it's done more often than necessary.
• Simply drawing the basemap takes a while, and IPC for all the WebGL-calls / data-uploads can become a CPU hungry event for Chrome.
  So now we are experimenting with not re-rendering the basemap, and instead using raster-maps for (mostly) view-independent features.
• Additional HTML elements that overlay the map canvas take additional time for compositing.
  At this high-DPI display, this is a lot of additional stress for the GPU or memory bandwidth apparently, because lowering the browser window size helps with performance.

Take this with a grain of salt, becauser we also have some unknown factors contributing to slow performance in our products and we are still analyzing the extend of the issues.

However, there are many more bottlenecks, because, seemingly, mapbox simply isn't well-prepared for animated maps (animated features / animated camera).
We can reproduce many performance issues with a simple HTML page which does nothing, other than mapbox at fullscreen resolution.

The animate-a-line sample has a couple of major differences to our use-case:

• It's barebones: there's only a single source.
• There are no additional HTML elements on top of the map which trip the browsers compositor.
• It's not running fullscreen, which makes a big difference on high-DPI.
• It starts out fully zoomed out, so there's little label placement necessary, and only 1 tile.

If you change enough of these parameters, it will break performance again.

(Much of this also applies to the website you linked)

mapbox also has other ways of animating things, like using a video-overlay, HTML overlays (like mapbox-gl-js markers), .., - they all circumvent map.triggerRepaint(), but most of them aren't suitable for us. We want animations as part of the rendered map, including depth-buffer.
So either we go to another product which can handle this (For example, Cesium appears to render much quicker) or we optimize mapbox / maplibre.

• Short-term, we'll move our animations out of the map, into a second canvas, which we overlay ontop of the mapbox canvas (with slow browser compositing and at the expense of some design flexibility - like the mapbox depth buffer).
• Longer-term, if we stick to mapbox, I could imagine that we'll rewrite much of the mapbox rendering; namely caching text-strings into textures, caching rendered tiles into a raster-cache, prefetching tiles on flyTo and rotateTo, do the symbol mercator-coordinate → screen-coordinate transform in the shader, micro-optimizing shaders, potentially moving to WebAssembly for CPU-side vertex transforms, ...
• For case where the camera moves and new tiles are loaded, on the tileserver, we could do a more configurable ear-clipping or douglas-peuckert, so the client can avoid doing it locally; another thing to experiment with would be SDF-"compressed" tiles for the basemaps, or style parsing like mapbox-?optimize=true; a ?layer_first= + ?layer_last= option could be added to split between server and client-render parts.

All of this should help rendering performance and also avoid draining the battery on mobile (for example, we can reduce some shader execution time by about 50% by optimizing away some unused features / constant-propagation).

Many of these ideas probably deserve their own issues or further analysis; some were already discussed on the mapbox issue tracker or on tileserver repositories.

---

We also have other products which still use static mapbox maps (without animations), and even in those we found incorrect use of the map API which triggered many avoidable repaints (which led to higher power-usage). Hence, #97

[zakjan]

Linking the original issues. Currently, an animated custom WebGL layer really causes a high load because of map.triggerRepaint. It performs better in a separate context.

mapbox/mapbox-gl-js#7629
mapbox/mapbox-gl-js#8159

[CharcoalStyles]

@JannikGM Sorry I misunderstood when you said animation; I thought you meant animating the camera position. Here is a video of something I'm working on that does that and works quite well. But I've only tested it with slow pans that give the tiles time to load and render.

I've also done things with animating the paint properties of features. But never done anything with modifying feature geometry in real-time; that sounds quite intense.

I'd be very interested in seeing what you're doing; sounds quite cool!

[JannikGM]

Sorry I misunderstood when you said animation; I thought you meant animating the camera position.

It doesn't matter what kind of animation. Anything that repaints the map (map.triggerRepaint) is very bad.

Things which trigger map repaints (small selection):

• Changing any property (feature-state, layout, paint, ...), redundant or otherwise (until Avoid map repaint when setting same parameter value that is already being used #97)
• Moving / rotating / zooming the camera
• A tile finished preparing and can be rendered for the first time
• Animations like opacity-duration
• A manual map.triggerRepaint

Some repaints are more expensive than others (touching anything that affects layout is more expensive than paint properties for example; moving camera can also be very expensive).
But any of them are slower than they should be.

Here is a video of something I'm working on that does that and works quite well.

The video is not reachable without logging into that Slack ("Sign in to psma-au.slack.com" which I tried, but I got permission errors + I probably shouldn't join such Slack channels on this company account).

But never done anything with modifying feature geometry in real-time; that sounds quite intense.

We not only modify feature geometry. We also have features with static geometry which still change their texture / colors (custom shaders in our mapbox fork). We also have camera animations etc.

All of these animations are problematic, because all of them trigger a map.triggerRepaint.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This issue was closed because it has been stalled for 7 days with no activity.

[karussell]

This is an important issue - will reopen it. (btw: I'm skeptical that automatically closing an issue just due to lack of discussion makes a lot of sense in open source communities)

Thanks a lot for the discussion @JannikGM. We have the same problems. It turned out that even with a simple circle moving around you get this ugly stuttering and high load. Especially with firefox (it got slightly better with the recent FF94), a bigger map window and if you do not have a dedicated graphic card. (In my case all 3 points are true for desktop and it is really ugly experience.)

I created an issue for firefox here: https://bugzilla.mozilla.org/show_bug.cgi?id=1732049 and they confirmed a considerable difference between chrome and firefox.

Do you have a small code snippet that shows what you mean with using 2 canvases?

[JannikGM]

Do you have a small code snippet that shows what you mean with using 2 canvases?

You can look at https://deck.gl which has mapbox interop and moves its own layers into a second canvas by default (https://deck.gl/docs/get-started/using-with-map).

[github-actions]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[HarelM]

I'm not sure what is the expected output of this issue.
I can obviously convert this to a discussion or I can simply say "maplibre needs to be faster" but I don't see how this will help anyone...
If you would like to share a more detailed problem or use case I think it will help.
Or let me know if I should convert this to a discussion.

[gsimko]

@HarelM I've created a demo that shows a (hopefully actionable) performance problem. It consumes 40% cpu+30% gpu on a macbook pro m1, and arguably it should be able to cache everything and perform at nearly 0% resource usage. Does that help?

<html>
  <head>
    <link rel="stylesheet" href="https://unpkg.com/maplibre-gl@3.6.2/dist/maplibre-gl.css" />
    <script src="https://unpkg.com/maplibre-gl@3.6.2/dist/maplibre-gl.js"></script>
  </head>
  <body>
    <div id="map" style="height: 100%"></div>
    <script>
      const map = new maplibregl.Map({
        container: "map",
        style: "https://demotiles.maplibre.org/style.json",
        center: [45.445, 14.59],
        zoom: 4,
      });
      setInterval(() => {
        const t = performance.now();
        map.jumpTo({ center: [45.445 + Math.cos(t / 1000) / 100, 14.59] });
      });
    </script>
  </body>
</html>

[HarelM]

The above code renders the map over and over again, which uses both the CPU and GPU.
I'm not sure why you would say that it should be around 0%.

[JannikGM]

I'm not sure what is the expected output of this issue.
I can obviously convert this to a discussion or I can simply say "maplibre needs to be faster" but I don't see how this will help anyone...

This issue describes a higher level problem / goal.
Ideally this issue would be split into smaller tasks which can actually be implemented, such as:

• Improving docs to warn about slowness
  • Avoid samples like "animate-a-line" which don't reflect real-world performance with many sources, more visible features, ...
  • Avoid examples like the maplibre.org website: camera animations in a small canvas on a very empty map which doesn't reflect performance in more complicated scenarios (fullscreen and more sources / layers).
• Better caching of rendering results (merge layers into a new raster tiles, as explained in New paint property fill-comp-op for fill type layer #1191 (comment)).
• Better prefetching (avoid calculation or fetches while re-rendering).
• Refactors to the custom-layer API (to allow for tile-less rendering).
• Redesign of how information flows from GeoJSON to the actual map (tiling overhead).
• Move per-feature attributes into the GPU (potentially a bad example: is the per-feature opacity, which could be done by running a global timer in a uniform and only keeping timestamps when the fade started per-feature).
• Reconsider design of existing features (clipping based on stencil, but there's also clip-planes in WebGL now which could safe some fill-rate).

Since I created this issue, we've moved all our webapp animations to a custom WebGL canvas (now based on deck.gl), meaning this has lost priority for our company. It's unlikely I'll work on these problems anymore.
We avoid most maplibre problems by avoiding camera movement / any changes affecting the basemap.
We have to accept the stuttering while moving the camera for now.

Long term I'm hoping we can replace the base-map by https://github.com/maplibre/maplibre-rs or similar to improve basemap performance.

It consumes 40% cpu+30% gpu on a macbook pro m1, and arguably it should be able to cache everything and perform at nearly 0% resource usage.

As @HarelM says: it can't be 0. It must still render (when the camera moves).

I get stable 60/120Hz on my M1 machine; so I don't care about actual numbers on this platform - although 40% CPU sounds too much (if it's true); optimizations would be able to reduce this. This would mostly be about power savings.

However, we have clients with ~10 year old office machines and depending on what gets displayed they struggled with usability of maplibre. They were getting less than 10 FPS and it would affect other apps (like Excel / Word / Slack / Zoom / ..); fan noise caused issues in their offices.
In these cases it's not about power savings, but maplibre breaking the product.

Regarding the GPU measurement: Don't trust it.
I'd often see 100% usage at the base clock speed; the GPU would then get clocked higher and the usage fell down to ~10%.
GPUs are also a lot less homogeneous than CPUs, so you might be bottlenecked even at low usage numbers.