Improve accuracy of CJK glyphs with higher resolution TinySDF textures

[bdon]

User Story

As a viewer of the map in Chinese or Japanese language, the quality of rendered text should look as good as Latin text and other scripts.

Rationale

• All text displayed in MapLibre GL, web or native, is rendered using Signed Distance Fields (SDFs). This means text can be smoothly rotated and rescaled between different sizes as a function of zoom level, and text halos - a common visual design for map labels - are computationally "free".

• The drawback to the SDF approach is that it isn't the format fonts are commonly stored in. Fonts are defined as a collection of Bézier curves in a .TTF or .OTF file. In order to display them as SDFs they need to be "baked" into a texture via preprocessing with a program like font-maker.

• The conversion from .TTF to SDFs requires the choice of a specific font size for rasterization into a bitmap via FreeType. Early on in GL JS development this was determined to be a 24 point font as a compromise between text sharpness and SDF bitmap size. SDF bitmaps are then stored in ranges of 256 glyphs and served over HTTP. A "font stack" is 256 files of 256 glyphs each, covering the entire 65,536 codepoint Basic Multilingual Plane

• SDF 24 point font is good enough to render most Latin fonts with a small amount of rounding. The issue arises when rendering glyphs with more internal detail. These are common where CJK is used with "traditional" scripts - mainly Japan, Taiwan, Hong Kong and Macau. From the MapLibre example page I screenshotted a few¹:

• Later on in GL JS development, it was decided to special case Chinese, Japanese and Korean glyphs to ignore the above code path, because the amount of network traffic required impacted map performance. Natural CJK text has poor locality over the space of UTF-8 codepoints, so a single map tile could result in several, even dozens of SDF files requested. The alternative localIdeographs code path renders text via mapbox/tinysdf (BSD) and can be controlled by the localIdeographFontFamily option.

• TinySDF skips over the FreeType pre-baking step by using the browser's Canvas API to draw text to a canvas, and then converts that canvas into an SDF. This reduces network requests for CJK glyphs to 0, but limits the display of CJK text to only the basic fonts built into the browser (Sans Serif for Gothic style, Serif for Ming/Song style)

• TinySDF takes the 24 point font size as as parameter, this is hardcoded in GL JS to 24pt

tinySDF = entry.tinySDF = new GlyphManager.TinySDF({
              fontSize: 24,
              buffer: 3,
              radius: 8,
              cutoff: 0.25,
              fontFamily,
              fontWeight
          });

Because TinySDF takes any font size as a parameter, we can increase the rendering resolution; most obvious would be doubling it to 48pt instead of 24. You can see the difference on the mapbox/TinySDF demo page (BSD).

• Bumping this value is not enough - now the program needs to deal with 2 different glyph SDF sizes instead of just one, so shaders like symbol_sdf.vertex.glsl need to be updated:edit: see below, no shader modifications necessary

float fontScale = u_is_text ? size / 24.0 : size;

• Any other code that assumes 24pt fonts needs to be updated.
• Parameters like assumptions of typographic baselines and gamma hardcoded into MapLibre need to be updated. Relevant issues:

#1051
#1002
maplibre/maplibre-style-spec#174

Impact

• TinySDF text takes longer to render. If font size is doubled, the underlying textures are 4x larger.
• More VRAM (4x for doubling) is needed for storing CJK glyphs in glyph atlases.
• The code is more complicated because shaders need to deal with glyph textures of different sizes.
• There should be no runtime impact if CJK is not rendered.

Alternatives

• We can do nothing, but that results in the visual quality problems as described above.
• We can bump all textures to a higher number like 48px across the codebase, for all scripts. This would require re-generating all font SDFs, resulting in multiple versions per font (24, 48), and increase the SDF download size significantly.
• We can wait for a larger rework of the text rendering system that addresses complex text layout.

Footnotes

1. Note the display of U+FA10 塚 may look unusual, this is my browser's locale setting and is not relevant to this issue. ↩

[HarelM]

Generally speaking, and this is only my personal opinion, increasing to 48 feels like a patch and not a real solution, I would prefer to have solution that really solves this problem, preferably even one that allows using regular font files instead of this pre-generated glyphs.
Again, my personal opinion.

[bdon]

@HarelM sure, I agree at a high level a solution that does not pre-rendered SDFs is better, for example:

• Rasterizing fonts in the browser via FreeType compiled to WebAssembly.
• Depending on TinySDF for all operations, letting the browser block on loading a standard Web Font (woff2) and then drawing glyphs or runs of text, possibly also solving text shaping/complex text layout

But even if you have a non-pregenerated glyph solution, the rendering and WebGL components of MapLibre will always need to deal with signed distance fields. Any move away from SDFs would have a major impact on the user experience and styling capabilities of MapLibre. So an enhancement to the SDF pipeline for higher res textures is complementary to the solution you mentioned, not an alternative.

Other hardware-accelerated text display systems like video games use SDFs, but they can load fonts on disk and have higher VRAM budgets, so are not subject to the same constraints. A reasonable alternative to this proposal is to keep 24pt fonts but use multi-channel SDFs:

https://github.com/Chlumsky/msdfgen (MIT license)
https://github.com/heremaps/harp.gl/tree/master/%40here/harp-text-canvas HarpGL map renderer (Apache License)

I think moving all of MapLibre to MSDFs is viable, but it would also require re-generating all font stacks, would triple the TinySDF cost and triple VRAM usage even if all glyph textures in the atlas are 24pt - you'd then need red, green and blue channels.

[ChrisLoer]

This is a great write-up @bdon! The Mapbox GL JS equivalent of this is one of the few things I worked on post-fork, so I think I need to stay a little bit at arms-length from this, but just to share a few bits of perspective from that effort:

• It's really hard to communicate the difference in quality to non-native CJK readers. This is really a glaring problem that just ends up looking like "somewhat crisper pixels" to a non-native reader. I think it's correct to see bumping to 48px as a "stopgap" measure, but this is why we thought having a stopgap solution was necessary -- and this was not very hard to implement.
• IIRC, actual render time was almost unaffected, tile buffer upload time was somewhat slower (because of the larger textures), but the biggest impact we worried about was the time to actually generate the SDFs with TinySDF (which fed through to longer times until the first tiles displayed on the map). We played around with a bunch of different optimizations here (moving some of the TinySDF work off the foreground, caching glyphs in localStorage, optimizations to TinySDF which were published open-source) -- but an important point for us was that this cost was only incurred on maps displaying CJK glyphs, and in general "slow but legible" trumped "illegible and fast".

[wipfli]

Thanks for this nice summary @bdon, I could not have written it better. Your proposal seems to be the right choice from my point of view. It is an incremental improvement of our text rendering stack and the risk of doing something wrong is very small.

[bdon]

I've made an interactive comparison of my branch vs maplibre 3.3.0:

https://bdon.github.io/maplibre-cjk/

Video:

jp-demo.mp4

It turns out this is easier than I expected to implement, without having to modify any shaders or vertex buffer layouts, only code that touches JavaScript. I haven't tested 100% of all cases or looked closely at the typographic metrics to make sure the spacing is the same between old and new code.

[neodescis]

Thanks for making a live demonstration. It is really eye-opening!

[bdon]

I've updated the demo to align the new glyphs pixel-perfect with the old ones, so visual styling should remain the same with a minor version bump. (At least on Apple devices, need to test others)

https://bdon.github.io/maplibre-cjk/

I've also fixed text-writing-mode: vertical cases.

Another comparison showing a particularly bad case that is fixed now:

cjk-pixel-align.mp4

[HarelM]

Looks great!
Feel free to open a PR if you want the tests to run on every change.
If this is a small increment towards a wider solution and improves the current situation without changing a lot of code, let's push it forward.

[bdon]

Released in 3.4.1