3D Fluent Emojis

[tusharsnx]

Description of the new feature

Windows added 3D Fluent emoji support in 23H2. Supporting 3D Fluent emoji in Windows Terminal would be a nice addition.

Proposed technical implementation details

3D Fluent emojis are implemented in COLRv1 format. We'd have to support drawing DWRITE_GLYPH_IMAGE_FORMATS_COLR here:

terminal/src/renderer/atlas/Backend.cpp

Lines 73 to 105 in bd7e317

	void Microsoft::Console::Render::Atlas::ColorGlyphRunDraw(ID2D1DeviceContext4* d2dRenderTarget4, ID2D1SolidColorBrush* emojiBrush, ID2D1SolidColorBrush* foregroundBrush, const DWRITE_COLOR_GLYPH_RUN1* colorGlyphRun) noexcept
	{
	ID2D1Brush* runBrush = nullptr;
	if (colorGlyphRun->paletteIndex == /*DWRITE_NO_PALETTE_INDEX*/ 0xffff)
	{
	runBrush = foregroundBrush;
	}
	else
	{
	emojiBrush->SetColor(&colorGlyphRun->runColor);
	runBrush = emojiBrush;
	}
	const D2D1_POINT_2F baselineOrigin{ colorGlyphRun->baselineOriginX, colorGlyphRun->baselineOriginY };
	switch (colorGlyphRun->glyphImageFormat)
	{
	case DWRITE_GLYPH_IMAGE_FORMATS_NONE:
	break;
	case DWRITE_GLYPH_IMAGE_FORMATS_PNG:
	case DWRITE_GLYPH_IMAGE_FORMATS_JPEG:
	case DWRITE_GLYPH_IMAGE_FORMATS_TIFF:
	case DWRITE_GLYPH_IMAGE_FORMATS_PREMULTIPLIED_B8G8R8A8:
	d2dRenderTarget4->DrawColorBitmapGlyphRun(colorGlyphRun->glyphImageFormat, baselineOrigin, &colorGlyphRun->glyphRun, colorGlyphRun->measuringMode, D2D1_COLOR_BITMAP_GLYPH_SNAP_OPTION_DEFAULT);
	break;
	case DWRITE_GLYPH_IMAGE_FORMATS_SVG:
	d2dRenderTarget4->DrawSvgGlyphRun(baselineOrigin, &colorGlyphRun->glyphRun, runBrush, nullptr, 0, colorGlyphRun->measuringMode);
	break;
	default:
	d2dRenderTarget4->DrawGlyphRun(baselineOrigin, &colorGlyphRun->glyphRun, colorGlyphRun->glyphRunDescription, runBrush, colorGlyphRun->measuringMode);
	break;
	}
	}

Resources:

https://learn.microsoft.com/en-us/windows/win32/directwrite/color-fonts-preview#layers-of-solid-color-glyphs
https://learn.microsoft.com/en-us/typography/opentype/spec/colr#colr-version-1-rendering-algorithm
https://blogs.windows.com/windows-insider/2023/07/12/announcing-windows-11-insider-preview-build-25905
https://learn.microsoft.com/en-us/windows/win32/api/dcommon/ne-dcommon-dwrite_glyph_image_formats

[JohnnyWestlake]

If it does get added, make it optional please. They're extremely expensive to render compared to the COLRv0 variants.

[tusharsnx]

Atlas engine caches all the glyphs it renders, so you are only really paying the rendering cost on first use of any glyph. Later rendering just copies the cached glyph, which is extremely fast. I do have to confirm whether we cache emojis output or not. @lhecker

[lhecker]

I refuse to add support for COLRv1 emojis until their performance is fixed. Notably, they're not actually expensive to render at all! You can easily render all of them at 1000s of FPS even on dated hardware. So, this is purely due the currently poor implementation in Direct2D. I've already given this feedback to the respective maintainers, and we've been waiting for this to be fixed.

Aside from a subjective refusal there's also a real, objective problem: The only API that Direct2D offers to get the pixel size of a glyph is ID2D1DeviceContext::GetGlyphRunWorldBounds. We need that API to allocate sufficient space in the glyph atlas. MSDN fails to mention that this API was written for Windows 7 and was never updated to properly support Emojis since then. If you use an SVG emoji font with Windows Terminal, it'll fail in funny ways because of this. The only reason it appears to work is because COLRv0 emojis are basically just colored layers of regular TTF glyphs which are supported by that API. COLRv1 emojis thus cannot be supported either, because just like SVG emojis, their rasterized size information is bogus. We're similarly waiting for a fix on that.

[lhecker]

Later rendering just copies the cached glyph, which is extremely fast. I do have to confirm whether we cache emojis output or not.

Oh, to clarify: Yes, we cache these glyphs, but only for the D3D renderer. But that doesn't fix the severe hitches you see when the COLRv1 emojis first appear on the screen. We also still have an important D2D renderer which is used when accessing remote servers without discrete GPU.

[DHowett]

Considering the discussion here, I think we'll probably be closing it Won't Fix + Tracking-Internal... for now.