Skip to content

Fix integer-overflow heap out-of-bounds write in RenderSDF#202

Merged
kkaefer merged 3 commits into
masterfrom
kk/fix-oob-write
Jul 2, 2026
Merged

Fix integer-overflow heap out-of-bounds write in RenderSDF#202
kkaefer merged 3 commits into
masterfrom
kk/fix-oob-write

Conversation

@kkaefer

@kkaefer kkaefer commented Jul 2, 2026

Copy link
Copy Markdown
Member

A crafted font could trigger a heap out-of-bounds write in RenderSDF (include/mapbox/glyph_foundry_impl.hpp).

RenderSDF computed the SDF bitmap allocation size as buffered_width * buffered_height in 32-bit unsigned int. Glyph dimensions are derived from the uploaded font's outline bounding box with no upper bound, so a glyph large enough to make that product exceed 2³² wrapped to a small value. The buffer was then under-allocated while the render loop still wrote one byte per pixel, corrupting the heap (CWE-190 → CWE-787) and crashing the process. A 680-byte font is enough to drive FreeType to a 65536×65536 buffered glyph, wrapping the size to 0.

Fix

  • Cap glyph dimensions at 256px and drop oversized glyphs before allocating. The render size is fixed at 24px, so legitimate glyphs stay far below this bound; abusive glyphs are omitted from the output (matching existing empty-glyph handling) and the range request still succeeds.
  • Widen the bitmap size and write index to size_t as a backstop against any 32-bit overflow.
  • Clamp glyph.left/glyph.top to the same ±256px bound, keeping the doubleint32_t conversion well defined.
  • Tighten integer handling flagged by the compiler: stop narrowing FreeType metrics through int (which truncated font-controlled values before the range checks in glyphs.cpp), and make the remaining width/height, segment-box, buffered-dimension, and SDF-value conversions explicit casts.

Test plan

  • Added test/overflow.test.js + test/fixtures/evil.ttf: renders the overflow-triggering font in a child process and asserts it completes without crashing and the oversized glyph is dropped. Confirmed it crashes with SIGBUS before the fix and passes after.
  • npm test — full suite passes (424 assertions), with byte-identical output on the existing font fixtures, confirming the changes are behavior-preserving for legitimate fonts.
  • Clean Release rebuild produces no integer-conversion warnings in glyph_foundry_impl.hpp.

kkaefer and others added 3 commits July 1, 2026 13:16
A crafted font could produce a glyph whose buffered dimensions, multiplied
together in 32-bit arithmetic, overflowed and wrapped to a small value. The
resulting bitmap buffer was undersized while the render loop still wrote one
byte per pixel, corrupting the heap (CWE-190 -> CWE-787) and crashing the
process from a single font render.

Cap glyph dimensions at 256px and drop oversized glyphs before allocating;
the render size is fixed at 24px so legitimate glyphs stay well under this
bound. Also widen the bitmap size and write index to size_t as a backstop
against 32-bit overflow.

Add a regression test that renders an overflow-triggering font in a child
process and asserts it completes without crashing and drops the glyph.

Co-authored-by: Cursor <cursoragent@cursor.com>
Make the conversions of font-derived values explicit and bounded, resolving
the integer-related compiler warnings:

- Clamp glyph.left/top to the same +/-256px bound as the dimension cap; at the
  fixed 24px render size the position never approaches the int32 limits, and
  this keeps the double->int32_t conversion well defined.
- Widen FreeType metrics straight to the double fields instead of narrowing
  through int, which truncated font-controlled values before the range checks
  in glyphs.cpp.
- Make the width/height, segment-box, buffered-dimension and SDF-value
  conversions explicit casts.

Co-authored-by: Cursor <cursoragent@cursor.com>
@kkaefer kkaefer requested a review from a team as a code owner July 2, 2026 10:52

@AlexanderBelokon AlexanderBelokon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kkaefer kkaefer force-pushed the kk/fix-oob-write branch from 77bc1b2 to dd14aa5 Compare July 2, 2026 10:57
@kkaefer kkaefer merged commit dd14aa5 into master Jul 2, 2026
@kkaefer kkaefer deleted the kk/fix-oob-write branch July 2, 2026 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants