Add SGR-Pixels mouse mode (1016) by sebgod · Pull Request #19949 · microsoft/terminal

sebgod · 2026-03-06T01:00:03Z

Summary of the Pull Request

Add support for SGR-Pixels mouse mode (DECSET 1016), which reports mouse
events using pixel coordinates instead of character-cell coordinates. This
uses the same CSI sequence format as SGR mode (1006) but with pixel
positions, enabling sub-cell mouse precision for applications that need it
(e.g., sixel-aware programs, smooth scrolling in TUI frameworks like
Textual).

References and Relevant Issues

Detailed Description of the Pull Request / Additional comments

What is SGR-Pixels (mode 1016)?

SGR-Pixels is an extension of the SGR extended mouse mode (1006). It uses
the identical sequence format:

CSI < button ; x ; y M    (button press)
CSI < button ; x ; y m    (button release)

The only difference is that x and y are pixel coordinates (1-based,
relative to the top-left of the text area) instead of character-cell
coordinates. This allows applications to detect mouse positions within
individual cells, which is useful for:

Interacting with sixel graphics at pixel precision
Smooth scrolling implementations
Sub-cell UI elements in TUI frameworks

Implementation details

New mode constant: SGR_PIXEL_MODE = DECPrivateMode(1016) added to
DispatchTypes.hpp.

New encoding mode: SgrPixelMouseEncoding added to
TerminalInput::Mode enum. It is mutually exclusive with Utf8MouseEncoding
(1005) and SgrMouseEncoding (1006) — enabling one disables the others.

Encoding logic: When SGR-Pixel mode is active, HandleMouse passes the
raw pixel position (instead of the cell position) to _GenerateSGRSequence.
The sequence generator is reused as-is since the format is identical; only
the coordinate source differs.

Pixel coordinate pipeline: A pixelPosition parameter was added to
HandleMouse and propagated through the calling chain:

Layer	Pixel source
`ControlInteractivity` (WinUI)	Raw DIP position from pointer events
`HwndTerminal` (Win32)	Raw pixel position from `lParam`
`InputBuffer` (conhost)	Approximated from cell coords × font size

The conhost path approximates pixel coordinates from cell positions since the
Win32 console input API only provides character-cell coordinates. This gives
cell-boundary-aligned pixel values, which is the best available without
refactoring the conhost input pipeline.

DECRPM support: Mode 1016 is queryable via DECRPM (CSI ? 1016 $ p),
reporting enabled/disabled status.

Validation Steps Performed

Added SgrPixelModeTests to MouseInputTest.cpp covering:
- Multiple pixel coordinates (0,0), (5,10), (100,200), (1920,1080)
- All button types: left/middle/right down/up and mouse move
- All modifier key combinations: none, Shift, Ctrl, Alt, Alt+Ctrl
- Verified 1-based pixel coordinate output format
- Verified mutual exclusivity with SGR mode (1006) and UTF-8 mode (1005)
Code review confirmed all function signatures match between headers and
implementations
Verified encoding priority order: UTF-8 > SGR-Pixel > SGR > Default

PR Checklist

Closes Add SGR-Pixels (mouse mode 1016) #18591
Tests added/passed
Documentation updated
- If checked, please file a pull request on our docs repo and link it here: #xxx
Schema updated (if necessary)

sebgod · 2026-03-06T01:02:19Z

@microsoft-github-policy-service agree

Implements DECSET 1016 (sgr-pixels), which uses the same CSI sequence format as SGR mode (1006) but reports pixel coordinates instead of character-cell coordinates. This enables sub-cell mouse precision for sixel-aware programs and smooth scrolling in TUI frameworks. - Add SgrPixelMouseEncoding mode, mutually exclusive with 1005/1006 - Thread pixel coordinates through the input pipeline - Add DECSET/DECRST/DECRPM support for mode 1016 - Add SgrPixelModeTests covering coords, buttons, modifiers, exclusivity Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

j4james · 2026-03-08T22:05:38Z

I haven't reviewed the code in detail, but I think there are couple of high level issues that need to be addressed:

The most important thing is that the coordinates need to be scaled to match the virtual pixel resolution, otherwise an app won't be able to determine what that position actually means. To get started with, it's probably OK to hardcode the cell size as 10x20. So for example if the mouse is in the middle of the the cell in the top left corner, the cordinates would be something like (5,10).
It looks like you might not be taking the scroll offset into account. If a user has scrolled up slightly to view something in their scrollback buffer, and then clicks in the app that requested the mouse mode, the app will be expecting coordinates relative to the top of the VT buffer, and not the currently visible viewport (you can see the adjustment used for the the cell coordinates in _sendMouseEventHelper).
Once you've adjusted the coordinates as mentioned in point 2, make sure they're still in range. You shouldn't be reporting mouse positions outside the VT display area, or at least the coordinates should be clamped (you can see the equivalent clamping for the cell coordinates in Terminal::SendMouseEvent).
If we're not going to support conhost, I think it's best not to generate fake coordinates, and instead make the DECRQM report indicate that the mode is permanently reset. That way an app won't be misled into thinking it can do something meaningful with this mode. I'm hoping there is still some way for the AdaptDispatch class to determine if it's serving conhost or not.

lhecker · 2026-03-09T16:04:46Z

src/host/inputBuffer.cpp

+        // Approximate pixel position from cell coordinates for SGR-Pixel mode (1016).
+        const auto fontSize = gci.GetActiveOutputBuffer().GetCurrentFont().GetSize();
+        const til::point pixelPosition{ position.x * fontSize.width, position.y * fontSize.height };


I concur with what @j4james said (who conceived and guided development the majority of this code). To add: This VT sequence doesn't report actual display pixels. Instead it reports them using the pixel size of the original DEC which was (among others) 10*20 pixels. It's almost like a legacy "DPI-independent coordinate format".

Regarding the implementation specifically, I think we should pass the viewportPos as two float parameters and remove the pixelPosition parameter. Clicking in the middle of the cell at 10,23 will then pass 10.5, 23.5. The TerminalInput class can then either truncate the float to an integer (using e.g. lrint) for existing VT sequences or multiply by 10/20 and then truncate to an integer for your new code.

This requires minimal changes in the UI code since it already calculates viewport position in pixels / cell size in pixels. It does it using integers right now and we'd just need to turn that into floats as far as I can see.

Rework the coordinate pipeline per reviewer feedback: 1. Use virtual pixel resolution (10x20 per cell) instead of raw display pixels. The DEC VT340 convention defines character cells as 10x20 virtual pixels, so coordinates are computed as viewportX*10 and viewportY*20, then reported 1-based in SGR format. 2. Pass fractional cell coordinates (float) through the mouse event chain instead of a separate pixelPosition parameter. The UI layer already computes pixel/cellSize — switching from integer to float division preserves sub-cell precision. TerminalInput truncates to int for cell-based modes or multiplies by 10/20 for SGR-Pixels. 3. Scroll offset is now applied to the float Y coordinate in _sendMouseEventHelper, preserving sub-cell precision through the adjustment. 4. Float-based clamping in Terminal::SendMouseEvent ensures coordinates stay within the VT display area [0, viewport dimensions). 5. Disable SGR-Pixel mode in direct conhost (no ConPTY) since the Win32 console input API only provides cell coordinates. Add ITerminalApi::IsConhost() to detect this case. DECRQM reports PermanentlyDisabled for mode 1016 in direct conhost. Remove the fake pixel coordinate approximation from inputBuffer.cpp. 6. Update tests to use the float API and verify virtual pixel math (e.g., center of cell (0,0) at float pos (0.5, 0.5) produces virtual pixel output (6, 11) in 1-based SGR format). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

sebgod · 2026-03-09T23:20:48Z

Hey thanks for the valuable feedback @lhecker and @j4james. All these coordinate transformations do my head in, so I did had to ask my buddy Claude for help again. There is also still the problem that the only build/dev box I can build this on is still broken. Is it possible to have those CI builds enabled?. Also one thing I was not sure about is the clamping to (0, width-1) etc, seems to be a bit hairy with floats, maybe should it use IEEERemainder instead?

lhecker · 2026-03-11T16:39:28Z

src/terminal/adapter/adaptDispatch.cpp

+        // SGR-Pixel mode requires real sub-cell pixel coordinates. It's not
+        // supported in direct conhost because the Win32 console input API only
+        // provides character-cell coordinates. In ConPTY mode (IsVtInputEnabled),
+        // the request is passed through to the Terminal frontend which has pixel data.
+        if (!_api.IsConhost() || _api.IsVtInputEnabled())


It should be relatively easy to add conhost support in this PR actually, even if we have to refactor some code to achieve this. conhost is not limited by what the console API supports, after all - those are separate concerns (internal VS external API).

lhecker · 2026-03-11T16:41:11Z

src/cascadia/TerminalCore/Terminal.cpp

-    return _getTerminalInput().HandleMouse(viewportPos, uiButton, GET_KEYSTATE_WPARAM(states.Value()), wheelDelta, state);
+    const auto viewport = _GetMutableViewport().ToOrigin();
+    viewportX = std::clamp(viewportX, 0.0f, static_cast<float>(viewport.Width() - 1));
+    viewportY = std::clamp(viewportY, 0.0f, static_cast<float>(viewport.Height() - 1));


I believe this is wrong. Let's say the viewport is 1*1 cell large. This code would clamp the float value to between 0 and 0, permitting no fractional positions within that one cell.

Additionally, the clamping isn't quite identical to what it was before. The ToOrigin().Clamp() would essentially intersect the viewport-relative mouse coordinates with the scrollbar-relative buffer viewport. Thus, the coordinates would get correctly clamped.

When I'm writing this, I realize, however, that the old code seemingly doesn't shift the cursor-coordinates according to the scroll offset (= when the user has scrolled slightly upward, clicking into the partially visible VT viewport should translate the mouse coordinates back down the scroll offset). I think I'll have to check out this PR locally.

w.r.t to scrolling and just to clarify the coordinate system: What this whole change really means in terms of VT command output:
Cell size: 10x20
SGR 1006 reports mouse at 7,11 (row, col)
SGR 1016 would report something like 75,230 (pixels), in case when the mouse pointer is exactly in the centre, correct?

This is what my small test program spits out at least: https://gist.github.com/sebgod/9b9a38a7d934907584978c9bafdd0489

sebgod force-pushed the feature/sgr-pixels-1016 branch from 1845cbd to b3bf440 Compare March 6, 2026 01:08

sebgod mentioned this pull request Mar 7, 2026

Add SGR-Pixels (mouse mode 1016) #18591

Open

schrmh mentioned this pull request Mar 7, 2026

DEC Locator mode #19954

Open

lhecker requested changes Mar 9, 2026

View reviewed changes

microsoft-github-policy-service bot added the Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something label Mar 9, 2026

microsoft-github-policy-service bot removed the Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something label Mar 9, 2026

lhecker reviewed Mar 11, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add SGR-Pixels mouse mode (1016)#19949

Add SGR-Pixels mouse mode (1016)#19949
sebgod wants to merge 2 commits intomicrosoft:mainfrom
sebgod:feature/sgr-pixels-1016

sebgod commented Mar 6, 2026

Uh oh!

sebgod commented Mar 6, 2026

Uh oh!

j4james commented Mar 8, 2026

Uh oh!

lhecker Mar 9, 2026 •

edited

Loading

Uh oh!

sebgod commented Mar 9, 2026

Uh oh!

lhecker Mar 11, 2026

Uh oh!

lhecker Mar 11, 2026

Uh oh!

sebgod Mar 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

sebgod commented Mar 6, 2026

Summary of the Pull Request

References and Relevant Issues

Detailed Description of the Pull Request / Additional comments

What is SGR-Pixels (mode 1016)?

Implementation details

Validation Steps Performed

PR Checklist

Uh oh!

sebgod commented Mar 6, 2026

Uh oh!

j4james commented Mar 8, 2026

Uh oh!

lhecker Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sebgod commented Mar 9, 2026

Uh oh!

lhecker Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

lhecker Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

sebgod Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lhecker Mar 9, 2026 •

edited

Loading