REP of SP characters produces incorrect results

[jdebp]

Windows Terminal version

1.17.1023

Details

1.16.10261 had significant problems with REP of Unicode block graphics and other such characters. That's mostly fixed in 1.17.1023, with one exception: REP of Supplementary Plane characters still produces incorrect results, with the characters repeated in the wrong places on the line or sometimes not at all.

I've experienced this with MouseText characters like U+0001FB81 in particular, and have not (of course!) tested every SP code point.

I did some screenshots.

[j4james]

As currently implemented, REP just repeats the last wchar_t that was output (which on Windows is 16-bit), so for anything outside the BMP, you're just going to be repeating the second half of a surrogate pair.

If we don't want to support supplementary planes, maybe we should filter out anything in the surrogate range, rather than writing out garbage. That way apps could at least detect whether it was supported or not.

[j4james]

@lhecker Regarding your comment here: #15751 (comment)

I'm not entirely sure which part in our code base doesn't handle surrogate pairs, etc., but I suspect it can be discovered by setting a breakpoint on AdeptDispatch::_FillRect.

This issue isn't related to FillRect - it's a limitation of the StateMachine implementation, which works with 16-bit wchar_t elements. In the case of the REP operation, it's repeating the last such character. See here:

terminal/src/terminal/parser/OutputStateMachineEngine.cpp

Lines 570 to 581 in 5daf498

	case CsiActionCodes::REP_RepeatCharacter:
	// Handled w/o the dispatch. This function is unique in that way
	// If this were in the ITerminalDispatch, then each
	// implementation would effectively be the same, calling only
	// functions that are already part of the interface.
	// Print the last graphical character a number of times.
	if (_lastPrintedChar != AsciiChars::NUL)
	{
	const size_t repeatCount = parameters.at(0);
	std::wstring wstr(repeatCount, _lastPrintedChar);
	_dispatch->PrintString(wstr);
	}

If that happens to be the second half a surrogate pair, it's not going to work correctly.

But this is just one symptom of the problem. For another example, compare the difference between these two statements, which use an emoji as the final character of a CSI sequence:

printf '[\e[✅]\n'
printf '[\e[💓]\n'

The first outputs [], while the second shows an error glyph [�]. Admittedly this is a ridiculous edge case, but if you are working on a rewrite of the state machine code, it would nice if we could fix the underlying problem, and not just hack in a workaround for REP. Not essential though.