Interactivity bugs post-#9820

[zadjii-msft]

This is a tracking thread for things I probably broke in #9820. I'd rather not file 100 bugs for these

• (A) Mousedown, select, SCROLL does a weird thing with endpoints that doesn't happen in stable
  • fixed in 4f4df01
  • fixed in First three interactivity fixes #9980
• (B) Trackpad scrolling down with small increments seems buggy. But scrolling with a big increment goes fast? And scrolling with the mouse is fine. And trackpad scrolling in mouse mode is fine.
  • fixed in First three interactivity fixes #9980
• (C) Looks like there's a selection issue when you click and drag too quickly. Seems like the start endpoint doesn't always get set.
  • fixed in 9d33e89
  • fixed in First three interactivity fixes #9980
• (D) Mysterious DX crash - this never reprod, so we're assuming it's unrelated

Mysterious DX crash

I experienced a crash when I killed DWM, and it took particularly long to recover

crash was on:

            RETURN_IF_FAILED(_dxgiFactory2->CreateSwapChainForComposition(_d3dDevice.Get(),
                                                                          &_swapChainDesc,
                                                                          nullptr,
                                                                          &_dxgiSwapChain));

in

0:000> k
 # Child-SP          RetAddr               Call Site
00 00000043`cef8ec80 00007ff9`3a4ac5a1     Microsoft_Terminal_Control!Microsoft::Console::Render::DxEngine::_CreateDeviceResources+0x47b [E:\BA\242\s\src\renderer\dx\DxRenderer.cpp @ 630] 
01 (Inline Function) --------`--------     Microsoft_Terminal_Control!Microsoft::Console::Render::DxEngine::GetSwapChain+0x14 [E:\BA\242\s\src\renderer\dx\DxRenderer.cpp @ 1010] 
02 00000043`cef8ed50 00007ff9`3a55667d     Microsoft_Terminal_Control!winrt::Microsoft::Terminal::Control::implementation::ControlCore::GetSwapChain+0x21 [E:\BA\242\s\src\cascadia\TerminalControl\ControlCore.cpp @ 1183] 
03 00000043`cef8ed80 00007ff9`3a4e1f71     Microsoft_Terminal_Control!winrt::Microsoft::Terminal::Control::implementation::TermControl::RenderEngineSwapChainChanged$_ResumeCoro$1+0x2bd [E:\BA\242\s\src\cascadia\TerminalControl\TermControl.cpp @ 582] 
04 (Inline Function) --------`--------     Microsoft_Terminal_Control!std::experimental::coroutine_handle<void>::resume+0x9 [C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29910\include\experimental\coroutine @ 107] 
05 (Inline Function) --------`--------     Microsoft_Terminal_Control!std::experimental::coroutine_handle<void>::operator()+0x9 [C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29910\include\experimental\coroutine @ 99] 
06 00000043`cef8ee50 00007ff9`5e2623a6     Microsoft_Terminal_Control!winrt::impl::resume_apartment_callback+0x11 [E:\BA\242\s\src\cascadia\TerminalControl\Generated Files\winrt\windows.ui.core.h @ 1088] 
07 00000043`cef8ee80 00007ff9`5e2af93f     Windows_UI!<lambda_59517943c03487243f9bea31c6c1a784>::operator()+0x84 [onecoreuap\windows\advcore\winrt\onecoreiwindow\corewindow\common\dispatcher.cpp @ 908] 
08 00000043`cef8eeb0 00007ff9`7de6a440     Windows_UI!Microsoft::WRL::Details::DelegateArgTraits<long (__cdecl Windows::System::IDispatcherQueueHandler::*)(void)>::DelegateInvokeHelper<Microsoft::WRL::Implements<Microsoft::WRL::RuntimeClassFlags<2>,Windows::System::IDispatcherQueueHandler,Microsoft::WRL::FtmBase>,<lambda_59517943c03487243f9bea31c6c1a784>,-1>::Invoke+0xf [onecore\external\sdk\inc\wrl\event.h @ 354] 
09 00000043`cef8eee0 00007ff9`7de4229c     CoreMessaging!Windows::System::DispatcherQueue::DeferInvokeCallback+0x20 [mincore\coreui\dev\dispatcherqueue\wrtdispatcherqueue.cpp @ 903] 
0a (Inline Function) --------`--------     CoreMessaging!Microsoft::CoreUI::ActionCallback::ImportAdapter$::__l2::<lambda_a81ff790741c2a62f2197c2561f5fe49>::operator()+0x1f [mincore\CoreUI\Dev\System\Api\IExportMessageSession.cs @ 22] 

the _dxgiFactory2 was null. That's unexpected. It shouldn't be null.

• (E) It doesn't seem like we're calling this when we create the automation peer -- we are not guaranteed to always call UpdateSettings after creating a peer, so the first peer may have no padding. See Only access ControlInteractivity through the projection #10051 (comment). (fixed by Initialize the padding for the Control UIA provider #10874)

[DHowett]

Looks like there's a selection issue when you click and drag too quickly.

(Sorry for the terse body; I have to post this using powershell because of a new MSIT rule about personal devices :|)

[zadjii-msft]

Looks like there's a selection issue when you click and drag too quickly.

There sure is, just hit like 4 times this morning. Added to the list ☺️

[zadjii-msft]

haha, found it

if (_singleClickTouchdownPos) { // Figure out if the user's moved a quarter of a cell's smaller axis away from the clickdown point auto& touchdownPoint{ *_singleClickTouchdownPos }; auto distance{ std::sqrtf(std::powf(cursorPosition.X - touchdownPoint.X, 2) + std::powf(cursorPosition.Y - touchdownPoint.Y, 2)) }; const til::size fontSize{ _actualFont.GetSize() }; const auto fontSizeInDips = fontSize.scale(til::math::rounding, 1.0f / _renderEngine->GetScaling()); if (distance >= (std::min(fontSizeInDips.width(), fontSizeInDips.height()) / 4.f)) { _terminal->SetSelectionAnchor(_GetTerminalPosition(touchdownPoint)); // stop tracking the touchdown point _singleClickTouchdownPos = std::nullopt; } }

if (_singleClickTouchdownPos) { // Figure out if the user's moved a quarter of a cell's smaller axis away from the clickdown point auto& touchdownPoint{ *_singleClickTouchdownPos }; float dx = ::base::saturated_cast<float>(pixelPosition.x() - touchdownPoint.x()); float dy = ::base::saturated_cast<float>(pixelPosition.y() - touchdownPoint.y()); auto distance{ std::sqrtf(std::powf(dx, 2) + std::powf(dy, 2)) }; const auto fontSizeInDips{ _core->FontSizeInDips() }; if (distance >= (std::min(fontSizeInDips.width(), fontSizeInDips.height()) / 4.f)) { _core->SetSelectionAnchor(terminalPosition); // stop tracking the touchdown point _singleClickTouchdownPos = std::nullopt; } }

        _terminal->SetSelectionAnchor(_GetTerminalPosition(touchdownPoint));

vs

        _core->SetSelectionAnchor(terminalPosition);

We're now using the location of the drag event as the selection anchor, instead of the location that the user initially clicked. Oops. Now to write a test.

[zadjii-msft]

Oof. B is hard, I think. The touchpad sends very many small scroll deltas, less than one row at a time. The control scrollbar can store a double, so small deltas can accumulate. Originally, these would accumulate in the scrollbar, and we'd only read that out as an int in the scrollbar updater, which is throttled.

In the interactivity split, there's no place for us to store that double. We immediately narrow to an int for ControlInteractivity::_updateScrollbar.

Theoretically, we could store a double in Interactivity that's a fake scrollbar position, to accumulate to. Then, ControlInteractivity would need to listen to scroll events from the core, unsure how easy that is right now...

[DHowett]

Oh yes, that is frightening for sure.

This is a wackadoo idea, but... what if the scrollbar's size was WHEEL_DELTA * BufferHeight and we just ... let the deltas accumulate and only scroll every WHEEL_DELTA units

[DHowett]

No no it would probably be absolutely wack for the UI to display that (or Accessibility would say "line 300 of 1991661")

[DHowett]

Point [E] may cause an accessibility regression in the 1.11 timeframe.

[ghost]

🎉This issue was addressed in #10874, which has now been successfully released as Windows Terminal Preview v1.11.2421.0.:tada:

Handy links:

• Release Notes
• Store Download