Scenario: Windows Terminal 2.0 Process Model Improvements

[zadjii-msft]

This scenario is tracking a number of enhancements we'd like to make to the fundamental process model of the Windows Terminal in v2.0. The following deliverables all have something to do with the way the Terminal is launched, or the way connections or instances are managed, and as such, the solution to any one of these areas should make sure to take into account the others.

Window / Session Managment

• Add support for tab tearoff and tab merge #1256 Tab Tearoff/ Reattach
  • Spec for tab tear off and default app #2080 is a draft spec for tearoff and default terminal app (below)
  • Presumably this will involve some way to communicate different terminal instances from one window to another.
    • Proposed ideas include a manager process that's hosting all the terminal instances and connections, and then separate windows for actually drawing these instances on the screen
• Run wt in the current window
  • See Add support for wt.exe to run commands in an existing Terminal Window #4472.
  • This is being included because it ties strongly with the above deliverable.
  • See Add spec for adding commandline arguments to wt.exe #3495 / #607 - Commandline Arguments for the Windows Terminal.md, which also brought this idea up.
  • This involves finding some way for the user to specify on the commandline "I want to perform [some command] in [the current window | a specified other WT window | a new WT window]"
  • ✔️ Implemented in Add support for running a commandline in another WT window #8898
• Single Instance Mode
  • See Feature Request: Option to only allow one instance #2227
  • This would be an important parallel work thread for Feature request: hot key drop down ala quake/guake/tilda #653 and Change Windows OS to support default terminal [defterm] #492
  • ✔️ Implemented in Add support for the windowingBehavior setting #9118
• Feature request: hot key drop down ala quake/guake/tilda #653 Quake-style dropdown
  • Again, included in this list for it's fundamental strong ties to session management.
  • Often mentioned in the discussion of quake-style dropdown is the need for a single-session mode, so pressing the global hotkey would always activate the single instance, and new windows would instead glom to the existing instance
  • See [MEGATHREAD] Quake Mode & Global Summon #8888 for follow-up tasks related to Quake mode

Elevation

• Megathread: sudo, runas, mixed elevation of tabs, etc. #1032 Mixed Elevation
  • This is a hard problem, and I'm making no firm commitments that we'll be able to solve it for sure in 2.0.
  • The goal of this deliverable is to find a safe way that we can support both elevated (High-IL) and unelevated (Medium-IL) tabs, panes, etc all in the same window.
  • Ideally, if a user creates a new elevated tab from an unelevated window, then all the unelevated tabs would seamlessly appear in a new elevated HWND hosting both unelevated and elevated tabs simultaneously.
  • Elevated connections can't be hosted in an unelevated HWND, because that allows for a trivial escalation-of-privilege vector utilizing the Terminal.
  • This problem should arguably be investigated first. If this is something that's technically possible, it will certainly have an influence on the way the rest of these deliverables are architected.
  • ❌ Resolution: This is not going to be technically achievable, unfortunately.
• [Question] Configuring Windows Terminal profile to always launch elevated #632 Open a Profile as elevated
  • If the above is not achievable technically, then the user should still be able to specify a way to force a profile to open elevated.
  • If mixed elevation isn't possible:
    • If the current window isn't currently elevated, open a new window when someone's profile is marked "elevated: true"
    • If the current window is elevated, open in the same window.

Default Terminal

• Change Windows OS to support default terminal [defterm] #492 Default Terminal Application
  • This one's a little different than the others. While the rest were Windows Terminal specific features, this deliverable is more of a OS feature than a Terminal feature.
  • This concerns the ability for an arbitrary application to be registered with the OS as the "Default Terminal Application", and be started automatically when the user starts an arbitrary commandline application.
  • This also has some interaction with the above deliverables. When a commandline application is started this way, should they open as a new tab in an existing Terminal window, or start a new window?

Specs

• Spec for Windows Terminal Process Model 2.0 Spec for Windows Terminal Process Model 2.0 #7240
• Spec for Windows Terminal Window Management Spec for Windows Terminal Window Management #8135
• Spec for Elevation QOL improvements Spec for Elevation QOL improvements #8455
• Default Terminal Spec Default Terminal Spec #7414
• Quake mode spec Spec for Quake Mode #9274

TODOs

These are also tracked in https://github.com/microsoft/terminal/projects/5. This provides a nested list, to mentally track which things are dependent upon other things. This is basically a copy of a page of my notebook.

Window Management

• Window Management, PR: Add support for running a commandline in another WT window #8898
  • Next/Prev window keybindings
  • Windowing Behavior, PR: Add support for the windowingBehavior setting #9118
    • Smart wt -w 0 handling
    • Name windows on commandline
      • nameWindow action (Add support for renaming windows #9662)
      • openWindowRenamer action (Add support for renaming windows #9662)
      • window-level toast for displaying window ID, action: idenfifyWindows (Add an action for identifying windows #9523)
        • Originally I thought I'd need generic toasts for this, but I'm gonna just do this first, and come back to generic toasts
• Quake Mode has been moved to [MEGATHREAD] Quake Mode & Global Summon #8888
• New Window Action PR: Add support for the newWindow action #9208

Tear Out

Currently planned state, notes:

See #5000 (comment) for the most up-to-date todo list. There were some dramatic changes to our plans in early 2023, which basically entirely obsoleted "Process Model 2" (where each termcontrol is its own process) in favor of "Process Model 3" (where all windows are all one process).

Original tear-out plans

• wt.exe accepts --content guid on the commandline to start in content mode. Additionally, Make the Control able to have the Core OOP.
  • these two were originally different bullet points, but I think they need to be combined into one singular PR. They don't make sense alone.
  • in content process mode, it registers ContentProcess for that GUID with CoRegisterClassObject
  • ContentProcess::Initialize(IControlSettings, IConnection) instantiates a new ControlInteractivity (in the content process) (if one hasn't already been initialized)
  • ContentProcess::RequestSwapChainHandle(pid) will duplicate the handle if the pid is different
  • wt properly tracks the lifetime of its content. When that's discarded by the last window process, we'll exit.
  • Do the above line, but better. Now we're using a lock and an event, which is ew. Instead, we should do the Conhost ExitThread() thing on the main thread, and then ExitProcess() when the content process is dtor'd.
  • Move all the chaos I've introduced in main.cpp into its own dedicated file/class or something.
  • An embedding process needs a way to know that the content process is ready, better than a Sleep(2000)
    • A handle to an event is probably the cleanest way - have the window create the event, then pass the handle in CreateProcess & on the commandline to wt --content {guid} --signal {handle}, then have the content process set the event when it's ready.
  • 📝 in dev/migrie/oop/the-whole-thing dev/migrie/oop/infinity-war
  • The Connection is made in the core
  • When the Core signals that the swapchain changed, and the TermControl asks what the swapchain handle is, the Control passes in it's PID. If the PID isn't the same, DuplicateHandle the swapchain to that PID.
  • The sample is updated with a version of the Control that's also OOP
  • If the content process dies, the control needs to be able to display a message box.
  • Make sure UIA signaling still works (accevent.exe in the SDK, C:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\x64)
  • Make sure that reading the contents of the buffer with Accessibility Insights still works
    • 💩
    • InteractivityAutomationPeer extends Windows.UI.Xaml.Automation.Peers.AutomationPeer. But we're making the interactivity peer in the content process, not in the control layer.
    • When we construct the IAP, we throw instantly, because the WUX..AutomationPeer can't be created off the main thread.
    • How do we only expose the WUX..AutomationPeer via TermControl, but not IAP?
  • The _raiseContentDied should display the error internal to the TermControl. Kinda like the renderer warning, but for this, we need to kill the SwapChainPanel too.
  • in dev/migrie/oop/the-whole-thing dev/migrie/oop/infinity-war
  • Relevant diff: 23a19c5...5c17603
  • Add a velocity flag to gate behind dev builds only. Probably safe to just gate the --content flag for now.
  • Guard calls to _interactivity, _core in TermControl. Those objects might go out of scope at any moment, so pretty much all access needs to be try/caught.
    • we may want to do a function-level try, with a CATCH_RAISE_DIALOG() macro we write to raise the connection died notice. Might be the most minimal diff.
    • Could likely be a follow up to original PR.
    • There's a silly idea in dev/migrie/oop/2/cptn-marvel. I really hate it, it's a HUGE amount of churn.
  • There's a lot of debug spew that makes me think the OOP core does something off the main thread.
    • BriAl and I synced - this is nothing to worry about. This is basically just COM logging "i'm about to increment the refcount"
    • Still trying to find a viable way to suppress that logging.
  • What the hell did I do with the automation peer. Was what was in main correct? Revisit the original PRs for inspiration.
    • Did it ever work? there's seemingly no record of it working in 23a19c5...5c17603
    • Only access ControlInteractivity through the projection #10051 has a fairly different implementation than the above diff
    • I may have simply forgotten that narrator / UIA uses exceptions for control flow, which makes debugging a pain
  • ControlCore::_IsClosing TODO!
  • ContentProcess::RequestSwapChainHandle tracelogging TODO!
  • TermControlAutomationPeer::GetLocalizedControlTypeCore that string used to come from localized resources, it can't anymore.
  • The Control properly manages the lifetime of the HANDLE duplicated to it. A wil::unique_handle?
  • IMPORTANT: For whatever reason, when the second window attaches to the content, it renders basically nothing. Input gets sent just fine, but the output doesn't render.. Fixed in 7b3ca83
  • 📝 Now moved to dev/migrie/oop/2/infinity-war
  • 👀 in Add support for running the terminal core as a content process #12938
• TermControls need to have a local TerminalSettings instance. (Related: Make all the ICoreSettings / IControlSettings properties getters-only #7219)
  • What a shockingly simple oversight. We did all this work to get the connections in the same proc as the TermControl, we totally forgot the Settings.
  • Make sure updating/changing settings works fine across processes
  • Make sure previewing the color scheme still works. That works with some parent/child trickery that won't work OOP.
  • 📝 in dev/migrie/oop/ragnarok
• Some misc cleanup before endgame:
  • existingConnection in NewTab was redundant / unused
  • _SplitPane renamed to _SplitPaneActiveTab etc.
  • Refactoring to allow the TerminalTab to be initialized without a control
  • Other stuff to, to minimize the diff
  • not started, in dev/migrie/oop/2/black-widow cause this one's not really consequential, is it now.
• Make the app able to do the OOP instantiation.
  • Notes in Scenario: Windows Terminal 2.0 Process Model Improvements #5000 (comment)
• Make the sample able to pop a control into a new window.
  • ✔️ in dev/migrie/oop/the-whole-thing. I didn't pop it into a new window, just made it so multiple windows could connect to the same one.
• Defterm is out of proc.
  • DEFTERM: Incoming connections need to be tossed to the ContentProcess. Guh.
  • not started in dev/migrie/oop/2/captain-falcon.
• Tabs can be serialized to a json blob, with their tab & pane structure.
  • See Enable Terminal to persist / restore instance settings #766, because this is highly relevant
  • This may have already been done for me by Persist window layout on window close #10972
  • Serialize TerminalTabs
  • Serialize SettingsTabs
• move-pane -w <id> to move a pane/tab to another Terminal window. This will prove we can move panes without the whole instantiation of a new window process.
  • Notes in Scenario: Windows Terminal 2.0 Process Model Improvements #5000 (comment)
  • 📝 in dev/migrie/oop/wandavision (link to relevant diff from dev/migrie/oop/endgame)
• When a tab is torn out of a Terminal window, it can be dropped & re-attached onto another terminal window
  • Notes in Scenario: Windows Terminal 2.0 Process Model Improvements #5000 (comment)
  • 📝 in dev/migrie/oop/2/loki
• When a tab is torn out of a Terminal window, it can be dropped somewhere else and turned into a new terminal window
  • not started yet, in dev/migrie/oop/no-way-home
• When a tab is torn out, we'll create a new window for the tabs left behind, and move the current window around.
  • We'll need to still be able to drop this window on an existing window somehow.
  • Use EvanK's prototype here. We're gonna need to extend the drag bar over the TabView and do everything ourselves
  • No padding in tab well to move the window #12616
  • not started yet, in dev/migrie/oop/multiverse-of-madness

other notes

Just moving these lower in the body to make managing this doc easier

This is a pseudo design that Dustin and I discussed. There's a lot more work that needs to be done here, but I need to save this somewhere outside of a Teams chat history

---
struct TerminalControl: winrt::implements<MUX::IUIElement> {} // XAML Control
// calls: interactivity.PointerMoved(Point{x, y});
// calls: interactivity.PointerClicked(Point{x, y});
// links: TerminalCore.lib <----> or consumes TerminalCore.dll from RT
^^^ TerminalControl.dll ^^^

---
struct WPFTerminalControl { HWND _h; }
// calls: interactivity.PointerMoved(Point{x, y});
// calls: interactivity.PointerClicked(Point{x, y});
// links: TerminalCore.lib
^^^ WPFTerminalControl.dll ^^^

vvv TerminalCore.lib vvv // Cancelled. TerminalCore.lib or .dll
struct Coord {};
struct PointerInfo {};
struct KeyInfo {};
struct TerminalControlInteractivity;
struct ControlCore; // < implements ... something? // this should just be TerminalCore?
struct TerminalCore; // < implements the "conhosty APIs (ITerminalDispatch, etc.)
// links: dx.lib buffer.lib types.lib

vvv TerminalCore.UnitTests.dll vvv
// contains: interactivity tests
// contains: core tests
// links: TerminalCore.lib

• Split TerminalControl into lib and dll
• Split TermControl into Control.TermControl and Control.ControlCore and Control.Interactivity
• Add some Core.ControlCore tests from things in Tests we should maybe write one day #7001
• Create a WinRT boundary on Core.Interactivity between it and TermControl
• Enable Core.ControlCore to exist out-of-proc with XAML, or in-proc
  • This might involve creating a scratch project to consume an in-proc one and an out-of-proc one
• the Core can create connections, or have them passed in. (so an OOP core can have a connection in it's own process)
• Much more. This is just what's immediately on my radar.

_4-30-21_:

I need to do the DX `HANDLE` thing before I can do the rest of the projection boundary. The projection boundary is in `dev/migrie/interactivity-projection-000`, which is almost done now.

dev/migrie/oop-scratch-4 has the final state of the OOP prototyping on this element. In that branch,

• HostAndProcess::CreateSwapChain creates a swapchain with DCompositionCreateSurfaceHandle.
• It attaches it to the SwapChainPanel with ISwapChainPanelNative2::SetSwapChainHandle.
• It duplicates that handle to the content process, and calls HostClass::ThisIsInsane
• Later on, the HostClass passes that handle to the DxEngine

So we'll need to do basically that, but a little different, to handle all the actual edge cases we have.

pre-1.10 era work

• First, update DxEngine to accept the handle, but don't otherwise change anything (Use DComp surface handle for Swap Chain management #10023)
  • Then, change TermControl to create the handle with DCompositionCreateSurfaceHandle, hook it up to the SwapChainPanel with ISwapChainPanelNative2::SetSwapChainHandle.
    • DxEngine::_CreateDeviceResources will need to trigger a callback in the Core, that will then ask the TermControl to create a new swapchain for it.
    • Wait no that's wrong. the DxEngine will always be making the swapchains, because it knows when it needs a new one.
  • DxEngine will always make new swapchains, when needed, with DCompositionCreateSurfaceHandle. It'll raise an event when that happens. The Core will handle that, then raise an event the control will listen to. The Control will handle that event, then ask the Core for the new swapchain HANDLE, as a uint64_t. The Control will use that uint64_t to attach to the SwapChainPanel via ISwapChainPanelNative2::SetSwapChainHandle.
    • This is also the way it is in (Use DComp surface handle for Swap Chain management #10023)
  • 👀 in Use DComp surface handle for Swap Chain management #10023
• Then we'll come back to the rest of the projection
  • 👀 in Only access ControlInteractivity through the projection #10051
• At this point, we should make a sample app that's just a grid with two spaces in it. One for an in-proc term control, and one for an out of proc one. We won't do the OOP one yet.
  • 👀 in Add a Scratch.sln for prototyping #10067
• Merge Fix a number of shutdown crashes in TermControl #10115 into Add a Scratch.sln for prototyping #10067
• Enable the Connection able to be created OOP from the rest of the TerminalPage. The content process will be the one to instantiate the connection.
  • The Core is passed a (guid? String?) of a type of connection to spawn, and an IConnectionSettings object. The Core instantiates a type of connection based on that guid/string, and passes the IConnectionSettings from the window process to that new constructor.
  • How does the azure connection work? I think that one's implemented in TerminalApp. If it's a String, we could pass the winrt name, ITerminalConnection conn = RoCreateInstance("TerminalApp.AzureConnection"); conn.Create(iConnectionSettings); , right?
    • No, turns out the Azure connection is in TerminalConnection. So this was actually fine the whole time.
    • Whatever, we're keeping the RoGetActivationFactory thing though, for future needs.
  • How does the debug tap work?
  • Switch from RoActivateInstance&Initialize to RoGetActivationFactory

    • Even neglecting the Windows Runtime, this isn’t even possible in C++ or C#.

  • Reattach event handlers that I disabled for performance's sake
  • Merge Fix a number of shutdown crashes in TermControl #10115 into this branch, or into dev/migrie/oop/connection-factory-rebase, which is attempting to rebase these changes onto dev/migrie/oop/sample-project
  • 📝 in dev/migrie/oop/connection-factory
• 🦶 PUNTED Move the Core, the Interactivity, into Core.dll. We'll need that for:
  • We may not actually need this. It'd be nice to not have to load all the settings, but we could live with it.