Improve console resizing behavior w.r.t. large terminals

[rprichard]

Currently, if the user attempts to make the terminal too big, winpty simply issues a SetConsoleWindowInfo call with invalid size values, and the API call fails. The result is a wide buffer, and a window with whatever width it most recently had. It should probably try to do more.

At the absolute least, if the width is too large, it should still change the height.

It should probably cap the window size to the largest possible. Ordinarily, winpty tries to keep the window and buffer widths equal, but maybe it should allow the buffer width to be greater. If the buffer width is greater, winpty now scrapes the entire buffer width, but only in Scrolling Mode, not Direct Mode.

Dynamically adjusting the font sizes is also a possibility. It's probably necessary, but it can't fully solve the problem, and it creates an awkward situation where we can handle small terminals and big terminals, but not "tall and skinny" terminals.

See my analysis of IDEA-117552. The IDEA-136296 issue also has a screenshot demonstrating the cursor positioned beyond the line's visible characters.

I poked at the console size APIs a bit tonight. Tentatively, I think this is the behavior:

• If a size requirement isn't satisfied, the API fails and nothing happens. (Out-of-bounds coordinates aren't modified to be in-bounds.)
• SetConsoleWindowInfo sets the window position and size. The new window rectangle must be completely inside the buffer size, and the new size must be no greater than the current GetLargestConsoleWindowSize value. Each coordinate must be at least 1, but a 1x1 size is OK. Changing the console window position with this API does not move the graphical window. That is, the GUI window's (Left, Top) coordinate is unchanged. Rather, the (Right, Bottom) coordinate changes.
• SetConsoleScreenBufferSize sets the buffer size. The new buffer size must be >= the current window size. It is OK to exclude cells that are currently visible--Windows will automatically move the window, but it won't shrink the window. The buffer width (in columns) must be at least large enough that a window of minimum pixel width (e.g. 130 pixels) would not have more columns than the buffer. The buffer height must be at least 1.
• GetLargestConsoleWindowSize returns the largest allowed value for the console window size, in rows and columns. The console window is not allowed to be larger than the screen. Crucially, in a multi-monitor setup, the console window cannot be larger than some monitor, apparently whichever monitor is showing more of the console.

I am wondering precisely which monitor GetLargestConsoleWindowSize uses. Maybe it's the monitor containing the center-point of the window? I suspect it's possible to position a window so that the center-point is not on a monitor at all. Then what?

In a multi-monitor setup, it is possible to call SetConsoleWindowInfo to change the console size successfully, then call it again unsuccessfully using the exact same rectangle. The first call changes the monitor that the console associates with, such that the new console size is no longer valid. Given that terminal resizing is a multi-step affair in the agent, this has the potential for complicating things.

Part of the trouble here is that an IntelliJ window can span multiple monitors, but a console window generally can't. People with multiple monitors are therefore more likely to hit this issue.

I think it's possible to query monitor positions with EnumDisplayMonitors.

Aside: For an interesting example of Direct Mode, see the wmic.exe utility included with Windows. The utility resizes the buffer on startup--by making the buffer 1500 columns wide and 300 lines tall! Once in Direct Mode, if you type a line longer than the terminal width, it doesn't wrap--instead, the entire visible contents of the terminal scroll off the left side of the terminal. Pressing Home shows part of the contents again, but only part, because the cursor hasn't moved all the way to the left. Pressing Enter scrolls back left. Resizing the terminal resets the buffer width to the new terminal width, which restores ordinary wrapping behavior, but it does not end Direct Mode, because the buffer height is still 300. Exiting wmic.exe does restore Scrolling Mode, but not if the terminal has been resized in the interim.

[rprichard]

Regarding the tall and skinny problem: it seems much more important to get the width correct than the height. If we can't get both, then we could increase the font to the largest sensible amount and then show as many rows as possible. Applications will think the window is shorter than it really is, but that has a fairly minimal impact. (I think it could affect mouse Y coordinates.)

Using the console API, I think it's possible to exceed the 72pt limit that the console properties dialog imposes. Conceivably, the number of columns could then drop all the way to 1. I'm not sure this is a good idea.

[rprichard]

If I configure my console with misc/SetFont.exe -face Consolas -h 3, then each cell is 1x3px, and the widest possible console window is 1920 columns. (My monitor is 1920x1200.) The smallest font allowed by the console properties dialog is 2x5px.

I'd have to evaluate how this affects half-vs-full-width CJK characters.

[rprichard]

FWIW: the current fix doesn't pay attention to monitor, but that might be something to improve on later.

I noticed some interesting Windows APIs a while ago: MonitorFromPoint, MonitorFromRect, MonitorFromWindow. The flags to these APIs control which monitor is returned if it's ambiguous -- e.g. if the point isn't on any monitor.