Pop out remote screen

[mtlynch]

I received a request from a custom to offer the ability to pop out the remote screen to its own window. I know this is a feature that's available in VM managers like Proxmox, and it could be a useful feature in TinyPilot.

We should have a View option that pops out a minimal browser window that displays just the remote screen without any other menus, similar to what Proxmox does in displaying a new console connection.

Proxmox also causes the browser to hide the forward/back buttons and bookmark bar, which I'm not sure how they're doing but it would be good to apply similar behavior.

This is a bit similar to #523, but they're not quite the same feature. But implementing one would likely bring us very close to the other. (From thinking about this more, I feel like they effectively are the same feature)

[jotaen4tinypilot]

Notes to self from the POC branch:

• Consider wording of menu item. Candidate: “Dedicated Window”
• Disable main remote screen while popup is active
  • It’s probably too complicated to track the open state of the popup dynamically, so we could just turn off the main remote screen and say something along the lines of: “Remote screen disabled due to dedicated window. Refresh page to get back the remote screen.” The user could of course still refresh the main window while the popup is open, but that’s okay, since this is only a convenience feature.
  • We also need to “mute” the keyboard/mouse forwarding from the main window while the popup is open. (We currently don’t have such functionality yet, so we need to implement it first.)

More notes to self:

• Remove stray <div> from index.html

[jotaen4tinypilot]

I just realized that (while the popup is open) in addition to hiding the remote screen and “muting” keyboard input on the main window, we also should consider what to do with other interactive elements on the main window:

• On-screen keyboard: should the user be able to use the on-screen keyboard, to send keystrokes to the target machine?
• Menu items: should the user be able to use actions from the menu? E.g. Virtual Media, changing the hostname, but also pasting, or sending “Alt+Ctrl+Del”?

Unless you have a clear preference here @mtlynch (since I don’t right now), we could consider to defer these questions, and only start out with hiding the remote screen and muting the keyboard input in the first step. Then we could see how it feels in real life, and optimize the experience later.

[mtlynch]

@jotaen4tinypilot - I think we can leave those as-is now.

My thinking with hiding the remote screen and muting keystrokes is that the duplicate screens would likely confuse users, and the user might not expect keystrokes to forward when they don't see the remote screen.

I'm less concerned with menus and other stuff because it's hard to interact with them accidentally, and I'd imagine if the user is interacting with them, they expect them to have their normal effect. Like, there's no reason to click Actions > Keyboard Shortcuts > Ctrl + Alt + Del if you expect that the functionality is disabled due to the popout screen.

[jotaen4tinypilot]

I ran into a pretty tricky issue here, unfortunately. The good news is, however, that we might be able to work around the problem with a pretty pragmatic, unconventional approach. Therefore, I wanted to discuss things first, before opening a PR.

Problem

The issue is around disabling the remote screen on the main window. I realized that this isn’t just a mere convenience or a matter of aesthetics, but it’s actually a technical necessity to prevent duplicate video streaming, and hence duplicate bandwidth consumption.

Disabling the MJPEG stream is relatively simple, because the MJPEG stream stops as soon as we’d remove the src attribute of the <img> element. It’s not that simple for H.264, however: Janus appears to continue delivering the full video signal, no matter what I tried, be it pausing the <video> element via JS, removing the <video> element from the DOM, or even removing the entire <remote-screen> component from the DOM. It seems as if we’d have to issue a separate procedure to get Janus to stop sending data. To make matters worse, the Janus connection itself isn’t controlled from within the <remote-screen> component, but it’s actually having its own lifecycle that’s entirely facilitated via the webrtc-video.js script.

So apart from the fact that I couldn’t figure out so far how to make Janus stop reliably, I’m also worried that the control flows in the code would become complex and brittle. I eventually realized that the entire code of the frontend is set up under the implicit assumption that the app is running perpetually as long as the window is open. We don’t have any explicit teardown mechanisms for stopping keyboard input, stopping video, or deliberately disconnecting from the backend.

We also make some pragmatic assumptions in the code, such as that there is either MJPEG or H.264 enabled, but we don’t cover the case where it’s neither of those, i.e. when video isn’t available at all. Once we’d halt and hide the remote screen on the main window, we can’t know what the current streaming mode is, and hence the streaming-mode information in the status bar would be outdated or unreliable.

I’m not sure it’s worth to try to implement such teardown mechanisms, and harden the app against all these sorts of quirks, only for the sake of displaying a placeholder after opening the dedicated window. We wouldn’t need these mechanisms and control flows anywhere else at the moment, so I’d expect a rather high price in effort, code complexity, and error-proneness, for a rather low overall value.

Proposed Solution

To avoid the hassle that might stem from the dual window situation described above, we could provide the following experience:

Screen.Recording.2023-08-24.at.20.08.10.mov

This branch contains a working implementation of this approach.

• When clicking “Dedicated Window” in the “View” menu, we open the popup window with ?viewMode=standalone.
• In the main window, we issue an external (!) redirect to /dedicated-window-placeholder, which is a new and completely separate page. That’s how we’d effectively accomplish the “teardown” of the main window, and get all the following things for free:
  • All video signal is sure to be stopped
  • We aren’t connected to the web socket endpoint anymore
  • No keystrokes, mouse, or touch events can be forwarded
  • The status bar and menu items are gone, so we don’t risk showing inconsistent information
• Since it might feel weird for the user to see /dedicated-window-placeholder (or whatever) as path in their address bar, the dedicated placeholder page dynamically erases the path from the URL history right away. That effectively makes it look as if the user stayed on the main page, and /dedicated-window-placeholder also wouldn’t appear in the browser history.
• Since the URL continues to be /, the user can refresh their page as normal, to get back to the “real” TinyPilot web app.

I think the price in terms of code complexity is fairly low, and most of the “ugliness” of this approach is contained in one single file (dedicated-window-placeholder.html). Apart from some additional commentary, the branch would basically be good to go as is.

Placeholder Wording

Completely independent of the technicalities, it would be cool if you could revise the text of the placeholder message. My current, rather ragged draft is this:

The remote screen has been opened in a dedicated window.
You can close this window to continue using TinyPilot in the dedicated window.
If you want to restore this window to its original state, please refresh the page.

I find it borderline verbose, and it’s also confusing to talk about “this window” and “the dedicated window”, since it sounds very similar and might create ambiguity.

[mtlynch]

Oof, lots of hidden complexity in features this cycle!

I like your proposed solution. It feels hacky in principle but the implementation is pretty tidy and unobtrusive. We always have the option to do a more elegant teardown later if we feel like we have to.

it would be cool if you could revise the text of the placeholder message.

How about this?

The remote screen is displaying in another browser window.

To restore the remote screen in this window, please refresh the page.