View and control a connected environment's desktop alongside the agent

[patryksztuczka]

T3 Code's remote environments and mobile access already work really well for me. I can work from my main personal laptop while running agents on other machines, and check in from my phone. My main laptop effectively becomes the place where I coordinate the work.

The missing piece is occasional manual access to the other machine's desktop. Some apps, accounts, and results are only available there. For example, I might need to open Outlook on my work laptop to check a particular email, or inspect a design an agent created in a desktop app running on that laptop. I can direct the agent from T3 Code, but I still have to go back to the other laptop to check the result or make a few clicks.

I'd like to open the desktop of the selected connected environment in a panel beside the conversation, look at what I need, click or type, then continue working from my main machine. The agent could keep running, with pausing available when I need to avoid conflicting input. Opening the panel should follow the environment already selected in T3 Connect, without looking up another URL or signing in separately.

This would make quick checks and small manual interventions part of the same remote workflow that T3 Code already supports. Mobile access would be useful too, though the prototype below was tested between two Macs.

I've built a working experimental prototype in a fork. It uses T3 Connect and VNC/noVNC to open and control the remote desktop inside T3 Code. It proved the flow works, but the experience is still far from ideal. Cursor feedback, input latency, scrolling, and trackpad gestures all need attention. There is performance and UX work to do before it would be comfortable for everyday use.

Before investing more time, I'd like to understand whether this fits your direction:

• Is quick manual access to a connected environment's desktop something you would want in T3 Code?
• If so, what is the smallest useful scope you would support, and how should it relate to the live window preview mentioned in feat: let Claude and other agents use apps and browser tabs on your Mac #15006?
• Would you prefer an integration with existing remote desktop software or a user-installed helper? I'm open to changing the prototype's approach rather than committing to VNC/noVNC.

I'm willing to work on this, but would like to agree on the direction and scope with maintainers first.