Make reliable computer use a priority for Goose

[Mikkelka]

Hi! I’d like to suggest making computer use a higher priority for Goose.

Goose is a general-purpose agent, but many everyday tasks still require interacting with applications that don’t have a convenient API or MCP integration. Being able to see what’s on the screen, use the mouse and keyboard, and check the result would make Goose much more useful beyond coding and terminal-based workflows.

I’m using Goose Desktop 1.57.0 on Windows. The Computer Controller extension is enabled, but in my session I can only see the DOCX, PDF and XLSX tools – not tools for interacting with the desktop. Goose can use Windows scripts as a workaround, but that doesn’t feel like a complete or discoverable computer-use experience.

One real-world example from my own experience: I’ve used Codex to help set up and test games on my PC. It could open a launcher, interact with its controls, start the game, navigate menus, check whether things were working, and close it again. Being able to observe and interact with the actual application made the difference between simply installing something and verifying that it worked.

I’d love to see Goose offer a similarly capable open-source computer-use experience, including on Windows. Ideally, it should work across platforms and have clear permissions and safeguards so users can stay in control.

Is computer use something the project plans to invest in? Would this be a good candidate for a community effort or the goose grant program? I’d also be interested to hear what approaches or extensions people currently recommend.

Thanks!

[antropos17]

A useful first milestone would be a Windows capability panel showing which desktop tools are actually available and why others are missing. For acceptance, I would include two windows with identical titles and a target closing mid-action: the next action must stop rather than land in another window. AI-assisted design suggestion; I have not tested this Goose setup.

[Mikkelka]

Thanks for the feedback! I’ve also been looking at existing open-source projects that might be relevant here.

Windows-MCP looks like a possible starting point for Windows: it exposes desktop interaction through MCP, including screenshots, mouse and keyboard input, and window management. I haven’t tested it with Goose yet, so I can’t say how reliable it is in practice or how well it handles games and launchers.

UI-TARS Desktop is another interesting example of visual computer use, although it’s a separate agent application rather than a drop-in Goose extension.

Would it make sense for Goose to explore integrating or recommending an existing open-source solution, rather than building everything from scratch? What I’d especially like to see is a reliable, discoverable experience where Goose can observe an application, interact with it, and verify the result – with clear permissions and safeguards.

I’d be interested to hear whether anyone has tried Windows-MCP with Goose, or knows of other projects worth looking at.