HandBrowser

HandBrowser is an Apple Vision Pro experiment that lets you open and control a floating browser with your left hand.

The core idea is simple: make an L shape with your left hand using your thumb and index finger, and a browser appears in space. The browser can be positioned relative to the hand gesture, giving it a lightweight, sci-fi feel that is a little closer to “summoning” UI than opening a traditional app window.

This project started as a prototype, but it also explores a bigger idea: using natural hand gestures to reveal small, focused utility interfaces inside a spatial app.

Why this project exists

Most browser experiences in spatial computing still feel like standard 2D windows placed in 3D space. HandBrowser explores a different interaction model:

• UI that feels invoked, not launched
• a browser that can feel more like a tool than a separate app
• gesture-driven interactions that fit the more physical language of Vision Pro

It is also a useful foundation for other Vision Pro projects that need:

• lightweight in-app control surfaces
• spatial debug panels
• floating utility views
• remote web-based control interfaces

Current prototype

HandBrowser currently focuses on a few core pieces:

• detects a custom left-hand L pose
• opens or reveals a floating browser in a Vision Pro immersive experience
• supports a URL bar so you can navigate to different websites
• experiments with attaching the browser to the hand or placing it nearby in space
• uses web content as a fast, flexible UI surface inside a visionOS project

How it works

At a high level, the project combines:

• ARKit hand tracking to detect a custom gesture
• RealityKit / SwiftUI attachments to place UI in space
• WebKit to render browser content
• a spatial interaction model where the hand acts as the trigger for a lightweight floating interface

The result is a small browser that can feel more like a summoned object than a standard window.

Example use cases

HandBrowser can be useful as a prototype on its own, but the bigger value is as a pattern for other spatial apps.

1. Quick in-app browser

Open documentation, dashboards, help pages, or internal tools without switching context.

2. Spatial debug panel

Use a small web-based admin or debug interface while testing a Vision Pro experience.

3. Demo companion UI

Pull up a live browser for a supporting web experience during presentations, demos, or prototype walkthroughs.

4. Remote control surface

Use a web UI as a live controller for objects or state inside a Vision Pro app.

5. Utility overlay for immersive apps

Surface small-purpose tools such as checklists, media controls, scene settings, or operator controls without turning them into full app windows.

One especially interesting direction

A natural extension of this idea is a web control panel that can drive behavior inside a Vision Pro app.

For example:

• a web front end sends changes to Firebase
• the Vision Pro app listens for realtime updates
• those updates immediately change something in the spatial scene

That could be used for things like:

• switching modes in a demo
• advancing scenes in a guided walkthrough
• changing lighting, media, or content states
• triggering animations or events
• controlling a prototype from an iPhone, browser, or another device

In that setup, HandBrowser could become either:

• the in-headset version of the control panel, or
• the foundation for apps that respond to remote operator input

Potential future enhancements

There are a lot of directions this could go.

Interaction improvements

• smoother gesture recognition
• a clearer visual cue when the gesture is detected
• better open/close behavior based on pose confidence
• support for resizing, docking, or pinning the browser
• hand-following modes vs. fixed-in-space modes

Browser improvements

• back/forward/reload controls
• bookmarks or preset URLs
• a home screen with app-specific shortcuts
• support for internal web tools and dashboards
• safer domain restrictions for demo or enterprise use

Spatial UI ideas

• summon other tools using different gestures
• create a family of “micro utilities” that open from the hand
• support a phone-sized view, a tablet-sized view, or a pinned panel mode
• animate the browser in and out so it feels more alive and intentional

Multi-device workflows

• use an iPhone as a paired controller
• use a browser dashboard to control the Vision Pro experience remotely
• sync state across devices through Firebase or another realtime backend
• create operator / viewer modes for installations, demos, or presentations

Broader platform uses

• internal enterprise tools
• spatial prototyping workflows
• XR training scenarios
• live presentations and guided demos
• exhibit or museum installations
• remote-assisted workflows

Why web content matters here

One of the most practical parts of this concept is that the browser is not just “a browser.” It is a fast way to bring flexible UI into a spatial app.

A web-based control surface gives you:

• rapid iteration without rebuilding everything in visionOS
• easy cross-device support
• a clean path for remote control or operator dashboards
• the option to reuse existing web tooling inside a Vision Pro experience

That makes HandBrowser interesting not only as a novelty, but as a development pattern.

Project status

This project is an active prototype and exploration of spatial interaction patterns for Apple Vision Pro.

The current goal is not to build a full standalone browser. The goal is to explore how gesture-invoked, web-powered utility surfaces can feel more natural and useful inside spatial computing experiences.

Ideas to explore next

• a polished “phone in hand” mode
• a pinned floating browser that detaches from the hand after summon
• a remote operator dashboard backed by Firebase
• app-specific web tools for demos, scene control, and content switching
• a reusable framework for summoning other web-based panels in Vision Pro apps