Teleop restructuring

[ruthwikdasyam]

Teleop Device Categories

teleop stack divided into device categories, each with its own base module:

1. VR Headsets (Browser-based WebXR)
   Meta Quest
   Apple Vision Pro
   PICO
2. Smart Devices (Browser-based / Native App)
   iPhone IMU
   Smart watches (Maybe)
3. Hand Controllers (USB/Bluetooth)
   Joysticks (single/dual stick)
   Game controllers (Xbox, PlayStation)
   SpaceMouse
   Keyboard

Design Rationale: Each category has distinct input modalities (6DoF tracking vs IMU vs discrete axes), so a single base class for all devices would be bloated. Instead, each category has its own base module (VRBaseModule, SmartDeviceBaseModule, HandControllerBaseModule) with shared interfaces where appropriate.

class VRBaseModule(Module):
    """Base for all VR headset teleop devices."""
    
    # Lifecycle
    def start(self) -> None: ...
    def stop(self) -> None: ...
    
    # Teleop session (toggled by button press)
    def start_teleop(self) -> None: ...
    def stop_teleop(self) -> None: ...
    
    # Core logic
    def calibrate(self) -> bool:
        """Reset t0, store P0/R0 as reference. Called on teleop start."""
    
    def compute_delta(
        self,
        controller_poses: list[PoseStamped | None],
    ) -> list[PoseStamped | None]:
        """Compute delta from calibrated reference. Always PoseStamped."""
    
    def convert_to_output(
        self, 
        pose: PoseStamped, 
        output_type: type
    ) -> PoseStamped | TwistStamped:
        """Convert to target output type at publish boundary."""

class QuestModule(VRBaseModule):
    """Meta Quest teleop via WebSocket/LCM bridge."""
    
    # Inputs (from Deno bridge)
    left_transform: In[LCMTransform]
    right_transform: In[LCMTransform]
    left_trigger: In[Float32]
    right_trigger: In[Float32]
    teleop_enable: In[Bool]  # X button
    
    # Outputs (user configurable - PoseStamped or TwistStamped)
    left_controller: Out[PoseStamped] 
    right_controller: Out[PoseStamped]
    left_trigger_value: Out[Float32]
    right_trigger_value: Out[Float32]
    
    # Callbacks
    def _on_teleop_enable(self, msg: Bool) -> None:
        """Toggle start_teleop/stop_teleop on button press."""
    def _on_transform(self, index: int, transform: Transform) -> None: ...
    def _on_trigger(self, index: int, msg: Float32) -> None: ...
    
    # Main loop
    def control_loop(self) -> None:
        """Compute deltas, convert to output type, publish."""

• X press to start, and X press to stop. When pressed X again, it starts from zero

[leshy]

tofix

    # Inputs (from Deno bridge)
    left_transform: In[LCMTransform]
    right_transform: In[LCMTransform]
    left_trigger: In[Float32]
    right_trigger: In[Float32]
    teleop_enable: In[Bool]  # X button

If you are already outputing these from Deno bridge directly (are you using our js lib? nice!)
there is no need to re-process here. make your deno bridge actual QuestModule no?

    left_trigger: In[Float32]
    right_trigger: In[Float32]

also don't use float use uint32 or uint64

I assume uint32 is enough for everything you want and can just output
buttons:

buttons: Out[Uint32]

again not sure why these are Inputs and not Outputs on a quest module

    def _on_transform(self, index: int, transform: Transform) -> None: ...

on transform receives transforms from where? I thought QuestModule generates transforms

    left_transform: In[LCMTransform]
    right_transform: In[LCMTransform]

never receive or send raw transforms, use self.tf for transform sending btw, grep our docs for self.tf

Buttons bitmask decoder

of course you want a nice parsing on the receiver end, I want to feed buttons uint and get actual human understandable format

state = Quest.buttonstate(integer_value)

print(state.left.x.pressed)

# etc, invent a good api

Just so you and others know this is possible (and best api for this imo)
you can locally subclass our Uint32 in your quest code

class QuestButtons(Uint32):
    @property
    def left_y() -> bool:
        ... parse self

and you have

Out[QuestButtons]

it can be easy to construct if you write setters also

buttonstate = QuestButtons()
if quest.y_pressed:
    buttonstate.left_y = True

receiver can define In[QuestButtons] and will receive your parsed type as input ready to check button states on

general arch suggestion

I'd actually start with the simplest possible quest module

Quest(Module):
    # also emits transforms
    left_controller: Out[PoseStamped]
    right_controller: Out[PoseStamped]
    buttons: Out[Uint32]

then you can subclass for different fancy controls,
like this one translates poses into twist in some fancy way that's useful

class MustafasFavoriteQuest(Quest):
    left_controller: Out[Twist]
    right_controller: Out[Twist]
    buttons: Out[Uint32]

This one processes posestamped as I proposed
https://discord.com/channels/1341146487186391173/1357833666751107174/1463780585859186782

class PoseQuest(Quest):
    left_controller: Out[PoseStamped]
    right_controller: Out[PoseStamped]
    buttons: Out[Uint32]

etc

control suggestions

X press to start, and X press to stop. When pressed X again, it starts from zero

IMO irl it's always more convinient to work only when button is held, otherwise you can press the button, then a fly runs into your VR headset and you kill mustafa with an arm, but your call

In terms of how you actually treat pose changes/button holding I assume 90% of the cases you want this behaviour

https://discord.com/channels/1341146487186391173/1357833666751107174/1463780585859186782

but these are suggestions, pretty sure above is what manip guys would want but I'd let them decide

[ruthwikdasyam]

If you are already outputing these from Deno bridge directly (are you using our js lib? nice!)
there is no need to re-process here. make your deno bridge actual QuestModule no?

• Yes, using js lib.
• Keeping teleop_server.ts intentionally dumb, it just forwards Websocket -> LCM without processing, thus QuestTeleop can be a proper module reading inputs, and streaming out PoseStamped
  Device-specific logic lives in Python modules.

---

I assume uint32 is enough for everything you want and can just output

• Subclassing as QuestButtons(Uint32) for readable accessors

---

on transform receives transforms from where? I thought QuestModule generates transforms

The flow is:

Quest Headset -> WebXR (browser) -> WebSocket -> Deno bridge -> LCM -> QuestModule

• QuestModule receives raw transforms from the Deno bridge - it doesn't generate them. The Deno bridge forwards WebXR controller data over LCM (we switched from JSON to LCM for the WebSocket - Python communication)

---

I'd actually start with the simplest possible quest module

It's easy to create a sub-class as needed upon standard base. Seems like a good feature to have

IMO irl it's always more convinient to work only when button is held

One reason why i wanted just a press to start/stop is - I am able to get controller poses with bare hands, no controllers.
Thus, start/stop is gonna be with touching thumb and index button (acts as X press). As we cannot replicate press and hold X with hands, I prefered this way.

Although, with 'adding sub-class as needed' approach, i should be able to provice both

[ruthwikdasyam]

Updated Spec v2 — Post-Review Implementation

Architecture

Quest 3 Browser (WebXR ~80Hz)
        │
        ├─ PoseStamped (left/right controller)
        ├─ Joy (buttons + analog axes)
        │
        ▼
   Deno Bridge (teleop_server.ts)
   WebSocket ⟷ LCM packet relay (no processing)
        │
        ▼
   LCM Network
        │
        ▼
   QuestTeleopModule (Python, 50Hz control loop)
        │
        ├─ left_controller_output: Out[PoseStamped]   (delta)
        ├─ right_controller_output: Out[PoseStamped]   (delta)
        └─ buttons: Out[QuestButtons]                  (bitmask)

TeleopProtocol

Structural interface — all teleop modules (Quest, keyboard, joystick) implement this. No base class, just the contract:
Might refine once other devices are added

@runtime_checkable
class TeleopProtocol(Protocol):
    def start(self) -> None: ...
    def stop(self) -> None: ...
    def engage(self, hand: Any = None) -> bool: ...
    def disengage(self, hand: Any = None) -> None: ...

QuestButtons(UInt32)

Packed bitmask for both controllers. Readable attribute API:

class QuestButtons(UInt32):
    BITS = {
        "left_trigger": 0, "left_grip": 1, "left_touchpad": 2,
        "left_thumbstick": 3, "left_x": 4, "left_y": 5, "left_menu": 6,
        "right_trigger": 8, "right_grip": 9, "right_touchpad": 10,
        "right_thumbstick": 11, "right_a": 12, "right_b": 13, "right_menu": 14,
    }

# Usage:
buttons = QuestButtons.from_controllers(left_state, right_state)
print(buttons.left_x)       # True/False
print(buttons.right_grip)   # True/False

Analog values (trigger, grip) thresholded at 0.5 for bitmask. Full floats preserved in QuestControllerState for subclasses that need proportional control.

QuestTeleopModule (Base)

class QuestTeleopModule(Module[QuestTeleopConfig], TeleopProtocol):

    # Inputs (from Deno bridge via LCM)
    vr_left_pose:  In[PoseStamped]
    vr_right_pose: In[PoseStamped]
    vr_left_joy:   In[Joy]
    vr_right_joy:  In[Joy]

    # Outputs
    left_controller_output:  Out[PoseStamped]   # delta pose
    right_controller_output: Out[PoseStamped]   # delta pose
    buttons:                 Out[QuestButtons]   # packed bitmask

    # RPC Methods
    @rpc start()                          # Subscribe inputs, start control loop
    @rpc stop()                           # Stop control loop
    @rpc engage(hand) -> bool             # Capture initial pose, return False if no data
    @rpc disengage(hand)                  # Stop publishing for hand
    @rpc get_status() -> QuestTeleopStatus

Callbacks (LCM → state):

• _on_pose(hand, PoseStamped) → webxr_to_robot() → stores in _current_poses
• _on_joy(hand, Joy) → QuestControllerState.from_joy() → stores in _controllers

5 Override Points (called with lock held — don't re-acquire):

Method	Default	Override to...
_handle_engage()	Primary button hold = engage	Toggle, grip-based, etc.
_should_publish(hand)	return is_engaged[hand]	Add conditions (trigger gate)
_get_output_pose(hand)	current - initial (delta)	Absolute pose, scaling, filtering
_publish_msg(hand, pose)	Publish PoseStamped	Convert to Twist, scale
_publish_button_state(l, r)	Pack → QuestButtons → publish	Different format, extra streams

---

Subclasses

QuestTeleopModule              ← hold-to-engage, PoseStamped delta
    ├── ArmTeleopModule        ← toggle engage (X/A rising edge)
    │   └── VisualizingTeleopModule  ← + Rerun debug visualization
    └── TwistTeleopModule      ← outputs TwistStamped with linear/angular scaling

ArmTeleopModule — overrides _handle_engage():

• Tracks _prev_primary per hand
• Rising edge (not pressed → pressed) toggles engage/disengage
• X press to start, X press to stop — re-engage starts from zero (new initial pose)

TwistTeleopModule — overrides _publish_msg():

• Converts delta PoseStamped → TwistStamped
• linear = position * linear_scale, angular = orientation.to_euler() * angular_scale
• Publishes to left_twist: Out[TwistStamped] / right_twist: Out[TwistStamped]

VisualizingTeleopModule — overrides start() + _get_output_pose():

• Initializes Rerun viewer on start
• Logs controller poses as 3D transforms + axes
• Logs button states as scalar time series

---