Add gamepad controller module with Ports & Adapters architecture#10
Open
research-developer wants to merge 8 commits intomainfrom
Open
Add gamepad controller module with Ports & Adapters architecture#10research-developer wants to merge 8 commits intomainfrom
research-developer wants to merge 8 commits intomainfrom
Conversation
Introduces a comprehensive controller module for gamepad-based terminal
and agent orchestration. This enables navigation and control of iTerm
sessions using game controller inputs.
## New Components
- **GamepadEmulator** (`emulator.py`): Software mock of Xbox-style controller
with buttons, d-pad, analog sticks, and triggers
- **ControllerMapper** (`mapper.py`): Maps controller inputs to actions
with preset profiles for terminal navigation and Vim mode
- **Action Executors** (`executors.py`): Multiple backends including
PrintExecutor, AppleScriptExecutor, and ITermExecutor
- **GamepadMenu** (`menu.py`): Interactive selection menus navigable
with d-pad and buttons
- **NotificationCenter** (`notification_center.py`): macOS-style popup
triggered by HOME button, showing notifications/teams/agents
- **AgentHub** (`agent_hub.py`): Unified agent communication platform
with LB/RB cycling, quick messages, and team broadcasts
- **TeamWindowManager** (`team_windows.py`): Groups agents by team into
shared iTerm windows with layouts (QUAD, SPLIT, etc.) and role-based
tab colors
- **ITermMCPClient** (`mcp_client.py`): Python interface for iTerm MCP
integration with caching
## Controller Layout
```
LT: Quick Msg [LB] [RB] RT: Team Broadcast
D-pad [Y] Read Output
↑ ← ↓ → [X] Send Command
[B] Back
L-Stick [A] Select/Focus
[SELECT] Teams [HOME] Notifications [START] Actions
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR introduces a comprehensive gamepad controller module for orchestrating iTerm sessions and managing agent communications. The implementation includes ~5,300 lines of code across 11 new files, providing software-emulated Xbox-style controller functionality mapped to terminal operations.
Key additions:
- Software gamepad emulator with full button/stick/trigger support
- Configurable input-to-action mapping with preset profiles (terminal, vim)
- Multiple execution backends (print, AppleScript, iTerm)
- Interactive menu and notification center with gamepad navigation
- Agent communication hub and team window management
- Python interface for iTerm MCP integration
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 30 comments.
Show a summary per file
| File | Description |
|---|---|
| selectron/controller/emulator.py | Core gamepad emulation with state tracking and event callbacks |
| selectron/controller/mapper.py | Maps controller inputs to configurable actions and profiles |
| selectron/controller/executors.py | Action executors for different targets (print, AppleScript, iTerm) |
| selectron/controller/menu.py | Interactive menu system with gamepad navigation |
| selectron/controller/notification_center.py | macOS-style notification center with agent/team views |
| selectron/controller/agent_hub.py | Central hub for agent communication and control |
| selectron/controller/team_windows.py | Groups agents by team into shared iTerm windows |
| selectron/controller/mcp_client.py | Python client for iTerm MCP server integration |
| selectron/controller/iterm_bridge.py | Bridge between gamepad and iTerm terminal sessions |
| selectron/controller/demo.py | Interactive keyboard-to-gamepad demo implementation |
| selectron/controller/init.py | Module initialization with comprehensive exports |
| selectron/init.py | Updated imports to reference renamed app scanner components |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7ff45d6de9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Remove unused imports and add explanatory comments to except clauses: - agent_hub.py: Remove time, json, subprocess, Tuple, Any, Union, Path, DPadDirection, NotificationCenter, NotificationLevel, Agent imports - demo.py: Remove threading, Optional imports - executors.py: Remove Any, Dict imports - iterm_bridge.py: Remove sys, json, Any, Callable, Dict, Thread, Event imports; prefix unused variable with underscore - mapper.py: Remove GamepadState import - mcp_client.py: Remove json, subprocess, field imports - menu.py: Remove field, Callable, Dict imports - notification_center.py: Remove json, Tuple, Dict, MenuOption imports - team_windows.py: Remove auto import, consolidate asyncio imports at module level Add explanatory comments to empty except KeyboardInterrupt clauses in agent_hub.py, demo.py, and notification_center.py. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Reorganize the gamepad controller module to decouple core logic from iTerm-specific implementations, enabling future VS Code support. Architecture changes: - types.py: Shared data types (Agent, Team, Notification, MCPResponse) - interfaces.py: AgentClient protocol + MockAgentClient for testing - core/: Platform-agnostic components (emulator, mapper, menu, etc.) - bridges/iterm/: iTerm-specific adapters (client, executor, bridge) - factory.py: Dependency injection factory functions Key improvements: - Core modules no longer import from bridges (clean dependency flow) - MockAgentClient enables testing without iTerm connection - ITermAgentClient implements AgentClient protocol - All existing exports preserved for backward compatibility - ITermMCPClient/get_mcp_client aliases maintained 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
research-developer
changed the title
Owner
Author
|
@copilot I need you to approve this with a review |
|
@research-developer I've opened a new pull request, #11, to work on those changes. Once the pull request is ready, I'll request review from you. |
Clarify code change agent role in PR approval process
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 20 out of 21 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Integrates with the notification system for display. | ||
| """ | ||
|
|
||
| import sys |
There was a problem hiding this comment.
Import of 'sys' is not used.
Suggested change
| import sys |
| """ | ||
|
|
||
| import sys | ||
| import time |
There was a problem hiding this comment.
Import of 'time' is not used.
Suggested change
| import time |
Comment on lines
+22
to
+24
| Any, Callable, Dict, List, Optional, Tuple, Union, | ||
| AsyncGenerator, Protocol, TypeVar | ||
| ) |
There was a problem hiding this comment.
Import of 'Tuple' is not used.
Import of 'Union' is not used.
Import of 'AsyncGenerator' is not used.
Import of 'Protocol' is not used.
Import of 'TypeVar' is not used.
Suggested change
| Any, Callable, Dict, List, Optional, Tuple, Union, | |
| AsyncGenerator, Protocol, TypeVar | |
| ) | |
| Any, Callable, Dict, List, Optional | |
| ) |
| from datetime import datetime | ||
| from threading import Lock | ||
| import uuid | ||
| import json |
There was a problem hiding this comment.
Import of 'json' is not used.
Suggested change
| import json |
| Implements the AgentClient protocol from selectron.controller.interfaces. | ||
| """ | ||
|
|
||
| from dataclasses import dataclass |
There was a problem hiding this comment.
Import of 'dataclass' is not used.
Suggested change
| from dataclasses import dataclass |
| from .interfaces import AgentClient, MockAgentClient | ||
| from .core.emulator import GamepadEmulator | ||
| from .core.notification_center import NotificationCenter, create_notification_center | ||
| from .core.agent_hub import AgentHub, create_agent_hub |
There was a problem hiding this comment.
Import of 'create_agent_hub' is not used.
Suggested change
| from .core.agent_hub import AgentHub, create_agent_hub | |
| from .core.agent_hub import AgentHub |
- Create core/hardware.py with HardwareGamepad class using pygame/SDL - Auto-detect macOS and use main-thread polling (SDL requirement) - Same callback interface as GamepadEmulator for seamless integration - Support Xbox, PlayStation, Luna, and other SDL-compatible controllers - Add list_controllers() and create_hardware_gamepad() factory functions - Export hardware types from core/__init__.py and main __init__.py Tested with Amazon Luna Controller on macOS. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implements a learning subsystem that observes user actions when unmapped gamepad buttons are pressed and learns to replicate those actions using Google's ADK with Gemini vision. Key features: - 2-second hold triggers learning for new buttons - 5-second hold forces re-learning for mapped buttons - Auto re-learn after 3 consecutive failures - Visual feedback (purple/yellow/green/red flashes) - Audio feedback with pitch variations - JSON persistence at ~/.selectron/learned_actions.json New modules: - core/learner/ - LearningController, ActionStore, ButtonLearningAgent - bridges/macos/screen.py - Screen capture around cursor via Quartz Modified: - core/mapper.py - Added on_unmapped() callback hook - __init__.py - Exported learner module components Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 27 out of 28 changed files in this pull request and generated 13 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| def _run_async(self, coro) -> None: | ||
| """Run a coroutine, handling event loop creation.""" | ||
| try: | ||
| loop = asyncio.get_running_loop() |
There was a problem hiding this comment.
Variable loop is not used.
Suggested change
| loop = asyncio.get_running_loop() | |
| asyncio.get_running_loop() |
| return | ||
|
|
||
| print(f"Connecting to session: {session_id}") | ||
| _controller = GamepadTerminalController(session_id) |
There was a problem hiding this comment.
Variable _controller is not used.
Suggested change
| _controller = GamepadTerminalController(session_id) | |
| GamepadTerminalController(session_id) |
Comment on lines
+42
to
+48
| # Fallback: use AppleScript | ||
| script = ''' | ||
| tell application "System Events" | ||
| set mousePos to get position of first process whose frontmost is true | ||
| return (item 1 of mousePos) & "," & (item 2 of mousePos) | ||
| end tell | ||
| ''' |
There was a problem hiding this comment.
Variable script is not used.
Suggested change
| # Fallback: use AppleScript | |
| script = ''' | |
| tell application "System Events" | |
| set mousePos to get position of first process whose frontmost is true | |
| return (item 1 of mousePos) & "," & (item 2 of mousePos) | |
| end tell | |
| ''' | |
| # Fallback: use AppleScript (not currently implemented) |
|
|
||
| def demo(): | ||
| """Interactive demo of the agent hub.""" | ||
| import threading |
There was a problem hiding this comment.
Module 'threading' is imported with both 'import' and 'import from'.
| print() | ||
|
|
||
| # For demo, simulate with keyboard | ||
| import threading |
There was a problem hiding this comment.
Module 'threading' is imported with both 'import' and 'import from'.
| Requires: pip install pygame | ||
| """ | ||
|
|
||
| import os |
There was a problem hiding this comment.
Import of 'os' is not used.
Suggested change
| import os |
| import sys | ||
| import time | ||
| import threading | ||
| from dataclasses import dataclass, field |
There was a problem hiding this comment.
Import of 'field' is not used.
Suggested change
| from dataclasses import dataclass, field | |
| from dataclasses import dataclass |
| import time | ||
| import threading | ||
| from dataclasses import dataclass, field | ||
| from typing import Callable, Dict, List, Optional, Any |
There was a problem hiding this comment.
Import of 'Any' is not used.
Import of 'Dict' is not used.
Suggested change
| from typing import Callable, Dict, List, Optional, Any | |
| from typing import Callable, List, Optional |
| import threading | ||
| from dataclasses import dataclass, field | ||
| from typing import Callable, Dict, List, Optional, Any | ||
| from enum import Enum |
There was a problem hiding this comment.
Import of 'Enum' is not used.
Suggested change
| from enum import Enum |
| capture_output=True, | ||
| timeout=duration + 1 | ||
| ) | ||
| except (subprocess.TimeoutExpired, subprocess.CalledProcessError): |
There was a problem hiding this comment.
'except' clause does nothing but pass and there is no explanatory comment.
Address Copilot code review feedback by removing: - Unused typing imports (Tuple, Union, AsyncGenerator, Protocol, TypeVar, Any, etc.) - Unused module imports (sys, time, json, os, dataclasses.field, etc.) - Unused local variables (script, loop) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Features
Core Components (
core/)Learning System (
core/learner/)~/.selectron/learned_actions.jsoniTerm Bridge (
bridges/iterm/)macOS Bridge (
bridges/macos/)Architecture Benefits
AgentClientprotocol withMockAgentClientfor testing without iTermcreate_mock_client(),create_iterm_client())core/never imports frombridges/Controller Layout
Test plan
from selectron.controller import *)ITermMCPClient,get_mcp_client)🤖 Generated with Claude Code