-
Notifications
You must be signed in to change notification settings - Fork 92
Port ComputerUseMixin workflow learning and replay from gaia6 #544
Copy link
Copy link
Open
Labels
agentcuaComputer Use AgentComputer Use Agentdomain:multimodalVoice (ASR/TTS), Vision (VLM), Image gen (SD), CUAVoice (ASR/TTS), Vision (VLM), Image gen (SD), CUAenhancementNew feature or requestNew feature or requestp1medium prioritymedium prioritytrack:consumer-appHermes-competitor consumer product — mobile-first, voice + messaging + memory + skillsHermes-competitor consumer product — mobile-first, voice + messaging + memory + skills
Metadata
Metadata
Assignees
Labels
agentcuaComputer Use AgentComputer Use Agentdomain:multimodalVoice (ASR/TTS), Vision (VLM), Image gen (SD), CUAVoice (ASR/TTS), Vision (VLM), Image gen (SD), CUAenhancementNew feature or requestNew feature or requestp1medium prioritymedium prioritytrack:consumer-appHermes-competitor consumer product — mobile-first, voice + messaging + memory + skillsHermes-competitor consumer product — mobile-first, voice + messaging + memory + skills
Summary
Port ComputerUseMixin from gaia6 (
agents/base/computer_use.py, ~1,176 lines). Enables agents to learn browser workflows by recording actions, then replay them with parameter substitution.Tools Provided
learn_workflow(name, url)— Record browser actions as a replayable skillreplay_workflow(name, params)— Execute a learned skill with parameter substitutionlist_workflows(filter)— List all learned skillstest_workflow(name)— Replay in visible mode for verificationArchitecture
category="skill"withmetadata.type="replay"~/.gaia/skills/{insight_id}/step_N.pngReconciliation with Existing CUA Issues
Existing issues (#224, #458-#461) approach CUA from an MCP perspective. This gaia6 approach is mixin-based with workflow learning. Need to reconcile:
Source
Port from
gaia6/src/gaia/agents/base/computer_use.pyAcceptance Criteria