ComfyClaw PyAutoGUI Agent

Experimental modular ComfyUI/ComfyClaw workflows for a persistent PyAutoGUI desktop agent.

The project is built around a Conscious/Subconscious agent loop with desktop perception, PyAutoGUI action, short-term memory, long-term embedding memory, planner/thinker branches, sleep/forgetting, emotion/role updates, and bounded cleanup.

Prototype Status

This is a prototype agent workflow. It is designed to show how this kind of agent could work, not to present a finished, fast, production-ready agent.

The current workflow is not optimized for speed and is expected to require roughly 16 GB to 24 GB of VRAM depending on the models and settings used. It has not been fully tested end to end.

I do not currently have the time to make it as efficient or polished as I would like, so I am looking for people who are willing to help test, improve, and experiment with the workflow. If you want to work with me, or just want to ask questions, you can reach me on LinkedIn:

Steven Hammon

Repository Status

This repository is currently a workflow development package. It contains:

• The complete agent specification: Agent_Spec0-06.md
• Agent prompt/config/state seed files under Agent_files/
• Modular lean workflow exports: module1_workflow.lean.json through module14_workflow.lean.json
• Full assembled ComfyUI workflow exports under workflows/
• Earlier reference workflows under Examples/

Requirements

• ComfyUI
• ComfyUI Manager, recommended for installing custom nodes
• ComfyClaw custom nodes: Steven-Hammon/ComfyClaw
• A local LLM and embedding setup matching the models configured in Agent_files/VARS.json, or equivalent model names substituted there
• A GPU with enough VRAM for the selected models. This prototype is currently expected to need roughly 16 GB to 24 GB of VRAM.

Recommended Repository Name

Recommended GitHub slug:

ComfyClaw-PyAutoGUI-Agent

This keeps the meaning of ComfyClaw_PyAutoGUI_Agent while using the hyphen style that reads cleanly in GitHub URLs.

Suggested GitHub description:

Prototype ComfyUI/ComfyClaw PyAutoGUI desktop agent workflow with memory, planning, and long-term recall.

Suggested GitHub topics:

comfyui, comfyclaw, pyautogui, agent, workflow, local-llm, memory, automation, prototype

Workflow Files

The module files are intentionally separate so each part of the agent can be loaded, tested, cleaned up, and re-exported independently from ComfyUI.

File	Purpose
module1_workflow.lean.json	Setup, paths, providers, startup reads
module2_workflow.lean.json	Conscious prompt and LLM
module3_workflow.lean.json	Conscious response cleanup/router
module4_workflow.lean.json	PyAutoGUI body path
module5_workflow.lean.json	Planner
module6_workflow.lean.json	State update routing
module7_workflow.lean.json	Sleep and forgetting
module8_workflow.lean.json	LTM query and retrieval
module9_workflow.lean.json	Thinker
module10_workflow.lean.json	Event merge and Last Responses append
module11_workflow.lean.json	Emotion and Role
module12_workflow.lean.json	STM summariser
module13_workflow.lean.json	LTM creator, duplicate check, and storage
module14_workflow.lean.json	Cleanup

The full assembled workflow is included in both standard ComfyUI JSON and lean JSON form:

workflows/PyAutoGUI_Agent_Full_Workflow.json
workflows/PyAutoGUI_Agent_Full_Workflow.lean.json

The full workflow contains an Agent_files root path node. After cloning, edit that node to point at the local Agent_files directory for the target machine.

Agent Files

Agent_files/ contains the agent's prompt and runtime seed structure:

• IDENTITY.md
• VARS.json
• PATHS.json
• prompts/
• state/
• memory/
• media/current/Current.png
• media/pyautogui_assets/
• Workspace/

The paths in PATHS.json are relative to the Agent_files root.

Current.png is included because the workflow expects a startup/current image to exist. Other runtime screenshots, logs, downloads, local build notes, and draft helper scripts are ignored by .gitignore so local machine state does not get uploaded accidentally.

Important Notes

• Lean workflow JSON files must be UTF-8 without BOM. A BOM can stop the workflow compiler before it reads the JSON.
• LLM calls in these workflows use generous max_output_tokens values. Earlier testing showed that a 128-token cap could prevent local models from completing useful output.
• The full workflow is the practical runnable artifact. The modules are the maintainable source pieces used to inspect and improve the system.
• The workflow cleaner/compiler is treated as an external tool and should live in its own repository. This repository stores the workflows produced by that tool.
• Workflow outlines and draft generator scripts are local development artifacts and are not part of the GitHub upload set.
• This is an experimental desktop-control agent. Review permissions, tool access, and local safety boundaries before running it on an unattended machine.

License

MIT License. See LICENSE.