KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill

Lychee Team, Harbin Institute of Technology, Shenzhen

Personal Assistant for desktop, Android, iOS, and HarmonyOS.

简体中文 · Paper · Website · Demo · CLI reference · Adapter contract ·

You can see the real-device experimental logs and trajectories at here

KnowAct-GUIClaw + open-source Kimi-2.6 achieves a state-of-the-art 64.1% on the long-horizon MobileWorld benchmark, outperforming all open agent frameworks and closed agents (Seed-2.0-Pro, GPT-5.5). The framework's knowledge memory and execution capabilities generalize to various base models: +8.5% on Kimi-2.6 and +16.2% on Qwen3.5-35B-A3B.

GUIClaw organizes personal GUI assistance as a Know–Route–Act–Reflect loop.

The current Python distribution is named nanobot-ai and ships both packages. The standalone path uses only the guiclaw executable at runtime; guiclaw does not import nanobot.

Requirements: Python 3.11+, uv, a multimodal model, and the tooling required by the selected backend.

git clone https://github.com/HITsz-TMG/KnowAct.git
cd KnowAct/GUIClaw
uv sync --extra web --extra desktop --extra cjk
uv run nanobot onboard --wizard

Add a gui block to ~/.nanobot/config.json. The provider must also be configured for nanobot.

{
  "providers": {
    "custom": {
      "apiKey": "your-api-key",
      "apiBase": "https://api.example.com/v1"
    }
  },
  "gui": {
    "backend": "adb",
    "provider": "custom",
    "model": "your-vision-model",
    "agentProfile": "default",
    "maxSteps": 15,
    "enableSkillExecution": true,
    "enablePromptSkillSelection": true,
    "promptSkillTopK": 5,
    "promptShortcutOnly": false,
    "promptSkillAppFilter": false
  }
}

Start the WebUI:

uv run nanobot webui

The host registers GUIClaw as gui_task. It routes screen work to GUIClaw and keeps files, shell, web, MCP, memory, and other tools in the host runtime.

Clone the repository, then install the command from the repository root:

git clone https://github.com/HITsz-TMG/KnowAct.git
cd KnowAct

# Android, HarmonyOS, or dry-run
uv tool install ./GUIClaw

# Use this instead for local desktop automation
# uv tool install './GUIClaw[desktop]'

Create ~/.guiclaw/config.yaml:

provider:
  base_url: "https://api.example.com/v1"
  model: "your-vision-model"

max_steps: 15
stagnation_limit: 0
image_scale_ratio: 0.5
agent_profile: default

Export the API key and run a smoke test:

export OPENAI_API_KEY="your-api-key"

guiclaw --dry-run "Describe the current screen and finish"
guiclaw --backend adb "Open Settings and enable Wi-Fi"

For another agent or script, request JSON output:

guiclaw --backend adb --json --task "Open Contacts and search for John"

See the CLI and configuration reference for installation extras, every flag, complete defaults, backend setup, shortcut validation, and subprocess integration.

GUIClaw keeps its runtime data outside the nanobot workspace:

When no POLICY entry exists, the first GUI task initializes a conservative permission policy: deny, cancel, or defer requests unless the task explicitly authorizes them. Users may edit the structured entry in policy.md. These rules are model guidance, not guaranteed enforcement; use OS, backend, host approval, or sandbox controls for mandatory restrictions.

Common defaults:

The complete standalone and adapter field tables are in GUIClaw CLI and Configuration Reference.

Static inference writes one cache file per package. Runtime validation can then launch candidates on an ADB device and optionally use a vision model before promotion:

export DASHSCOPE_API_KEY="your-api-key"

guiclaw shortcuts ~/.guiclaw/shortcut_cache/com.example.app.json \
  --validate \
  --promote \
  --llm-base-url https://dashscope.aliyuncs.com/compatible-mode/v1 \
  --llm-model qwen3.5-flash \
  --llm-api-key-env DASHSCOPE_API_KEY

Promotion writes eligible records to ~/.guiclaw/skill/skills.py. Validation launches intents and can change application state; use a test device when possible.

Linux background runs use Xvfb. Windows can use an isolated desktop when the target application is supported. macOS currently supports foreground desktop automation only.

The CLI supports:

default, general_e2e, gui_owl, venus, seed, qwen3vl, mai_ui, and gelab.

Use default for providers with reliable native function calling. Other profiles reproduce the action format expected by their corresponding GUI model families.

Pass@1 success rates reported in the paper:

KnowAct/
├── README.md
├── README_CN.md
└── GUIClaw/
    ├── guiclaw/       # Host-independent GUI runtime
    ├── nanobot/       # Bundled agent host and GUI adapter
    ├── webui/         # Browser workbench
    ├── tests/         # Python tests
    ├── docs/          # CLI, nanobot, and developer documentation
    └── pyproject.toml # nanobot and guiclaw entry points

cd GUIClaw
uv sync --extra dev --extra desktop --extra cjk
uv run pytest
uv run ruff check nanobot guiclaw tests

WebUI development additionally requires Bun:

cd GUIClaw/webui
bun install
bun run test
bun run build

• GUIClaw framework overview
• CLI and configuration reference
• Adapter patterns
• nanobot documentation
• Contributing
• Security
• License
• Third-party notices

If you use KnowAct-GUIClaw in research, please cite:

@misc{li2026knowactguiclawknowdeeplyact,
  title        = {KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill},
  author       = {Yunxin Li and Jinchao Li and Shibo Su and Zhenran Xu and Chenrui Zhao and Tongshu Bian and Xiaoman Liang and Meishan Zhang and Baotian Hu and Min Zhang},
  year         = {2026},
  eprint       = {2607.12625},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url          = {https://arxiv.org/abs/2607.12625}
}

GUIClaw builds on and draws inspiration from the following open-source and research projects:

• HKUDS/nanobot provides the lightweight agent host, provider, channel, tool, and configuration foundation used by the bundled distribution.
• Tongyi-MAI/MobileWorld informed the mobile GUI agent profiles, action conventions, and benchmark-oriented workflows.
• stepfun-ai/gelab-zero informed GELab model integration and practical mobile GUI agent runtime patterns.
• google-research/reasoning-bank inspired the experience-driven memory induction used to retain reusable lessons from GUI trajectories.

We thank the authors and contributors of these projects for making their work available to the community. Each project remains subject to its own license and attribution requirements; see the linked repositories and nanobot's third-party notices for details. Mention here denotes technical influence or reuse where documented, not official affiliation or endorsement.

GUIClaw is distributed under the MIT License. The repository retains nanobot's original LICENSE and THIRD_PARTY_NOTICES.md, with additional attribution in NOTICE. GUIClaw is an independent project and is not an official HKUDS/nanobot distribution.