Skip to content

v0.27.1

Choose a tag to compare

@Verlintas Verlintas released this 18 Sep 16:56
· 10 commits to main since this release

v0.27.1 — 深度工具强化:AI 现在能「看见」界面并精确点击

UI 自动化质变(核心)

此前 AI 只能盲猜坐标点击,不知道按钮在哪里。现在形成完整闭环:

  • ua_find_text 查看界面结构:列出屏幕上所有文字元素的坐标与可点击状态(可留空列出全部可点项)
  • ua_tap_text 按文字点击:「点登录按钮」——基于无障碍节点查找,比坐标点击可靠得多
  • screen_ocr 默认返回每行文字坐标 文字 @(x,y):AI 读到内容就知道位置,配合坐标点击精确操作
  • 全流程:看界面 → 按文字点 → OCR 验证结果,任意 App 自动化真正可用

其他深度强化

  • calculator 科学计算:sin/cos/tan/sqrt/log/ln/floor/ceil/round(角度制)、常量 pi/e、百分比(200*15% = 30)
  • get_time 日期计算:days_offset 回答「3 天后是几号星期几」
  • write_document 追加模式:分多次写长文档不再覆盖
  • media_control 查询当前曲目:歌名/艺术家/专辑/播放进度
  • read_notifications 过滤:按时间范围(最近 N 小时)+ 应用名筛选
  • fetch_rss 条目摘要:订阅条目附带内容摘要

工具总数 63。需要无障碍 + 截屏权限(设置 → 权限)。

SHA-256:

  • full: 2762820e2825d8eae8490a66c384e61d72a5ae2274da8ff6ceb3756c39b9b41e
  • lite: 03aaba5f2c39be92acc9ae59d110b24e10b85d0a3d6cc14b82925d7ac92a19dd

v0.27.1 — Deep tool upgrades: the AI can now SEE the UI and tap precisely

UI automation breakthrough

The AI used to guess coordinates blindly. Now it's a closed loop:

  • ua_find_text: lists every text element with coordinates and clickability (leave text empty to list all tappable items)
  • ua_tap_text: taps an element by its text ("tap the Login button") via the accessibility tree - far more reliable than coordinates
  • screen_ocr now returns per-line coordinates text @(x,y) by default, so the AI knows where things are and can tap precisely
  • Workflow: inspect UI -> tap by text -> verify with OCR. Real automation of any app.

Other deep upgrades

  • calculator: scientific functions (sin/cos/tan/sqrt/log/ln/floor/ceil/round, degrees), constants pi/e, percentages (200*15% = 30)
  • get_time: days_offset answers "what weekday is it 3 days from now"
  • write_document: append mode for multi-part long documents
  • media_control: status action returns the current track (title/artist/album/progress)
  • read_notifications: filter by time range (last N hours) and app name
  • fetch_rss: entries now include content summaries

63 built-in tools. Requires Accessibility + screen-capture permissions (Settings -> Permissions).

SHA-256:

  • full: 2762820e2825d8eae8490a66c384e61d72a5ae2274da8ff6ceb3756c39b9b41e
  • lite: 03aaba5f2c39be92acc9ae59d110b24e10b85d0a3d6cc14b82925d7ac92a19dd