Skip to content
BranchesTags

Latest commit

 

History

21 Commits

Folders and files

NameName
Last commit message
Last commit date
README.md
README.md
 
 

Repository files navigation

Awesome GUI Agent Datasets for Computer-Use and Phone-Use

A curated list of datasets for training GUI agents—AI systems that automate interactions with graphical user interfaces on computers, phones, and browsers. Sorted by year (most recent first).

2026

  • AndroidDaily
    • Description: A verifiable benchmark for mobile GUI agents on real-world closed-source Android applications, emphasizing automated result verification without human judges.
    • Data: Tasks across popular closed-source apps (e.g., Maps, YouTube) with executable verification scripts.
    • URL: https://arxiv.org/abs/2605.27761

2025

  • Aria-UI/Aria-UI_Data

    • Description: A comprehensive collection of GUI grounding data covering web, mobile, and desktop interfaces, designed for versatile grounding instruction understanding and context-aware grounding.
    • Data: Web Data (2.9M instructions, 173k images), Mobile Data (1.1M instructions, 104k images from AMEX), Desktop Data (150k instructions, 7.8k images from Ubuntu).
    • URL: https://huggingface.co/datasets/Aria-UI/Aria-UI_Data

  • Multimodal-Mind2Web

    • Description: A multimodal version of Mind2Web, pairing HTML documents with corresponding website screenshots to support the development of general-purpose web agents.
    • Data: 7,775 actions from 1,009 training tasks; 1,339 actions from 177 test tasks (same website), 1,019 actions from 142 test tasks (new website), 4,060 actions from 694 test tasks (new domain).
    • URL: https://huggingface.co/datasets/osunlp/Multimodal-Mind2Web

  • GUIMid

    • Description: A consolidated mid-training dataset designed to enhance the foundational agentic capabilities of VLMs for GUI tasks, by leveraging data from adjacent, non-GUI domains.
    • Data: 300,000 samples — MathInstruct (150k), CodeI/O (20k), Olympiad Math (50k), Multi-modal Math (80k).
    • URL: https://github.com/hkust-nlp/GUIMid

  • STEVE (Windows OS dataset)

    • Description: A Windows OS dataset for UI grounding, collected via Windows VM, OmniParser, screenshots, and accessibility tree data.
    • Data: 10,000 desktop images and 80,000 UI elements, augmented with AITW public data.
    • URL: https://github.com/FanbinLu/STEVE

  • Aguvis Data Collection (xlangai/aguvis-stage1 & stage2)

  • OS-Genesis

    • Description: An interaction-driven pipeline synthesizing high-quality GUI agent trajectory data without human supervision, using reverse task synthesis and a trajectory reward model.
    • Data: Raw triples <s_pre, a, s_post>, complete trajectory data, SoM (State-of-Mind) info. Covers Mobile and Web platforms.
    • URL: https://github.com/OS-Copilot/OS-Genesis

  • Mobile-R1 (PG23/Mobile-R1)

    • Description: A high-quality dataset for training VLM-based mobile agents, focused on Chinese mobile applications.
    • Data: 1,007 trajectories across 28 apps, 3,924 total steps, full interaction trajectories with screenshots and action parameters.
    • URL: https://mobile-r1.github.io/Mobile-R1/

  • ShowUI_desktop

    • Description: A vision–language–action dataset for desktop GUI element grounding with rich bounding-box and keypoint annotations.
    • Data: ~7,500 desktop screenshots across 15 apps; 8,000 element annotations with bounding boxes, spatial-relation, and intent queries.
    • URL: https://huggingface.co/datasets/Voxel51/ShowUI_desktop

  • GUI-Robust

    • Description: A dataset evaluating GUI agent performance under seven real-world anomaly conditions (occlusion, dynamic content changes, etc.), using a semi-automated collection combining RPA recordings and MLLM annotation.
    • Data: 10,000+ action sequences with seven anomaly types, task descriptions, stepwise instructions, screenshots, and grounding metadata.
    • URL: https://github.com/chessbean1/GUI-Robust

  • VideoGUI

    • Description: A multi-modal benchmark from professional instructional videos, evaluating agents across hierarchical levels: high-level planning, mid-level planning, and atomic action execution.
    • Data: 86 complex tasks (avg. 22.7 actions each), 463 subtasks, 2.7K manually annotated actions across 11 software applications.
    • URL: https://github.com/showlab/videogui

  • Mind2Web 2

    • Description: A benchmark of 130 realistic, long-horizon tasks requiring real-time web browsing and extensive information synthesis, constructed with 1,000+ hours of human labor. Introduces an Agent-as-a-Judge evaluation framework.
    • Data: 130 tasks with time-varying and complex answer structures.
    • URL: https://huggingface.co/datasets/osunlp/Mind2Web-2

  • Online-Mind2Web

    • Description: An online version of Mind2Web with tasks evaluated in real-world online environments across domains like clothing, food, housing, and transportation.
    • Data: 300 tasks from 136 websites, with task_id, website, task_description, and reference_length fields.
    • URL: https://github.com/OSU-NLP-Group/Online-Mind2Web

  • LearnGUI

    • Description: A dataset for studying illustration-based learning in mobile GUI agents, enhancing performance in unseen scenarios.
    • Data: 2,252 offline and 101 online tasks across 73 apps, with high-quality human demonstrations, screenshots, and action sequences.
    • URL: https://huggingface.co/datasets/lgy0404/LearnGUI

  • AndroidInteraction

    • Description: Focuses on user interaction needs and notifications in phone UI automation, enabling agent-initiated interactions.
    • Data: 750+ demonstrations across 250+ apps, including action-observation pairs with screenshots and accessibility metadata.
    • URL: https://arxiv.org/abs/2503.19537

  • WorldGUI

    • Description: An interactive benchmark for desktop GUI automation supporting tasks across multiple applications from any starting point.
    • Data: 611 tasks across 10 desktop and web apps, with user queries, instructional videos, and project files.
    • URL: https://github.com/showlab/WorldGUI

  • DeskVision

    • Description: Large-scale desktop region captioning dataset for advanced GUI agents, improving visual element understanding.
    • Data: Large-scale desktop GUI dataset with rich annotations for diverse UI systems and elements using automated captioning.
    • URL: https://arxiv.org/abs/2503.11170

  • GUI-Lasagne

    • Description: Multi-level, large-scale dataset for training agents like SpiritSight, focusing on GUI understanding and grounding.
    • Data: 5.73M samples, 2.24M screenshots, 57.8M elements, with a 3-tier structure for image-text alignment and navigation.
    • URL: https://arxiv.org/abs/2503.03196

  • TongUI / GUI-Net

    • Description: Builds generalized GUI agents by learning from multimodal web tutorials across multiple operating systems.
    • Data: 143K–1M trajectory data points across 5 OS and 200+ apps, with multimodal web instructions, text, and screenshots.
    • URL: https://tongui-agent.github.io/

  • ScreenSpot-Pro

    • Description: Benchmark for GUI grounding in high-resolution professional environments for multimodal LLMs.
    • Data: 1,581 task data points across 23 industries, with high-resolution screenshots, natural language instructions, and bounding box annotations.
    • URL: https://huggingface.co/datasets/Voxel51/ScreenSpot-Pro

  • WebGames

    • Description: A dataset for training agents to play web-based games, focusing on interactive GUI tasks.
    • Data: 50+ unique interactive challenges in JSONL format with 160×210px environments and text-based goals.
    • URL: https://github.com/convergence-ai/webgames

  • Explorer

    • Description: The largest-scale web trajectory dataset to date, dynamically exploring web pages to create contextually relevant tasks.
    • Data: 94K successful web trajectories, 49K unique URLs, 720K screenshots, generated by a multi-agent LLM pipeline.
    • URL: https://arxiv.org/abs/2502.11357

  • InSTA

    • Description: An Internet-scale dataset for training GUI-based web agents, generated through an automated LLM pipeline without human annotations.
    • Data: 150k diverse websites from Common Crawl, rich web navigation tasks, trajectories in Playwright API calls, LLM-based evaluation.
    • URL: https://huggingface.co/datasets/data-for-agents/insta-150k-v3

  • VideoCAD

    • Description: A large-scale synthetic dataset for learning UI interactions and 3D reasoning from CAD software, with annotated video recordings of CAD operations.
    • Data: 41K+ annotated video recordings of CAD operations (up to 20× longer time horizon than other datasets).
    • URL: https://arxiv.org/abs/2505.24838

  • AutomotiveUI-Bench-4K

    • Description: An open-source dataset for understanding and interacting with automotive infotainment systems, serving as a validation benchmark for automotive UI.
    • Data: 998 images with 4,208 annotations across 15 automotive brands (2018–2025 models); IVI UI in German and English, annotations in English.
    • URL: https://paperswithcode.com/dataset/automotiveui-bench-4k

  • OS-Atlas

    • Description: A foundational GUI action model excelling at GUI grounding and OOD agentic tasks, leveraging a large open-source cross-platform grounding corpus.
    • Data: 13M+ GUI elements across mobile, desktop, and web platforms, synthesized via a specialized toolkit. Includes ScreenSpot-v2 benchmark.
    • URL: https://osatlas.github.io/

  • UGround Web-Hybrid (UGround-V1)

    • Description: The largest GUI visual grounding dataset at release (ICLR 2025 Oral), training the UGround universal visual grounding model. Built from web-based synthetic data by OSU NLP + Orby AI.
    • Data: 10M GUI elements (~95% web) over 1.3M screenshots; Web-Hybrid split is ~8M elements / 775K screenshots. Format: screenshot + referring expression → coordinates.
    • URL: https://arxiv.org/abs/2410.05243 ; https://huggingface.co/osunlp/UGround-V1-7B

  • Jedi (with OSWorld-G benchmark)

  • AutoGUI-704k

  • AgentNet (OpenCUA)

  • OSWorld

    • Description: First scalable real-computer task environment for multimodal agents with execution-based evaluation across Ubuntu/Windows/macOS. Humans complete 72.36% vs. best model 12.24% at release. NeurIPS 2024; OSWorld-Verified released July 2025.
    • Data: 369 computer tasks, each with an initial-state config and a custom execution-based evaluation script.
    • URL: https://arxiv.org/abs/2404.07972 ; https://github.com/xlang-ai/OSWorld

  • WindowsAgentArena (WAA)

    • Description: Windows-focused, reproducible multimodal OS agent benchmark built on the OSWorld framework; parallelizable via Azure (full eval in ~20 minutes). The Navi baseline reached 19.5% vs. 74.5% human success.
    • Data: 154 diverse multi-step tasks across 7 Windows app categories (Office, browser, system settings, VS Code, media, etc.).
    • URL: https://arxiv.org/abs/2409.08264 ; https://microsoft.github.io/WindowsAgentArena/

  • VeriGUI

    • Description: First verifiable long-chain GUI dataset emphasizing subtask-level verifiability and long-horizon complexity; supports open-ended interaction within each subtask.
    • Data: Richly annotated GUI task trajectories across desktop and web (130 web task trajectories in initial release, expanding). Human-expert annotated.
    • URL: https://arxiv.org/abs/2508.04026 ; https://github.com/VeriGUI-Team/VeriGUI

  • GUI-ReWalk

    • Description: Reasoning-enhanced framework synthesizing diverse GUI trajectories via stochastic random-walk exploration plus task-guided completion. IJCAI 2026 accepted (ByteDance).
    • Data: 50k+ annotated tasks averaging ~22.5 steps, multi-platform (mobile + desktop). Pipeline released under Apache-2.0.
    • URL: https://arxiv.org/abs/2509.15738 ; https://github.com/bytedance/GUI-ReWalk

  • AgentSynth

    • Description: Scalable computer-use trajectory generation pipeline that chains executable subtasks into complex, long-horizon datasets by exploiting information asymmetry.
    • Data: Synthetic generalist computer-use tasks/trajectories (cost-efficient generation pipeline).
    • URL: https://arxiv.org/abs/2506.01968

  • WorkArena / WorkArena++

    • Description: Enterprise "knowledge worker" web-agent benchmark on ServiceNow, delivered via BrowserGym. WorkArena++ adds compositional planning and reasoning tasks. ICML 2024 / NeurIPS 2024.
    • Data: WorkArena: 29 tasks (20k+ instances); WorkArena++: compositional tasks built on the atomic set.
    • URL: https://arxiv.org/abs/2403.07718 ; https://arxiv.org/abs/2407.05291

  • BrowserGym

    • Description: Unified gym environment for web-agent research, bundling MiniWoB++, WebArena, VisualWebArena, WorkArena, WebLINX, AssistantBench under a single API with multimodal observations.
    • Data: Multimodal observations (HTML, accessibility tree, screenshot, element coordinates) with a standardized flexible action space.
    • URL: https://arxiv.org/abs/2412.05467 ; https://github.com/ServiceNow/BrowserGym

  • ST-WebAgentBench

    • Description: Benchmark for safety and trustworthiness of web agents built on WebArena/BrowserGym, covering six safety dimensions (user-consent, boundary, strict execution, hierarchy, robustness, error handling).
    • Data: 222 tasks across GitLab/ShoppingAdmin/SuiteCRM plus 646 policy instances.
    • URL: https://arxiv.org/abs/2410.06703

  • Falcon-UI

    • Description: GUI instruction dataset and model emphasizing understanding GUI context before following user instructions (Dec 2024).
    • Data: Large-scale GUI screenshot/context dataset for instruction-free pretraining across multiple platforms.
    • URL: https://arxiv.org/abs/2412.09362

  • SPA-Bench

    • Description: Comprehensive smartphone-agent benchmark with a plug-and-play execution framework, automatic evaluation pipeline, and bilingual support (English + Chinese). ICLR 2025 Spotlight (Huawei Noah's Ark Lab).
    • Data: 340 Android tasks (300 single-app + 40 cross-app) across system and third-party apps, with 7 task-completion/resource metrics.
    • URL: https://arxiv.org/abs/2410.15164 ; https://github.com/ai-agents-2030/SPA-Bench

  • AndroidLab (with Android Instruct)

  • MONDAY

    • Description: Scalable video-to-dataset pipeline producing cross-platform mobile-navigation data from YouTube instructional videos; improves cross-platform generalization. CVPR 2025.
    • Data: 20K instructional videos; 333K detected scenes; 313K annotated frames/actions; Android + iOS.
    • URL: https://arxiv.org/abs/2505.12632 ; https://github.com/runamu/monday

  • UI-Vision

    • Description: First license-permissive desktop-centric GUI benchmark for offline fine-grained evaluation with three tasks: Element Grounding, Layout Grounding, Action Prediction. ICML 2025 (Mila/ServiceNow).
    • Data: 83 desktop applications across 6 categories; dense human-demonstration annotations (bounding boxes, UI labels, click/drag/keyboard trajectories).
    • URL: https://arxiv.org/abs/2503.15661 ; https://huggingface.co/datasets/ServiceNow/ui-vision

  • WinSpot

    • Description: First GUI grounding benchmark specifically designed for Windows desktop environments. ACL 2025 (short paper).
    • Data: Windows GUI screenshots with grounding instruction → element annotations (high-quality train/eval set).
    • URL: https://aclanthology.org/2025.acl-short.85/

  • Mobile-Bench-v2

    • Description: More realistic and comprehensive VLM-based mobile agent benchmark with offline multi-path evaluation and an ad/noise-contaminated variant (AITZ-Noise) for robustness.
    • Data: Multi-path mobile trajectories with checkpoint-style metrics.
    • URL: https://arxiv.org/abs/2505.11891

  • ColorBench

    • Description: Benchmarks mobile agents using a graph-structured framework (trajectory graphs rather than rigid chains) for complex long-horizon tasks.
    • Data: Mobile long-horizon tasks with graph-structured decision modeling.
    • URL: https://arxiv.org/abs/2510.14621

  • UI-Genie (UI-Genie-RM-517k & UI-Genie-Agent-16k)

    • Description: Self-improving framework with a specialized GUI reward model; generates the first large-scale GUI reward dataset plus high-quality synthetic trajectories without manual annotation. NeurIPS 2025.
    • Data: UI-Genie-RM-517k (~458K synthetic reward samples + ~59K from exploration); UI-Genie-Agent-16k (16K synthetic mobile/Android trajectories, image-text interleaved).
    • URL: https://arxiv.org/abs/2505.21496 ; https://github.com/Euphoria16/UI-Genie

  • GUI-Critic-R1

  • GUI-Reflection

    • Description: Framework + data equipping multimodal GUI models with self-reflection and error-correction across pre-training, offline SFT, and online tuning phases.
    • Data: Reflection-oriented training data combined with grounding/navigation sources (AMEX, AndroidControl, GUI-Odyssey, etc.).
    • URL: https://arxiv.org/abs/2506.08012

  • MMBench-GUI

    • Description: Hierarchical multi-platform GUI evaluation framework with four ascending levels (content understanding, element grounding, task automation, task collaboration) and an Efficiency-Quality-Aware metric.
    • Data: Multi-level, multi-platform tasks spanning mobile, desktop, and web.
    • URL: https://arxiv.org/abs/2507.19478

  • GUI-Bee

    • Description: Aligns GUI action grounding to novel environments via autonomous exploration, generating environment-specific grounding data (Jan 2025).
    • Data: Self-collected, environment-specific grounding data across diverse GUIs.
    • URL: https://arxiv.org/abs/2501.13896

  • EDGE

    • Description: Enhanced grounded GUI understanding using enriched multi-granularity synthetic data (Nov 2024).
    • Data: Multi-granularity GUI understanding data (element-level, region-level, screen-level).
    • URL: https://arxiv.org/abs/2411.07618

2024

  • MultiUI

    • Description: A large-scale dataset for GUI agents' text-rich visual understanding, using structured accessibility trees to generate high-quality multimodal instructions.
    • Data: 7.3M multimodal instruction samples from 1M websites, covering element grounding, action prediction, and interaction modeling.
    • URL: https://huggingface.co/datasets/neulab/MultiUI

  • Mind2Web-Live

  • WebVLN

  • WebLINX

    • Description: Focuses on conversational GUI agents, emphasizing real-world web navigation through multi-turn dialogue.
    • Data: 2,300+ expert demonstrations with 100,000+ interactions across 155 real-world websites, including DOM trees and screenshots.
    • URL: https://huggingface.co/datasets/McGill-NLP/WebLINX

  • AgentTrek

    • Description: Synthesizes high-quality trajectory data by leveraging web tutorials.
    • Data: 4,902 trajectories with task metadata, step-by-step instructions, action sequences, visual observations, and reproducible native traces.
    • URL: https://agenttrek.github.io/

  • ScreenAI

  • VisualAgentBench

    • Description: A cross-platform benchmark assessing GUI agents in both mobile and web settings, emphasizing interaction-focused tasks.
    • Data: Uses Android Virtual Device and WebArena-Lite environments.
    • URL: https://github.com/THUDM/VisualAgentBench

  • VGA

    • Description: A VQA dataset minimizing "hallucinations" in VLMs for GUI understanding by balancing attention between image and text inputs. Uses a "Referent Method" to align responses to visual content.
    • Data: 63.8k high-quality VQA examples, based on the Rico dataset.
    • URL: https://github.com/Linziyang1999/VGA-visual-GUI-assistant

  • MobileViews

    • Description: Large-scale mobile GUI dataset with screenshot-view hierarchy pairs from thousands of Android apps.
    • Data: 600,000+ screenshot-view hierarchy pairs including .jpg screenshots, .json/.xml view hierarchies, and action logs.
    • URL: https://huggingface.co/datasets/mllmTeam/MobileViews

  • AMEX (Android Multi-annotation EXpo)

    • Description: High-resolution screenshots from popular mobile apps with detailed element-level and instruction-level annotations.
    • Data: 104,000+ high-resolution screenshots, 711,000 element-level functions, 3,000 unique instructions with multi-step GUI action sequences.
    • URL: https://huggingface.co/datasets/Yuxiang007/AMEX

  • AndroidControl

  • AndroidControl-Curated

  • B-MoCA

    • Description: Benchmark for evaluating mobile control agents across diverse device configurations.
    • Data: 131 daily tasks and 60 real-world tasks across varied device configurations.
    • URL: https://github.com/gimme1dollar/b-moca

  • MobileAgentBench

    • Description: User-friendly benchmark with tasks across open-source apps for testing mobile LLM agents.
    • Data: 100 tasks across 10 open-source Android apps, categorized by difficulty, with real-device execution support.
    • URL: https://MobileAgentBench.github.io

  • Mobile3M

    • Description: Large-scale mobile app dataset for training GUI agents with real-world UI transitions.
    • Data: 3M UI pages and real-world transitions from 49 popular Chinese apps, with XML documents and directed graph structures in Parquet format.
    • URL: https://huggingface.co/datasets/xwk123/Mobile3M

  • OmniACT

    • Description: Dataset for multimodal generalist agents performing tasks on desktop and web interfaces.
    • Data: 9,800+ image-instruction pairs with natural language tasks and PyAutoGUI-executable commands.
    • URL: https://arxiv.org/abs/2402.17553

  • VisualWebArena

    • Description: A dataset for training agents to interact with web interfaces using visual inputs, built on top of WebArena.
    • Data: Image-based tasks with executable evaluations focusing on multimodal web interactions.
    • URL: https://github.com/web-arena-x/visualwebarena

  • WebCanvas

    • Description: A dataset for training agents to interact with web-based environments, including drawing apps and other dynamic pages.
    • Data: Raw challenge data including HTML, screenshots, DOM trees, Axtrees, videos, and recorded actions.
    • URL: https://github.com/iMeanAI/WebCanvas

  • VisualWebBench

    • Description: A benchmark for evaluating visual web interaction tasks and multimodal LLM web understanding.
    • Data: Image and text data from websites for evaluating multimodal LLM web understanding and grounding.
    • URL: https://github.com/VisualWebBench

  • CRAB

    • Description: Cross-environment benchmark for multimodal agents supporting Ubuntu and Android tasks.
    • Data: 120 tasks in CRAB Benchmark-v0 with multimodal observations (screenshots) and Python-based task definitions.
    • URL: https://crab.camel-ai.org/

  • GUI-World

    • Description: Comprehensive GUI dataset with videos and queries for evaluating multimodal LLM-based agents.
    • Data: 12K+ videos, 100K queries with annotated keyframes, detailed captions, and diverse QA types in JSON/Parquet formats.
    • URL: https://gui-world.github.io/

  • GUICourse (GUIEnv)

    • Description: Large-scale page-caption pairs for training vision-based GUI agents across web and mobile.
    • Data: 10M page-caption pairs and 0.7M region-text QA pairs in JSON/Parquet, covering OCR, grounding, and navigation.
    • URL: https://github.com/yiye3/GUICourse

  • GUICourse (GUIAct)

    • Description: Single-step and multi-step GUI action instructions for web and Android scenarios.
    • Data: 67K single-step and 15K multi-step action instructions in JSON/Parquet.
    • URL: https://github.com/yiye3/GUICourse

  • GUICourse (GUIChat)

    • Description: Single-turn QAs and multi-turn dialogues for GUI interactions with bounding box annotations.
    • Data: 44K single-turn QAs and 6K multi-turn dialogues with text-rich images in JSON/Parquet.
    • URL: https://github.com/yiye3/GUICourse

  • AutoUI

    • Description: Leverages AITW to evaluate Auto-GUI, an LLM-based task automation system for Android.
    • Data: Utilizes AITW's 715,000 episodes and 30,000 unique instructions with screenshots and actions.
    • URL: https://github.com/cooelf/Auto-GUI

  • AndroidWorld

    • Description: An environment for building and benchmarking autonomous computer control agents on Android.
    • Data: 116 diverse tasks across 20 real-world Android apps, with dynamic task initialization for millions of variants.
    • URL: https://github.com/google-research/android_world

  • RICOSCA

  • WebVoyager

    • Description: A dataset for training agents to navigate and interact with web environments using pixel-based observations.
    • Data: Browser screen perceptions as pixels with mouse/keyboard actions, tested on 15 real-world websites.
    • URL: https://arxiv.org/abs/2401.13919

  • ScreenAgent

    • Description: A dataset for training vision-language model-driven computer control agents.
    • Data: Screen interaction data under Apache-2.0 license for computer control tasks.
    • URL: https://github.com/niuzaisheng/ScreenAgent

  • OpenDFM/MoGUI

    • Description: A mobile GUI interaction dataset for conversational agents with navigation graph structure.
    • Data: 2.6M+ GUI data from 250,000+ apps, each with screenshot and .xml metadata and navigation graphs.
    • URL: https://huggingface.co/datasets/OpenDFM/MoGUI

  • OpenDFM/MoCon

  • OpenDFM/MobA-MobBench

    • Description: A benchmark dataset for evaluating mobile phone agents, supporting English and Chinese.
    • Data: 50 tasks across 11 columns with task IDs, descriptions (EN/ZH), involved applications, scoring milestones, and expert steps.
    • URL: https://huggingface.co/datasets/OpenDFM/MobA-MobBench

  • WebUI (biglab/webui-all)

    • Description: Large dataset of rendered web pages with automatically extracted metadata, created by crawling the web to enhance visual UI understanding.
    • Data: 400,000 web UIs; HuggingFace version is a filtered subset; raw dataset on Google Drive.
    • URL: https://huggingface.co/datasets/biglab/webui-all

  • AGUVIS

    • Description: A unified pure vision-based framework for autonomous GUI agents operating across web, desktop, and mobile platforms.
    • Data: Stage 1 (Grounding): 4.2M samples; Stage 2 (Planning/Reasoning): 1.3M GUI agent trajectories with reasoning annotations.
    • URL: https://github.com/xlang-ai/aguvis

2023

  • UEyes

    • Description: Eye-tracking dataset for understanding visual saliency across various user interfaces.
    • Data: Eye-tracking data from 62 participants, 1,980 UI screenshots, with raw gaze logs, saliency maps, and scan paths.
    • URL: https://github.com/YueJiang-nj/UEyes-CHI2023

  • Android in the Wild (AITW)

  • GUI Odyssey

    • Description: A dataset for evaluating GUI agents across diverse multi-app tasks and environments.
    • Data: 7,735 episodes from 6 devices, covering 6 multi-app task types, 201 apps, and 1,400 unique app combinations.
    • URL: https://github.com/OpenGVLab/GUI-Odyssey

  • Mobile-Env

    • Description: A dataset for training agents to interact with mobile apps in simulated environments.
    • Data: WikiHow task set with screenshots, view hierarchies, and touch/type token actions in ProtoBuf 3 format.
    • URL: https://github.com/X-LANCE/Mobile-Env

  • Mind2Web

    • Description: A dataset for training agents to interact with web pages using natural language across diverse real-world websites.
    • Data: 2,350 tasks across 137 real-world websites in 31 domains, with HTML inputs and action sequences (Click, Type, Select).
    • URL: https://osu-nlp-group.github.io/Mind2Web/

  • WebArena

    • Description: A dataset for training agents to perform long-horizon tasks on realistic web pages.
    • Data: 812 long-term web tasks from 241 templates, with natural language intents, HTML/DOM trees, screenshots, and keyboard/mouse actions.
    • URL: https://webarena.dev/

  • Synapse

    • Description: A dataset for training agents to perform tasks across multiple web applications using CoT reasoning.
    • Data: 100,000 synthetic demonstrations across 21 domains, with Python programs, natural language plans, CoT reasoning, and HTML snippets.
    • URL: https://ltzheng.github.io/Synapse

  • ASSISTGUI

    • Description: A dataset for evaluating GUI agents across widely-used productivity software.
    • Data: 100 tasks across 9 productivity software with necessary project files for task execution.
    • URL: https://showlab.github.io/assistgui/

2022

  • META-GUI

    • Description: Benchmark for GUI-based task-oriented dialogue systems across six domains.
    • Data: 1,125 dialogues (4,684 turns) and 18,337 action prediction data points with screenshots, XML view hierarchies, and GUI actions.
    • URL: https://x-lance.github.io/META-GUI-Leaderboard/

  • UGIF

    • Description: A dataset for understanding user interactions with graphical interfaces in multiple languages.
    • Data: 523 multilingual natural language instructions with UI screen-action sequences, supporting 8 languages, including XML view hierarchies.
    • URL: https://arxiv.org/abs/2211.07615

  • WebShop

    • Description: Dataset for training autonomous agents in online shopping environments.
    • Data: 1.18M real-world products, 12,087 crowdsourced text instructions, and 1,600 human demonstrations in OpenAI Gym format.
    • URL: https://webshop-pnlp.github.io

2021

  • UIBert (AppSim & RefExp)

    • Description: A dataset for understanding and generating UI descriptions built on the Rico corpus.
    • Data: Built on Rico's 72,000 UI data points, with AppSim (similar UI element pairs) and RefExp (reference expressions for UI elements) in TFRecords.
    • URL: https://github.com/google-research-datasets/uibert

  • AndroidEnv

    • Description: A simulated environment for testing Android GUI agents with extensible task definitions.
    • Data: 100 example tasks with RGB pixel observations, (x,y) action spaces, and support for custom task extensions.
    • URL: https://github.com/deepmind/android_env

  • Screen Annotation

    • Description: Dataset for generating concise language descriptions of mobile screens.
    • Data: 22,417 mobile screenshots with 15,743 training, 2,364 validation, and 4,310 test annotations in CSV format, derived from Rico.
    • URL: https://github.com/google-research-datasets/screen2words

  • MoTIF (Mobile app Tasks with Iterative Feedback)

    • Description: A dataset for training agents to perform tasks on mobile apps with user feedback and feasibility annotations.
    • Data: 6,100+ free-form natural language commands across 125 Android apps, with action coordinates, screenshots, and feasibility annotations.
    • URL: https://vigilworkshop.github.io/static/papers-2021/26.pdf

2020

  • PixelHelp

    • Description: Multi-step instructions for common tasks on Google Pixel phones.
    • Data: 187 multi-step instructions across 4 task categories (general, Gmail, Chrome, Photos) with human-annotated step-by-step actions.
    • URL: https://arxiv.org/abs/2005.03776

  • ANDROIDHOWTO

    • Description: A dataset for training agents to follow step-by-step instructions on Android devices.
    • Data: 32,436 data points from 9,893 unique "How-to" instructions, with 190K action and 172K object segments in JSON/TFRecords.
    • URL: https://github.com/debymf/generating_android_howto

2018

This Awesome List is a comprehensive resource for GUI agent datasets, covering mobile, desktop, and web environments. Contributions are welcome to keep it updated with the latest advancements in computer interaction datasets!

About

This repository hosts a collection of datasets for training and evaluating CUA / GUI agents.

Resources

Readme
Activity

Stars

138 stars

Watchers

0 watching

Forks

7 forks
Report repository

Releases

Packages

Contributors