We are still working on further cleaning up the code and constantly updating it. We are also extending Cradle to more games and software. Feel free to reach out!

Download the StableSAM model file and copy it to the /cache folder.

First, set your main display scale to 125% and ensure your main display resolution is larger than 1920x1080, as shown in the picture below:

Then, set the folder that the agent will open to display in Large icons or Extra large icons format.

• To use debug mode, you need to change the --envConfig target in .vscode\launch.json to the software's JSON file in the conf\ directory that you want to test.
• To use terminal mode, you need to pass the --envConfig argument to the software's JSON file in the conf\ directory that you want to test.

Below are the exact software versions utilized in our paper:

In theory, any version can be used. However, if you want to reproduce our experimental results, we recommend using the software versions listed below.

Task Descriptions for Chrome, Outlook, CapCut, Meitu and Feishu. Difficulty refers to how hard it is for our agent to accomplish the corresponding tasks.

When completing each task in Chrome, you need to open the webpage in a new tab and ensure that there is nothing in the search history related to what you are about to search for.

We use the new version of Outlook, and you need to set the following preferences:

Go to Mail > Layout > Focused Inbox and select "Don't sort my messages." Set Mail > Layout > Text size and spacing to Large. Set Mail > Layout > Message organization to individual messages. For each task in Outlook, the initial page is shown in the figure:

• For Task 2, ensure there is at least one email in the junk mail folder.
• For Task 4, ensure there is at least one email in the inbox with the subject "Urgent meeting."

For each task in CapCut, the initial page is shown in the table below:

CapCut Homepage for task #1, #2, #4	CapCut Project Page for task #3, #5

• For Task #1, ensure you have a video placed in the default folder that CapCut will open.
• For Task #2, make sure that you have two videos added to the timeline in your first project.
• For Task #3 and Task #4, you need to have one video added to the timeline.
• For Task #5, ensure your task description involves a cropping operation relevant to the video content.

For each task in Meitu, the initial page is shown in the figure:

• For Task #4, ensure you have at least three pictures placed in the default folder that Meitu will open.
• For all other tasks, having one picture in the default folder is sufficient.

For each task in Feishu, the initial page is shown in the figure:

• For Task #3, ensure you have an AWS bill PDF file at your input path.
• For Task #4, set your status to "None" before you begin the test.

• Always pull the latest /main branch to your current work branch.
• Add [conf\env_config_xxx.json] to adapt to your target software.
• Copy the cradle\environment\chrome folder in cradle\environment\ and rename it to your software environment name. Replace all instances of "chrome" within the folder with your software's environment name.
• Copy the res\chrome folder in res\ and rename it to your software environment name. Replace all instances of "chrome" within the folder with your software's environment name. Modify the prompts and template-matching icon images as needed (for important UI elements that SAM2SOM cannot recognize).

Small tip: Use [log_proc.py] to visualize logs and see how to improve your prompts and skills.

1. Always Pull the Latest Branch:

   • Ensure that you always pull the latest /main branch to your current work branch to keep your repository up to date.

2. Configuring the Environment:

   • Add a configuration file in the format [conf\env_config_xxx.json] to adapt Cradle to your target software.

3. Setting Up the Environment:

   • Copy the cradle\environment\chrome folder located in cradle\environment\ and rename it to match your software environment name. Replace all instances of "chrome" within the folder with your software's environment name.
   • Copy the res\chrome folder located in res\ and rename it to your software environment name. Replace all instances of "chrome" within the folder with your software's environment name. Modify the prompts and template-matching icon images as needed for important UI elements that SAM2SOM cannot recognize.

4. Debug and Terminal Modes:

   • Debug Mode: Change the --envConfig target in .vscode\launch.json to point to the software's JSON file in the conf\ directory that you want to test.
   • Terminal Mode: Pass the --envConfig argument to the software's JSON file in the conf\ directory that you want to test.

5. Visualizing Logs:

   • Use [log_proc.py] to visualize logs. This helps you understand how your prompts and skills are performing and identify areas for improvement.

If you find our work useful, please consider citing us!

@article{tan2024cradle,
  title={Cradle: Empowering Foundation Agents towards General Computer Control},
  author={Weihao Tan and Wentao Zhang and Xinrun Xu and Haochong Xia and Ziluo Ding and Boyu Li and Bohan Zhou and Junpeng Yue and Jiechuan Jiang and Yewen Li and Ruyi An and Molei Qin and Chuqiao Zong and Longtao Zheng and Yujie Wu and Xiaoqiang Chai and Yifei Bi and Tianbao Xie and Pengjie Gu and Xiyun Li and Ceyao Zhang and Long Tian and Chaojie Wang and Xinrun Wang and Börje F. Karlsson and Bo An and Shuicheng Yan and Zongqing Lu},
  journal={arXiv preprint arXiv:2403.03186},
  year={2024}
}