This repository contains the code and data for the models and experiments described in "Glider: A reinforcement learning approach to extract UI scripts from websites" by Yuanchun Li and Oriana Riva, which was accepted at the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021).
If you use any of the materials in this repository, please cite the following paper.
@inproceedings{glider:sigir21,
title = {Glider: A reinforcement learning approach to extract {UI} scripts from websites},
author = {Yuanchun Li and Oriana Riva},
booktitle = {44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021)},
year = {2021},
month = jul,
address = "Online"
doi = {10.1145/3404835.3462905}
}Glider is a tool to extract UI scripts from the web. Given a website URL (e.g., http://www.unitconversion.org/) and a task description (e.g., “convert length 7333 from inch to foot”), Glider explores the website's UI to identify a sequence of UI actions (click, type, select, etc.) to complete the given task.
datadirectory contains the sample tasks we created for the experiments.- Each file is a json file containing the website URL, query, query annotations, etc. To define your own task, simply create a similar task file and fill in the
start_url,query_wordsandquery_annotationsfields. - For each task we have created a demonstration useful to verify whether the extracted UI script is correct. For some tasks, we also include several fields (e.g.,
target_url,target_text_res, etc.) used to more reliably verify whether the task completed correctly. However, both the demonstration and target matching rules are not the only indicators of task completion because there might be other ways to complete the task.
- Each file is a json file containing the website URL, query, query annotations, etc. To define your own task, simply create a similar task file and fill in the
srcdirectory contains the code of Glider, including the scripts to crawl tasklets, randomly explore websites, record/replay demonstrations, etc.configdirectory contains sample configuration files to run the tests.
- Install Python 3.6
- Install the Chrome browser and web driver
- Simply install the Chrome web browser application. On a Linux machine, install chromium with
apt install chromium-browser - Download the Chrome web driver and place it in the
src/resourcesdirectory.- The web drivers should be named as:
src/resources/chromedriver_linux64src/resources/chromedriver_mac64src/resources/chromedriver_win32.exe
- The web drivers should be named as:
- Simply install the Chrome web browser application. On a Linux machine, install chromium with
- Install required Python packages
pip install -r requirements.txtpython -m spacy download en_core_web_lg
To crawl web automation scripts:
- Add the task definitions in the
data/tasksdirectory. - Edit the configuration file in the
configsdirectory. In the configuration file, thetest_tasksfield is used to specify the task descriptions saved in thedata/tasksdirectory. - Start crawling using the
run.pyfile. An example command line is:
cd glider/src
python run.py --config_path ../configs/unit_conversion.jsonThe code and dataset in this repository are intended to be used for research purposes. Microsoft takes no responsibility for what users use this tool for or for any damages caused from using this code. By downloading and using this software, you agree that you take full responsibility for any damages and liability.
Query examples in the dataset that appear to be attributed to a user or related user contact information are for illustration only and are fictitious. No real association is intended or inferred.
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.