Skip to content
Go to file

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time


Authors: Evan Zheran Liu*, Kelvin Guu*, Panupong (Ice) Pasupat*, Tianlin Shi, Percy Liang (* equal contribution)

Source code accompanying our ICLR 2018 paper:
Reinforcement Learning on Web Interfaces using Workflow-Guided Exploration

Reproducible experiments using this code are located on our Codalab worksheet.


The goal of this project is to train machine learning models (agents) to do things in a browser that can be specified in natural language, e.g. "Book a flight from San Francisco to New York for Dec 23rd."


General setup

  • Python dependencies

    pip install -r requirements.txt
    • If this gives you problems, try again and add pip's --ignore-installed flag.
  • Node and npm

    • Make sure Node and npm are installed via brew install node. If they are, node -v and npm -v should print version numbers.
  • PyTorch

    • Install PyTorch v0.1.12. Newer versions of PyTorch are not backwards compatible.
  • Selenium

    • Outside this repository, download ChromeDriver. Unzip it and then add the directory containing the chromedriver executable to the PATH environment variable
      export PATH=$PATH:/path/to/chromedriver
    • If instead you're using Anaconda, use conda install -c conda-forge selenium.

Data directory setup

  • This code depends on the environmental variable $RL_DATA being set, pointing to a configured data directory.

  • Create a data directory mkdir -p /path/to/data and set export $RL_DATA=/path/to/data. In order for the code to run, $RL_DATA will need to be set to point at this directory.

  • Next, set up the data directory:

    cd $RL_DATA
    # Download glove from and place
    # in current directory however you want
    # Suggested: wget
    mv glove.6B glove

Demonstration directory setup

# Where $REPO_DIR is the path to the root of this Git repository.
git clone $REPO_DIR/third-party/miniwob-demos
export RL_DEMO_DIR=$REPO_DIR/third-party/miniwob-demos/

MiniWoB setup

  • There are 2 ways to access MiniWoB tasks:
    1. Use the file:// protocol (Recommended): Open miniwob-sandbox/html/ in the browser, and then export the URL to the MINIWOB_BASE_URL environment variable:
    export MINIWOB_BASE_URL='file:///path/to/miniwob-sandbox/html/'
    1. Run a simple server: go to miniwob-sandbox/html/ and run the supplied http-serve.
    • The tasks should now be accessible at http://localhost:8080/miniwob/
    • To use a different port (say 8765), run http-serve 8765, and then export the following to the MINIWOB_BASE_URL environment variable:
    export MINIWOB_BASE_URL='http://localhost:8765/'
  • Once you've followed one of the steps above, test MiniWoBEnvironment by running
    pytest wge/tests/miniwob/ -s

MiniWoB versions of FormWoB

Follow the "Run a simple server" instruction in the MiniWoB setup section above.

Launching an Experiment

To train a model on a task, run:

python configs/default-base.txt --task click-tab-2
  • This executes the main entrypoint script, In particular, we pass it a base HOCON format config file and the task click-tab-2.
  • Additional configs can be merged in by passing them as commandline arguments from configs/config-mixins
  • Make sure that the following environmental variables are set: MINIWOB_BASE_URL, RL_DEMO_DIR, REPO_DIR.
  • You may also want to set the PYTHONPATH to the same place as REPO_DIR to make imports work out properly
  • You can also run this via docker by first running python to launch Docker and then running the above command. Unfortunately, you will not be able to see the model train in the Docker container.
  • The different tasks can be found in the subdirectories of third-party/miniwob-sandbox/html

If the script is working, you should see several Chrome windows pop up (operated by Selenium) and a training progress bar in the terminal.

Experiment management

All training runs are managed by the MiniWoBTrainingRuns object. For example, to get training run #141, do this:

runs = MiniWoBTrainingRuns()
run = runs[141]  # a MiniWoBTrainingRun object

A TrainingRun is responsible for constructing a model, training it, saving it and reloading it (see superclasses and for details.)

The most important methods on MiniWobTrainingRun are:

  • __init__: the policy, the environment, demonstrations, etc, are all loaded here.
  • train: actual training of the policy happens here

Model architecture

During training, there are several key systems involved:

  • the environment
  • policies
    • the model policy
    • the exploration policy
  • episode generators
    • basic episode generator
    • best first episode generator
  • the replay buffer


All environments implement the Environment interface. A policy interacts with the environment by calling the environment's step method and passing in actions.

Note that an environment object is batched. It actually represents a batch of environments, each running in parallel (so that we can train faster).

We mostly use MiniWoBEnvironment and FormWoBEnvironment.


See the Policy interface. The most important methods are act, update_from_episodes and update_from_replay_buffer.

Note that all of these methods are also batched (i.e. they operate on multiple episodes in parallel)

The model policy is the main one that we are trying to train. See MiniWoBPolicy as an example.

Episode generators

See the EpisodeGenerator interface. An EpisodeGenerator runs a Policy on an Environment to produce an Episode.

Replay buffer

See the ReplayBuffer interface. A ReplayBuffer stores episodes produced by the exploration policy. The final model policy is trained off episodes sampled from the replay buffer.


All configs are in the configs folder. They are specified in HOCON format. The arguments to should be a list of paths to config files. then merges these config files according to the rules explained here.


Workflow-Guided Exploration: sample-efficient RL agent for web tasks




No releases published


No packages published


You can’t perform that action at this time.