Generating Interpretable Code with Reinforcement Learning

CS4033 - Short Reinforcement Learning Project

Caleb Carr - Caleb.M.Carr-1@ou.edu

Bibi Dang - Bibi.M.Dang-1@ou.edu

Installation

This project requires Textworld and all of its dependencies. TextWorld requires Python 3 and only supports Linux and macOS systems at the moment.

Requirements

TextWorld requires some system libraries for its native components. On a Debian/Ubuntu-based system, these can be installed with

sudo apt update && sudo apt install build-essential libffi-dev python3-dev curl git

And on macOS, with

brew install libffi curl git

Installing TextWorld

Clone the project repo with

 git clone https://github.com/bibidang/CS4033ShortRL.git

After cloning the repo, go inside the root folder of the project (i.e. alongside setup.py) and run

pip3 install .

Implementation

Modified TextWorld Files

The following files within the TextWorld environment have been changed to convert natural language responses into interpretable Python keywords and phrases:

• CS4033ShortRL/textworld/generator/inform7/world2inform7.py

• CS4033ShortRL/textworld/generator/data/text_grammars/house_instruction.twg

• CS4033ShortRL/textworld/generator/data/text_grammars/house_obj.twg

Creating Game to Test

The original scripts created for this project can be found in CS40333ShortRl/carr_dang_shortRL. Once in this folder, run the following command to compile the game created for this project:

python3 create_game.py

This will first print three random variables (say x, y, z) and then prompt the user to create the quests needed to define rewards within the game.
Four quests were defined for this project. To replicate these quests, enter the following commands into the interactive create_game.py session (replacing x, y, z with the random variables given):

> go east
> examine x
> ctrl C

> go west
> examine y
> ctrl C

> go south
> take z
> ctrl C

> go south
> examine z
> ctrl C

A new game and its associated files can now be found in the compiled_game folder.

Play Game with Agent

Run the following command while in the CS40333ShortRl/carr_dang_shortRL folder to start the playing agent:

python3 random_agent.py

Set-up for Experiment 1

Experiment 1 of this project tested sparse, dense, and balanced reward distributions.
The following code chunk must be modified to test each distribution:

#Create quests with assigned rewards
q1 = M.record_quest() #Create quest for including completed if statement
q1.reward = 1 #Reward for completing if statement (sparse=1, dense=5, balanced=1)
q2 = M.record_quest() #Create quest for including complete for loop
q2.reward = 1 #Reward for completing for loop (same distribution as if)
q3 = M.record_quest() #Create quest for including print statement
q3.reward = 1 #Reward for completing print statement (sparse=0, dense=1, balanced=1)
q4 = M.record_quest() #Create quest for including new variable assignment
q4.reward = 1 #Reward for completing variable assignment (same distribution as print)

Assign q1.reward, q2.reward, q3.reward, and q4.reward with the correct value based on the comments within the code. After changing these values for the distribution being tested, run create_game.py and random_agent.py as described above.

Set-up for Experiment 2

Experiment 2 of this project tested the max number of moves for each episode.
The following line of code must be modified for each test:

def play(agent, path, max_step=100, nb_episodes=10, verbose=True):

Run create_game.py and random_agent.py using max_step equal to 100, 50, and 10.

Results

The results for both of these experiments can be found in CS40333ShortRl/carr_dang_shortRL/experiment1_results and CS40333ShortRl/carr_dang_shortRL/experiment2_results