Local Run of Reinforcement Learning (RL) Recommender Systems Pipeline using MovieLens 100K dataset

To launch an interactive demo click here

TF-Agents

• A well written and detailed summary is here
• A flexible library enabling one to implement and test RL-based applications
• Policy: Equivalent of a model in a supervised learning model, mapping from an observation to an action
• Action: output of a policy
• Agent: an algorithm that uses a policy to output an action, and trains the policy
• Observation: input generated by the environment
• Environment: At each time step, the environment generates an observation, and respond with a reward as feedback given actions taken by an agent and the perceived observation

Seting up an experiment

• Users are represented as user embedding vectors
• Actions are the movies available for recommendation
• Reward is user rating of a movie

Training a RL recommender system

• During training, randomly sample users, use those user vectors as observations to query some policy for items to recommend, and then apply that recommendation to users and obtain their feedback as rewards

Making a prediction

• Given an obervation, which is a user vector and trained policy, return predicted feedback, that is predicted rating for movies

Code Structure

• This code base leverages and follow closely with the step-by-step tutorials from a google blog post

• Model parameters and i/o are defined in constants module

• Raw data of movie lens was downloaded from kaggle and was saved to ./ml-100k

• The source code is located at src folder, consisting of modeling, predicting, and plotting

• Jupyter notebook is used as a runner code

Repo setup

Fully local

1. install python, for example with pyenv: pyenv install 3.8.3
2. install poetry: pip install poetry
3. use poetry to install the dependencies: poetry install
4. Fire up a jupyter server and go from there: poetry run jupyter notebook tf-recommender-systems/step-by-step-reinforcement-learning.ipynb

Via Docker

In case it's easier to work with this data via docker, you can get a local image and access our interactive workbook via

1. build their image docker build -t tf-recommender-systems .
2. Run the image and access the lab! docker run -it --rm -p 8888:8888 tf-recommender-systems jupyter lab --NotebookApp.default_url=/lab/ --ip=0.0.0.0 --port=8888. However, this route needs you to set up jupyter notebook password.
3. Another route is docker run -it --rm tf-recommender-systems bash. After you are in the docker shell, set a password. jupyter notebook password.
4. Launch jupyter notebook from the docker shell.
5. Copy the url from terminal to a browser.