• 1. Setting up environment
  • 1.1. Few-clik-ready setup
  • 1.2. Manual setup
    • 1.2.1. Install dependencies
    • 1.2.2. Set up environment variables
• 2. Structure of the code

Reward-Bases: A Simple Mechanisms for Adaptive Acquisition of Multiple Reward Types

This repository contains the data and code for the paper Reward-Bases: Dopaminergic Mechanisms for Adaptive Acquisition of Multiple Reward Types. By Beren Millidge, Yuhang Song*, Armin Lak, Mark E. Walton, Rafal Bogacz*.

Experimental data and usage instructions with activity of primate dopamine neurons is in Data.

1. Setting up environment

1.1. Few-clik-ready setup

The repo is setup with dev container so can be run directly in a github codespace, without any additional setup.

To go with this way:

You need at least this tier:

If you do not see this tier, you might need a add payment info to your account.

Note the the codespace is not free, but you can get a free trial for 14 days.

Note that you can open codespace on your local vscode, or on the web (as the above).

Alternatively, you can open this repo locally on your vscode, the repo contains dev container configuration, vscode should prompt you to open the repo in a dev container, and you can click yes to open it in a dev container.

All above ways does not require you to install anything, and you can run the code directly within to dev container.

If you have to go with the manual setup, see below.

1.2. Manual setup

1.2.1. Install dependencies

You need to install the following python packages:

pip install ray[all] torch torchvision torchaudio seaborn tqdm visdom tabulate statsmodels h5py

1.2.2. Set up environment variables

You need to set up the following environment variables:

• RESULTS_DIR: The directory where you want to store the results.
• RB_CODE_DIR: The directory where you clone and store the code.

2. Structure of the code

The code is organized as follows:

• files in ./ are the main files for running the code. Following gives a brief description of each file.
  • main.py: Calling this file with configuration yaml file will will launch Simulations (the backbone of the code is ray, which will launch multiple processes to run the code in parallel)
  • utils.py: Utility functions
  • analysis_v1.py: Functions for analyzing the results. Calling this file with configuration file will load the results and plot the figures.
  • analysis_utils.py: Utility functions for analysis.
  • fit_data.py: Functions for fitting data from biological Simulations.
• Simulations: This folder contains all the simulations and data analyses. Specifically, each subfolder contains
  • the README.md file that describes the experiment, document the comments to run to reproduce the experiment and reproduce the figure in the paper.
  • the configuration file(s) (.yaml) for the experiment.
  • .py files that are specific to the experiment.
  • .png and .pdf files that are the figures generated by the experiment.
• Tests: This folder contains unit test for all layers of the code. They are coded in pytest style.

Then simply look into each subfolders in Simulations and follow the instructions in the README.md file, where the resulting figures are also documented.

By reading each README.md file in a markdown editor (such as viewing it on github), the resulting figures are attached inline, so it is obvious which corresponding figure it produces in the paper.

Before looking at each experiment folder, we explain the shared underlying logic of the code. In each experiment folder, the README.md documents two kinds of commands to run:

• python main.py -c <config_file.yaml>: This command will launch the experiment. The experiment will be launched in parallel with multiple processes (with ray as backend), and the results will be saved to $RESULTS_DIR in your environment variable.
• python analysis_v1.py -c <config_file.yaml>: This command will load the results from $RESULTS_DIR and plot the figures. The figures will be saved to the experiment folder.

Some simumations does not follow this logic, but directly runs a python script to produce figures, rather than through ray (main.py + analysis_v1.py). Their logic are more straight forward, and are documented in the README.md file in the corresponding simulation folder.