Deep Reinforcement learning for neuroscience.

Installation

Please install the following dependencies:

Pytorch

Tensorboard(Optional)

Running experiments in NeuRL:

NeuRL roughly follows the following design structure:

All of the components (training routine not yet), are sourced from their respective files. To specify where to source them, specify them in your config file like this:

{
    "experiment_name":"test",
	"environment":"SensoryDiscrimEnv",
	"architecture":"SimpleGRUModel",
	"algorithm":"BaseRNNDQNAlgo",
}

On top of sourcing different environments/models/algorithms, you should parameterise these components using extra nested configs:

"environment_params": {
    "n_classes":4,
    "dec_length":25,
    "noise_params":[0,0.5]

}

A full example config can be found in example_config.json

Monitoring experiments in TensorBoard:

When CoVid-19 struck, we realised that it might be useful to have a way to quickly look up how your model is doing. We can do this by using Tensorboard. Tensorboard makes it easy to track statistics of your model, as it's training, even remotely. Currently a Tensorboard server is exposed on the UZH INI VPN, ask Lucas about access. Tensorboard will automatically record your loss, your reward, epsilon, your learning rate, and the time it takes to complete a training epoch. But the pipeline will pass a TensorBoard writer object to your environment. So really, you can record anything you want, go crazy! Here are some things you can do with it.

defining your custom monitors:

By default NeuRL will always keep track of your reward, loss, epsilon and learning rate. But that doesn't mean you can't keep track of more scalars! By default, when NeuRL initialises your environment it will pass a tensorboard Writer object. Call this writer object during the run_trial call to log your own information. For example, in a discrimination task, we can log the time it takes to make a decision:

self.writer.add_scalar("DiscriminationEnv/decision_time", decision_time, epoch)

NOTE: To avoid clutter on the dashboard, please use the following syntax when defining your custom monitors:

"Environment name"/"variable name"