Deconfounding Reinforcement Learning in Observational Settings
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
ac_decon DRL Dec 31, 2018
model_decon_uBernoulli DRL Dec 31, 2018
model_decon_uGaussian DRL Dec 31, 2018
README.md DRL Dec 31, 2018

README.md

Deconfounding Reinforcement Learning (DRL)

This repository contains a clean version of the code for the Deconfounding Reinforcement Learning (DRL) model as developed at [1]. The code is easy to modify for your own applications. It is worth noting that this is my first work within the charming but mysterious kingdom of Causal Reinforcement Learning (CausalRL). Undoubtedly, CausalRL will become an indispensable part of Artificial General Intelligence (AGI). Please refer to CausalRL for more details.

Running Environment

Only tested the codes on

  • Tesla P100 with 16GB GPU memory
  • tensorflow-gpu v1.5.1.

Not sure about other options.

Data Format

In default the data are organised as follows. Feel free to modify these parameters for your own applications.

                     num    x    nsteps   x      dim
x_train:           140,000          5            784
a_train:           140,000          5             1
r_train:           140,000          5             1
mask_train:        140,000          5             1
x_validation:       28,000          5            784
a_validation:       28,000          5             1
r_validation:       28,000          5             1
mask_validation:    28,000          5             1
x_test:             28,000          5            784
a_test:             28,000          5             1
r_test:             28,000          5             1
mask_test:          28,000          5             1

N.B. 1st dim: the number of sequences; 2nd dim: steps of one sequence; 3rd dim: dim of data.

How to Run

This repo consists of three folders:

  • model_decon_uGaussian: learning M_Decon when the dimension of u is set to 2 and the prior over u is assumed to be a factorised standard Gaussian.

    python run.py

  • model_decon_uBernoulli: learning M_Decon when the dimension of u is set to 1 and the prior over u is assumed to be a Bernoulli with p=0.5.

    python run.py

  • ac_decon: learning and testing AC_Decon (In default M_Decon learned in model_decon_uBernoulli is required).

    python train_policy.py

    python test_policy.py

Note that before running the code, you are supposed to modify configs.py, especially the following lines which involve the directories of data and models.

########################################## data and model path Configuration ###########################################

model_config['work_dir'] = './training_results'
model_config['data_dir'] = './dataset'
model_config['training_data'] = './dataset/training_data.npz'
model_config['validation_data'] = './dataset/validation_data.npz'
model_config['testing_data'] = './dataset/testing_data.npz'
model_config['model_checkpoint'] = './training_results/model_checkpoints/model_alt'
model_config['policy_checkpoint'] = './training_results/policy_checkpoints/policy_alt'

########################################################################################################################

References

[1] Deconfounding Reinforcement Learning in Observational Settings Chaochao Lu, Bernhard Schölkopf, José Miguel Hernández-Lobato, 2018