Code accompanying "The Importance of Pessimism in Fixed-Dataset Policy Optimization" by Buckman, Gelada, Bellemare. https://arxiv.org/abs/2009.06799
The tabular experiments can be replicated with simply
Running the deep learning experiments requires quite a few more steps.
dqn_data_collection.pyto train a standard DQN model to convergence, checkpointing the resulting near-optimal policy.
policy_data_collection.py, pointing at a specific checkpointed policy, and setting a specific epsilon, to collect a dataset with epsilon-greedy, and save it.
main.pyto run an FDPO experiment.
compile_logs.pyto extract the final performance numbers from the logs of many completed experiments, and compile them into a table suitable for plotting.