Offline PPO?

[wangell]

Is there an easy way to train PPO offline using the current API, i.e. pre-collecting state, action, reward tuples and training on those (as opposed to using a memory agent with observations)?

The Runner documentation on https://tensorforce.readthedocs.io/en/latest/runner.html says you should be able to call something like agent.observe(state, action, reward, terminal_state)

[michaelschaarschmidt]

@krfricke Runner doc needs to be updated.

This is not directly supported because it's an on policy algorithm, even though one could supply a method to set the batch of the pg agent to some given values. Feel free to implement one in the pg agent, set_batch(), where you provide data in the correct layout.

If the goal is to do pre-training, someone posted the folllowing snippet on gitter a while ago:

# agent = PPOAgent(config=config)
agent = VPGAgent(config=config)
x = np.array(dialog_manager.pretraining_data['x'], dtype=np.float32)
y = np.array(dialog_manager.pretraining_data['y'], dtype=np.int)
reward = np.ones((batch_size,), dtype=np.float32)


# Training cycle
for epoch in range(100):
    avg_cost = 0.
    # shuffle indexes
    indices = np.arange(x.shape[0], dtype=np.int32)
    np.random.shuffle(indices)
    total_batch = int(len(x) / batch_size)
    # Loop over all batches
    for i in range(total_batch):
        index_list = indices[i*batch_size:i*batch_size + batch_size]
        batch_x, batch_y = x[index_list], y[index_list]
        # Run optimization op (backprop) and cost op (to get loss value)
        _, c = agent.model.session.run([agent.model.optimize, agent.model.loss],
                                              feed_dict={agent.model.state['state']: batch_x,
                                                         agent.model.action['action']: batch_y,
                                                         agent.model.reward: reward})
        # Compute average loss
        avg_cost += c / total_batch
    # Display logs per epoch step
    print("Epoch:", '%04d' % (epoch + 1), "cost=", "{:.9f}".format(avg_cost))`

where x - array of states, y- array of right actions

[michaelschaarschmidt]

I should also note that a reason this is not supported is that it breaks LSTM policy usage because one could not provide the internal state for the lstm.

[AlexKuhnle]

Improved