[rllib] rewards column is being discarded when using PG

[dibgerge]

What is the problem?

Ray version: nightly build
DL package: Torch

The documentation mentions that rewards column is never deleted, however when using PGTorchPolicy and PGTorchTrainer, only the advantages column is available, and there is no reward.

This prohibits using offline algorithm, like re-creating the offline datasets example in the docs here.

Reproduction (REQUIRED)

Can be recreated by following the offline datasets example. Generate data with PG:

rllib train
    --run=PG \
    --env=CartPole-v0 \
    --config='{"output": "/tmp/cartpole-out", "output_max_file_size": 5000000}' \
    --stop='{"timesteps_total": 100000}'

Looking at the outputs json file for this training, there are no rewards field. Also, when using the generated data to train DQN, it fails with KeyError: rewards (and KeyError: new_obs).

• I have verified my script runs in a clean environment and reproduces the issue.
• I have verified the issue also occurs with the latest wheels.

[sven1977]

Workaround for now: _use_trajectory_view_api=False.

Sorry about the delay. I thought this was fixed in an earlier PR, but it wasn't.
This PR here (see above) fixes the issue for both torch and tf. It makes sure that certain "core" keys are always available in the loss/output, such as rewards, dones, infos, etc..

[sven1977]

@dibgerge, thanks for filing this!