Using intermediate_reward causes randomly done=True

[tshrjn]

When we switch on the intermediate_reward for the environments made via tw-custom, there's some bug which causes the env to finish the epiode(done=True) and give info["has_lost"]=True whenever the agent eats anything (for eg eat pizza). This does not happen when we're not using intermediate_reward.

Also, in the sample notebook provided, there the reward = score - self.last_score # Reward is the gain/loss in score. Is there any theory/evidence as for why this helps or is done? I'd greatly appreciate that as well.

[MarcCote]

Hi, thanks for reporting this.

Do you still have the tw-make command line you used to generate the game along with the global seed (displayed in the terminal if you haven't used --silent), or the generated files (.ulx, .json and .ni)?

Regarding your question about reward vs score, we followed the same convention as in the DQN paper in which the reward represents the change in game score of Atari games. While it is not based on any theory/evidence, it does seem to work.