Not getting any TensorBoard logs for DQN model

[saulable]

Hi Guys,

I am having a problem with the DQN logging to TensorBoard, I don't get when the log is getting called. At the moment, to me it seems the log is only made when the "done" parameter returns True from step(), and this is only after the second iteration, done needs to return true twice?!

My code doesn't return a done = True unless it completely fails.

The code works fine in PPO method without the done returning true, and logs my values just fine at each iteration.

My data CSV is around 160k steps long. Am I missing something? Here's my model call. I also get the max score at every step when I print here, just my tensorboard is not working.... How do we get it working like PPO? I.E on every iteration after X steps? Thanks!

class TensorboardCallback(BaseCallback):
    Custom callback for plotting additional values in tensorboard.
 
    def __init__(self, verbose=0):
        super(TensorboardCallback, self).__init__(verbose)

    def _on_step(self) -> bool:
        # Log scalar value (here a random variable)
        # print(self.locals["rewards"])
        # print(self.locals)
        global top_score
        score = self.locals["infos"][0]["score"]
        self.logger.record('a_score', score)
        if (top_score == None):
            top_score = score
            model.save("./model_DQN")
        elif (score > top_score):
            top_score = score
            model.save("./model_DQN")
        return True

model = DQN("MlpPolicy", env, verbose=1, tensorboard_log="./a2c_cartpole_tensorboard/", device="cpu")
model.learn(total_timesteps=int(1e10), callback=TensorboardCallback())

[araffin]

I.E on every iteration after X steps?

You have to manually call self.logger.dump(self.num_timesteps) for that in the callback, see issue #457 and PR #492

[saulable]

I.E on every iteration after X steps?

You have to manually call self.logger.dump(self.num_timesteps) for that in the callback, see issue #457 and PR #492

Excellent. Quick fix with the below code done the damage. Muchos gracias! Can we update the docs with this 👍

if (self.num_timesteps % 10000 == 0):
            self.logger.dump(self.num_timesteps)