TensorBoard scalars, images, distributions, histograms legend/description

[piojanu]

Hi again!

I was able to run your code for adapted Sokoban environment (#9). Now I wonder what all of those statistics in TensorBoard mean. Would you be able to describe in one sentence the most important positions from scalars, images, distributions, histograms tabs? What am I looking at or maybe what SHOULD I look at?

Thanks for your time, I really appreciate that you (and whole team) open source your work and devote time to help the community work on their projects.

[piojanu]

Also, what does this statistic in the console mean: step/score/loss/zs_entropy/zs_divergence = [120200, -nan, 11374.6328, 37.9262047, 2.77373099]. Is that -nan a problem?

[danijar]

Nice! The nan is not a problem at all, it just means that there is no score during iterations where no planning happens. The most important summaries at the "cem" scalar summaries that show the planning performance. If any of the "divergence" scalar summaries is at zero the divergence_scale is too high. Besides that, you can look at the "openloop" image summaries to see future frames imagined by the agent and at the "cem" image summaries to see the planning policy in the environment.

[piojanu]

Thank you for help! Now, I'll reread the paper (and maybe refresh knowledge on VAE) and look deeper into the code to get back with better questions ;)
For now: (1) What does open/closedloop mean? (2) Also, I cannot find "cem" statistics beside return (e.g. there are no images with "cem" in their name). It learns something sensible, train images looks good, at least posterior ones. (3) Do I understand it correctly that prior images are just decoded noise (prior is normal distribution and it's not trained, right?).

[danijar]

Yes, I suggest looking at the paper again. The closed loop prior is a one-step prediction and the closed loop posterior is a 0-step reconstruction. Open loop means predicting forward for multiple steps without receiving intermediate image inputs. There should be a cem image summary once the agent has collected any test episodes.

[piojanu]

Thank! I'm discovering the paper once again, everything starts to make sense 👍 Also, here is the paper I was looking for: TensorFlow Agents: Efficient Batched Reinforcement Learning in TensorFlow 😄 Great job!

[piojanu]

Hi again! Do I understand it correctly that those two graphs:

show train and test returns? Also, after your last updates in images there are those two summaries:

but gif doesn't move and the second one "image" doesn't show anything :/ Should I open issue or I don't know something?

[danijar]

The GIF will only work if ffmpeg is available in your PATH. Otherwise, it falls back to showing the first frame as a static image. The second summary should show an image but it may be very wide (based on the sequence length) compared to its height. TensorBoard allows you to increase the image size which should make it visible.

[piojanu]

Okay, I've fixed ffmpeg (I'm able to do ffmpeg -version without error). We will see if now gif will work ;) But still, I wasn't able to show graph/summaries/simulation/should_simulate_gym_sokoban/summary-gym_sokoban-cem-12/image/image/0. Where do I need to click to increase the image size to make it visible?

[danijar]

[piojanu]

@danijar do I understand it correctly, the test step log is mean score after test_steps runs of the game? Here is the definition of the test phase:

      trainer.add_phase(
          'test', config.test_steps, score, summary,
          batch_size=config.batch_shape[0],
          report_every=config.test_steps,
          log_every=config.test_steps,
          checkpoint_every=config.test_checkpoint_every)

I've looked through the code and I see that there is some running average kept, but I'd like to confirm that.

[piojanu]

I ask because test scores seems to have a lot of variance from global step to step. collect_rollouts uses tf.scan to play for max_steps if I understand it right. And in each trainer step in test phase it's called, right? So test_steps games are played and then average is logged at the end (because log_every is set to test_steps).

[danijar]

It takes an average over the episodes simulation per test phase. Typically, there is only one simulation per test phase so, it's just a single score.

[piojanu]

How is that? Could you point to some line in code? E.g. here score is defined as mean od returns:

planet/planet/training/define_summaries.py

Lines 98 to 113 in 9cd9abe

	with tf.variable_scope('simulation'):
	sim_returns = []
	for name, params in config.sim_summaries.items():
	# These are expensive and equivalent for train and test phases, so only
	# do one of them.
	sim_summary, sim_return = tf.cond(
	tf.equal(graph.phase, 'test'),
	lambda: utility.simulate_episodes(config, params, graph, True, name),
	lambda: ('', 0.0),
	name='should_simulate_' + params.task.name)
	summaries.append(sim_summary)
	sim_returns.append(sim_return)
	summaries = tf.summary.merge(summaries)
	score = tf.reduce_mean(sim_returns)[None]
	return summaries, score

And I thought it's run at every test step (so e.g. 100 times) and then in trainer those 100 runs are averaged. Could you highlight more how it works?

[danijar]

Sorry, it takes the mean over episodes per test step, not per test phase. So if the test phase spans multiple steps, there will be multiple scores on TensorBoard. The mean only changes anything if the episodes terminate early and multiple episodes fit within the task.max_length steps or if the simulation batch size is changed to something other than 1.