question about the distribution of Q values at each epoch

[yanan1116]

hi authors,

it seems that in the current codebase, there is currently no built-in epoch summary proving that Q updates are broadly taking effect, which means that the distribution of the Q values among these memories, are unknown: how many of them are of zero Q values, and how many them are of long-tail Q values.

Logically, it is very possible that many memories may never be retrieved again and thus those may remain at their initial q value ( set to zero according to the paper) .

So my question is that have you ever tracked this Q value distribution ?
for example, this monitoring metrics about

• total memory count
• number and ratio of nonzero-Q memories
• number and ratio of positive / zero / negative Q memories
• mean / median / min / max Q

thanks.

[3205914485]

Hi, thanks for the thoughtful question.

In the current codebase, there is no built-in epoch-level summary that automatically reports the overall Q-value distribution across memories. So if one only looks at the default training logs, it is indeed hard to tell how broadly Q updates are taking effect, or how many memories remain at their initial Q value.

That said, this distribution can be obtained quite directly by parsing either:

1. the q-cache stored under local_cache, or
2. the final textual_memory.json dumped into Cube.

So although we do not currently expose these statistics as a standard monitoring dashboard, the underlying data is available and can be summarized offline with a simple pass over the saved memory entries.

As one concrete example, we computed the following Q-value distribution from our saved memories(ALFWorld):

• <= -0.9: 5
• (-0.9, -0.7]: 41
• (-0.7, -0.5]: 231
• (-0.5, -0.3]: 398
• (-0.3, -0.1]: 183
• (-0.1, 0): 559
• == 0: 33463
• (0, 0.1): 0
• [0.1, 0.5): 28
• [0.5, 0.9): 47
• [0.9, 0.99): 19
• [0.99, 1.0]: 211

This confirms your intuition that a large fraction of memories can remain at exactly zero, which is consistent with the possibility that many memories are never retrieved again after initialization.

Thanks again for pointing this out.

[yanan1116]

thanks. on my side it looks the same:

the epoch-level Q distribution summaries:

  - Epoch 1: total memories 3553; nonzero Q 1757 (49.45%); positive 96 (2.70%); zero 1796 (50.55%); negative 1661 (46.75%); mean -0.107384; median 0.000000; min -0.947540; max 0.971752
  - Epoch 2: total memories 7106; nonzero Q 2935 (41.30%); positive 131 (1.84%); zero 4171 (58.70%); negative 2804 (39.46%); mean -0.084525; median 0.000000; min -0.947540; max 0.998028
  - Epoch 3: total memories 10659; nonzero Q 3807 (35.72%); positive 129 (1.21%); zero 6852 (64.28%); negative 3678 (34.51%); mean -0.072732; median 0.000000; min -0.947540; max 0.999385
  - Epoch 4: total memories 14212; nonzero Q 4808 (33.83%); positive 138 (0.97%); zero 9404 (66.17%); negative 4670 (32.86%); mean -0.067401; median 0.000000; min -0.947540; max 0.999980
  - Epoch 5: total memories 17765; nonzero Q 5676 (31.95%); positive 113 (0.64%); zero 12089 (68.05%); negative 5563 (31.31%); mean -0.064121; median 0.000000; min -0.947540; max 0.999943
  - Epoch 6: total memories 21318; nonzero Q 6595 (30.94%); positive 119 (0.56%); zero 14723 (69.06%); negative 6476 (30.38%); mean -0.062103; median 0.000000; min -0.947540; max 0.999999