Some questions about the chunk and window

[runnerhdh]

Nice work！
I didn't quite understand some of the settings in the paper. What does "chunk" refer to in the paper, and how does it differ from "window"? Doesn't the key-value cache only exist within the window?
Besides， the paper states, "For a non-keyframe i assigned to keyframe k, its tokens attend only to tokens from k and from frames between k and i under the causal or window mask." Thus, for non-keyframe k+1, it can only interact with the kv chche of k-th？Wouldn't such a small attention window (1?) easily drift away? Is it equivalent to starting attn from k again? Am I misunderstanding something?
Looking forward to your reply.

[nicolascc1]

Thanks for the great question.

For a long batch, e.g. 200 frames, we cannot train on all frames at once due to memory limits. So we split the sequence into multiple chunks, and each chunk contains only around 20 frames.

By contrast, window size refers to how many frames of KV cache are kept. For example, window size 10 means the model can directly access only the most recent 10 frames.

For the second point, the paper is describing the pose token attention. Before readout, pose tokens have already gone through the aggregator with spatiotemporal mixing, so they already contain longer-range information. The attention mask here is mainly for cleaner semantic control, not because the model only has access to frame k. So it is not simply “restarting attention from k” each time.

[runnerhdh]

Thanks for the great question.

For a long batch, e.g. 200 frames, we cannot train on all frames at once due to memory limits. So we split the sequence into multiple chunks, and each chunk contains only around 20 frames.

By contrast, window size refers to how many frames of KV cache are kept. For example, window size 10 means the model can directly access only the most recent 10 frames.

For the second point, the paper is describing the pose token attention. Before readout, pose tokens have already gone through the aggregator with spatiotemporal mixing, so they already contain longer-range information. The attention mask here is mainly for cleaner semantic control, not because the model only has access to frame k. So it is not simply “restarting attention from k” each time.

I got :)
Another question， the config in yaml:
"detach_reference": rel_pose_head_cfg.get( "detach_reference", False ),
During training, do you always use detach_reference with the value true or false?
Besides, I'd also like to ask if, during training, the loss function, especially the camera loss, experiences significant spikes and fluctuations.
Thank you very much.