Question about gradient backpropagation to object queries.

[smartbarbarian]

Assume we only have four object queries, the batch size is 1, only the last decoder layer will output loss, and negative labels will not be calculated into the loss. If we set tgt_mask like
[[False, False, True, True],
[False, False, True, True],
[True, True, False, False],
[True, True, False, False],]
And only the first object query matches the True label within one step during training, the last two object queries will not be updated during gradient backpropagation. But I found that all object queries have been updated. Can anyone do me a favor to clarify it? Thank you so much.
I found it may be caused by the momentum in optimzer. So it's is correct such tgt masks can prevent interactions among object queries?