[QUESTION] Batch-normalization in pooling layer

[emjotde]

❓ Question

Hi,
Is the batch-normalization component from the pooling layer from here:

COMET/comet/modules/layerwise_attention.py

Line 139 in a96a688

* _layer_norm(tensor, broadcast_mask, num_elements_not_masked)

mentioned in any of the COMET papers? I don't seem to see it in the main COMET citation.

Can you elaborate a bit why this was useful? It seems unusual to see batch-normalization in NLP applications.

Also, it seems this could be fixed to normalize over a single sentence during inference (use dimensions to only average over time dim) to achieve deterministic behavior consistent with batch-size 1. Would there be any downsides to that?

Thanks,
Marcin

[ricardorei]

Hi @emjotde!

This is a good observation. In the paper we only say that we use the Layerwise scalar mix to combine multiple layers. The implementation we used actually comes from this model.

The idea is that the batch normalisation helps generalisation but in experiments for this year shared task we had more time to play with different hyper parameters and we ended up deactivating it. New models from this year do not use it anymore.

Also, your suggestion is good. I'll test it out

[emjotde]

Thanks for the answer! Feel free to close.