The impacts of BatchNorm instead of LayerNorm

[Ye-D]

In the CrypTen-based MPCFormer, such as MPC-Bert, I noticed that the LayerNorm is replaced by BatchNorm (due to CrypTen does not support LayerNorm now). Will this modification influence Bert's performance (e.g., accuracy)? Due to the source code did not give the model-loading scripts, it might be difficult for me to check it in experiments. Thanks for your help.

[DachengLi1]

@Ye-D thanks for the great question! We measure accuracy in plain PyTorch, and only system throughput with Crypten, as reported in the paper. We are implementing modules such as Layernorm in Crypten to measure the accuracy.

By module loading, do you mean something like this ?

[Ye-D]

@DachengLi1 Thanks for the reply!