New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

relative_position_bias_table initialization #11

Closed

gathierry opened this issue Oct 22, 2021 · 1 comment

gathierry commented Oct 22, 2021

Hi,
In the official code, relative_position_bias_table is initialized in a truncated normal distribution. Is that part missing in this repo?

Official code:
https://github.com/microsoft/Swin-Transformer/blob/6bbd83ca617db8480b2fb9b335c476ffaf5afb1a/models/swin_transformer.py#L110

This implem

Swin-Transformer-TF/swintransformer/model.py

Line 70 in 8986ca7

initializer=tf.initializers.Zeros(), trainable=True)

Owner

rishigami commented Oct 22, 2021

I forgot to implement that. thx!
btw when using pretrained weights, we can ignore the initialization things.

gathierry closed this as completed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment