Fix 29807, sinusoidal positional encodings overwritten by post_init()

[hovnatan]

What does this PR do?

Fixes #29807, sinusoidal positional encodings overwritten by post_init(). First time contributing, please let me know of any issues, comments.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
  Sinusoidal positional encodings overwritten by post_init() #29807
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@ArthurZucker, @younesbelkada, @amyeroberts

[ArthurZucker]

Hey! IMO we should rather place

        if config.sinusoidal_pos_embds:
            create_sinusoidal_embeddings(
                n_pos=config.max_position_embeddings, dim=config.dim, out=self.position_embeddings.weight
            )

in the _init_weights function.
Also this should be made BC as older version did not have the requirement on _init_weights being the only mode of initializing the weights

[hovnatan]

@ArthurZucker _init_weights() is applied to every module in the model. I moved sinusoidal positional encoding generation after post_init() in my latest commit. Is that ok?

[ArthurZucker]

It does but you can check if the name is "position_embeddings"! 🤗

[hovnatan]

AFAIK post_init() uses pytorch Module's apply(init_func) functionality which doesn't pass name of the module to the init_func(), see https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.apply.

[hovnatan]

Anyways, please look at my latest changes. They solve the problem, but not sure if this is optimal

[ArthurZucker]

Perfect! Thanks for iterating