Skip to content

Unable to Set the bias term for the QKV linear layer in SAB #5148

@a-parida12

Description

@a-parida12

Describe the bug
I pretrained a ViT using GMML method(https://github.com/Sara-Ahmed/GMML). I would like to be able to use this pretrained model as the ViT for UNETr. For every layer I am unable to load checkpoints forvit.blocks.1.attn.qkv.bias.

This largely due to on line

self.qkv = nn.Linear(hidden_size, hidden_size * 3, bias=False)

The bias term has been set False and their no way to access it from the higher level UnetR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions