Training a Model from Huggingface with InfoBERT

[eaberman]

Hi, I am attempting to use an ALBERT model, e.g. albert-base-v2 from https://huggingface.co/albert-base-v2, and usingalbert-base-v2 as the argument when I run source setup.sh && runexp.

Unfortunately, I am getting an AttributeError: 'AlbertForSequenceClassification' object has no attribute 'clear_mask'

Are there any functions I need to implement in order for this to work, or any advice about getting an approach like this to work with your code base?

Thank you!

[boxin-wbx]

Hi,

Thanks for your interest in our work. Our current code base only supports BERT and RoBERTa.

To make ALBERT support our framework, I think you should implement the mask reusing mentioned in the FreeLB paper (https://arxiv.org/pdf/1909.11764.pdf). You can follow how bert.py achieves mask reusing and implements that for ALBERT. Basically, you just need to take care of self.dp_mask and clear_mask().