BertForSequenceClassification.from_pretrained

[alshahrani2030]

Hi, Thank you for this great work.
can I use this code to plot my model(I am useing BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)

model_type = 'bert'
model_version = 'bert-base-uncased'
do_lower_case = True
model = model #(this my model)
#tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
tokenizer = BertTokenizer.from_pretrained(model_version, do_lower_case=do_lower_case)
sentence_a = sentences[0]
sentence_b = sentences[1]
call_html()
show(model, model_type, tokenizer, sentence_a, sentence_b)
I changed only the model with my model, and the sentences and I got this error??!!please help or share any blog that explain how to plot my model
AttributeError: 'BertTokenizer' object has no attribute 'cls_token'

Thank you in advance

[jessevig]

Are you importing the versions of BertTokenizer/BertForSequenceClassification that are in bertviz.pytorch_transformers_attn? This is a forked version of the transformers library specifically for bertviz.

[alshahrani2030]

Are you importing the versions of BertTokenizer/BertForSequenceClassification that are in bertviz.pytorch_transformers_attn? This is a forked version of the transformers library specifically for bertviz.

Hi
Thank you for the quick reply.
yes I did, here is the screenshots

[RuiPChaves]

Have you found a solution? I have a different error when trying to load a BERT fine tuned classification model to analyze via bertviz:

from bertviz.pytorch_transformers_attn import BertForSequenceClassification
from bertviz.pytorch_transformers_attn import BertModel, BertTokenizer
from bertviz.head_view import show

%%javascript
require.config({
  paths: {
      d3: '//cdnjs.cloudflare.com/ajax/libs/d3/3.4.8/d3.min',
      jquery: '//ajax.googleapis.com/ajax/libs/jquery/2.0.0/jquery.min',
  }
});

import torch
from pytorch_pretrained_bert import BertModel, BertTokenizer, modeling
from pytorch_pretrained_bert.file_utils import PYTORCH_PRETRAINED_BERT_CACHE
from pathlib import Path

save_dir = Path('/home/bert/savedmodelfolder/')
save_dir.mkdir(exist_ok=True)
bert_model = 'bert-base-cased'
save_file = save_dir / modeling.WEIGHTS_NAME
config_file = save_dir / modeling.CONFIG_NAME
model_base = BertModel.from_pretrained(
    bert_model,
    cache_dir=PYTORCH_PRETRAINED_BERT_CACHE / 'distributed_{}'.format(-1)
)


model_type = 'bert'
model_version = 'bert-base-cased'
do_lower_case = False


# Loading
model_state_dict = torch.load(save_file)
state_dict_with_prefix = {}
for key, value in model_state_dict.items():
    state_dict_with_prefix['bert.' + key] = value
model_loaded = BertModel.from_pretrained(save_dir, state_dict=state_dict_with_prefix)

tokenizer = BertTokenizer.from_pretrained(model_version, do_lower_case=do_lower_case)
sentence_a = "The cat sat on the mat"
sentence_b = "The cat lay on the rug"
show(model_loaded, model_type, tokenizer, sentence_a, sentence_b)

The error is:

AttributeError                            Traceback (most recent call last)
<ipython-input-15-a0212be47ce3> in <module>
     30 sentence_a = "The cat sat on the mat"
     31 sentence_b = "The cat lay on the rug"
---> 32 show(model_loaded, model_type, tokenizer, sentence_a, sentence_b)

~/bertviz/head_view.py in show(model, model_type, tokenizer, sentence_a, sentence_b)
     57         os.path.join(os.getcwd(), os.path.dirname(__file__)))
     58     vis_js = open(os.path.join(__location__, 'head_view.js')).read()
---> 59     attn_data = get_attention(model, model_type, tokenizer, sentence_a, sentence_b)
     60     params = {
     61         'attention': attn_data,

~/bertviz/attention.py in get_attention(model, model_type, tokenizer, sentence_a, sentence_b, include_queries_and_keys)
     57     else:
     58         if model_type == 'bert':
---> 59             tokens_a = [tokenizer.cls_token] + tokenizer.tokenize(sentence_a) + [tokenizer.sep_token]
     60             tokens_b = tokenizer.tokenize(sentence_b) + [tokenizer.sep_token]
     61             token_type_ids = torch.LongTensor([[0] * len(tokens_a) + [1] * len(tokens_b)])

AttributeError: 'BertTokenizer' object has no attribute 'cls_token'

The tokenizer seems to no know what BERT's CLS tokens are... Any ideas? Thank you.

[jessevig]

Hmm, that's an odd one. When I look at the version of BertTokenizer on github, it inherits from tokenization_utils.PretrainedTransformer, which has a cls_token method. Is it possible that somehow the code in either tokenization_bert or tokenization_utils (both in pytorch_transformers_attn library) are out of date? Or somehow being imported from a different path? BTW, I'm working on a change that will decouple bertviz from the transformers library, so this would likely resolve these sorts of issues. I hope to finish it in the next week or two. Thanks.

[RuiPChaves]

Ah yes, I found the problem. Thanks! Everything is beautiful at your end, the paths were incorrect. Thanks again for this tool.

[jessevig]

Thanks for letting me know. BTW, BertViz is now decoupled from transformers for the head and model views, so there shouldn't be a risk of these issues in the future.

[RuiPChaves]

That was fast! Hopefully transformers will stay stable and nothing will break in the future. They tend to keep changing how things work.

I now can visualize my tuned classification model, but interpreting it is hard in my case. It seems that all the attention heads generally attend equally to all words, across all layers. The pattern is always the same as this:

Is there a simple way to inspect the actual attention scores? If not, I'll probably just modify it so that the numbers appear as well.

Again, many many thanks for this.

[jessevig]

Hmm, this is a problem that has happened with other people who loaded fine-tuned models on the previous version of BertViz, e.g., #12 . They were able to resolve this issue but the cause was never clear. I would recommend trying the new version of bertviz in this case. Let me know if any of these solutions work for you.

[RuiPChaves]

Thanks. Ok, I'm trying to take the code in #12 and update it to the most recent bertviz version. Here's what I got so far (notice the UnicodeDecodeError error when saving the tuned model):

from bertviz import head_view
from transformers import BertTokenizer, BertForSequenceClassification

%%javascript
require.config({
  paths: {
      d3: '//cdnjs.cloudflare.com/ajax/libs/d3/3.4.8/d3.min',
      jquery: '//ajax.googleapis.com/ajax/libs/jquery/2.0.0/jquery.min',
  }
});

model_path = '/bert/outputs/'
save_model = BertForSequenceClassification.from_pretrained('/bert/outputs/pytorch_model.bin', num_labels=2)
torch.save(save_model.state_dict(), model_path)

from bertviz.transformers_attn import BertModel, BertTokenizer
from bertviz.head_view_bert import show
from bertviz.transformers_attn import BertForSequenceClassification

model_version = 'bert-base-cased'
do_lower_case = True
model_state_dict = torch.load(model_path)
assert model_state_dict is not None

model = BertModel.from_pretrained(model_version, state_dict=model_state_dict,output_attentions=True,num_labels=2)
tokenizer = BertTokenizer.from_pretrained(model_version, do_lower_case=do_lower_case)

sentence_a = "The cat sat on the mat"
sentence_b = "The cat lay on the rug"

inputs = tokenizer.encode_plus(sentence_a, sentence_b, return_tensors='pt', add_special_tokens=True)
token_type_ids = inputs['token_type_ids']
input_ids = inputs['input_ids']
attention = model(input_ids, token_type_ids=token_type_ids)[-1]
input_id_list = input_ids[0].tolist() # Batch index 0
tokens = tokenizer.convert_ids_to_tokens(input_id_list)
head_view(attention, tokens)

Error:

I1125 08:20:57.783921 140693827761984 configuration_utils.py:148] loading configuration file /bert//outputs/paraphrase/pytorch_model.bin
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-5-b4e6db36dc04> in <module>
      1 model_path = '/home/rpc/DeepL/Paraphrase/outputs/paraphrase/'
----> 2 save_model = BertForSequenceClassification.from_pretrained('/outputs/pytorch_model.bin', num_labels=2)
      3 torch.save(save_model.state_dict(), model_path)
      4 
      5 from bertviz.transformers_attn import BertModel, BertTokenizer

~/.local/lib/python3.6/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    285                 cache_dir=cache_dir, return_unused_kwargs=True,
    286                 force_download=force_download,
--> 287                 **kwargs
    288             )
    289         else:

~/.local/lib/python3.6/site-packages/transformers/configuration_utils.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    152 
    153         # Load config
--> 154         config = cls.from_json_file(resolved_config_file)
    155 
    156         if hasattr(config, 'pruned_heads'):

~/.local/lib/python3.6/site-packages/transformers/configuration_utils.py in from_json_file(cls, json_file)
    184         """Constructs a `BertConfig` from a json file of parameters."""
    185         with open(json_file, "r", encoding='utf-8') as reader:
--> 186             text = reader.read()
    187         return cls.from_dict(json.loads(text))
    188 

/usr/lib/python3.6/codecs.py in decode(self, input, final)
    319         # decode input (taking the buffer into account)
    320         data = self.buffer + input
--> 321         (result, consumed) = self._buffer_decode(data, self.errors, final)
    322         # keep undecoded input until the next call
    323         self.buffer = data[consumed:]

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

[jessevig]

Thanks for following up. A couple of thoughts:

• Those additional imports from bertviz.transformers_attn may cause problems downstream, though the error you are getting occurs before that.
• I found some related issues on the Transformers github (both open and closed) where users reported UnicodeDecodeErrors when loading fine-tuned models: https://github.com/huggingface/transformers/issues?utf8=%E2%9C%93&q=unicodedecodeerror . It seems that this error message is a catch-all for different types of problems with model configuration.
  Anyways, please let me know if you are able to find a solution there.

[RuiPChaves]

Thanks again for the quick reply. This helped. You were right, the extra calls were necessary and there were problems with paths again. I got it to work, but I'd like to see the "Attention" dropbox menu Sentence_A -> Sentence_B menu. How do I bring it back? Thank you for your patience.

[jessevig]

Hi Rui, I neglected to mention that as part of that change, I removed the functionality to filter the attention, because it was harder to implement when using transformers directly. I do plan to add the functionality back soon, but I don't have an exact timeline unfortunately.

[RuiPChaves]

I see. I found that view quite intuitive (especially when examining longer strings). I'll be on the lookout for it's return. Cheers!

[jessevig]

Hi, I've created a new branch which has the Attention dropdown: https://github.com/jessevig/bertviz/tree/sentence_pair_filter . See head_view_bert.ipynb. Please give it a try!

[RuiPChaves]

Fantastic. This works nicely.
Much appreciated!

[jessevig]

Good to hear! BTW, it's in master now too.

[wsy258-strar]

'bert-base-cased'

我也是这个问题 请问是哪个路径的问题？