Problem with integrating BERT (tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[0,512] = 512 is not in [0, 512))

[MajaRolevski]

Rasa version: 1.10.7 (also tried with 1.8.3)

Python version: 3.7

Operating system (windows, osx, ...): Linux and Windows

Issue:
I have a problem with integrating BERT into the configuration of my model. Namely, at the very beginning of training it gives the error message shown below. There is also a config.yml file I used.

It is interesting that I am not getting this error when using a very small train dataset, so the model is able to finish with training successfully.

Error (including full traceback):

2020-07-16 13:05:16 INFO     rasa.nlu.model  - Starting to train component HFTransformersNLP
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/bin/rasa", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/__main__.py", line 92, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/cli/train.py", line 140, in train_nlu
    persist_nlu_training_data=args.persist_nlu_data,
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/train.py", line 414, in train_nlu
    persist_nlu_training_data,
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/train.py", line 453, in _train_nlu_async
    persist_nlu_training_data=persist_nlu_training_data,
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/train.py", line 482, in _train_nlu_with_validated_data
    persist_nlu_training_data=persist_nlu_training_data,
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/nlu/train.py", line 90, in train
    interpreter = trainer.train(training_data, **kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/nlu/model.py", line 191, in train
    updates = component.train(working_data, self.config, **context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/nlu/utils/hugging_face/hf_transformers.py", line 472, in train
    batch_docs = self._get_docs_for_batch(batch_messages, attribute)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/nlu/utils/hugging_face/hf_transformers.py", line 423, in _get_docs_for_batch
    ) = self._get_model_features_for_batch(batch_token_ids)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/nlu/utils/hugging_face/hf_transformers.py", line 387, in _get_model_features_for_batch
    batch_attention_mask, padded_token_ids
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/nlu/utils/hugging_face/hf_transformers.py", line 347, in _compute_batch_sequence_features
    np.array(padded_token_ids), attention_mask=np.array(batch_attention_mask)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 822, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/transformers/modeling_tf_bert.py", line 708, in call
    outputs = self.bert(inputs, **kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 822, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/transformers/modeling_tf_bert.py", line 570, in call
    embedding_output = self.embeddings([input_ids, position_ids, token_type_ids, inputs_embeds], training=training)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 822, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/transformers/modeling_tf_bert.py", line 150, in call
    return self._embedding(inputs, training=training)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/transformers/modeling_tf_bert.py", line 173, in _embedding
    position_embeddings = self.position_embeddings(position_ids)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 822, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/layers/embeddings.py", line 184, in call
    out = embedding_ops.embedding_lookup(self.embeddings, inputs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/embedding_ops.py", line 323, in embedding_lookup
    transform_fn=None)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/embedding_ops.py", line 137, in _embedding_lookup_and_transform
    array_ops.gather(params[0], ids, name=name), ids, max_norm)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py", line 4106, in gather
    return params.sparse_read(indices, name=name)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 645, in sparse_read
    self._handle, indices, dtype=self._dtype, name=name)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_resource_variable_ops.py", line 556, in resource_gather
    _ops.raise_from_not_ok_status(e, name)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 6606, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[0,512] = 512 is not in [0, 512) [Op:ResourceGather] name: tf_bert_model/bert/embeddings/position_embeddings/embedding_lookup/

Command or request that led to error:

rasa train nlu

Content of configuration file (config.yml) (if relevant):

language: en
num_threads: 8
pipeline:
- name: HFTransformersNLP
  model_weights: "bert-base-uncased"
  model_name: "bert"
- name: LanguageModelTokenizer
- name: LanguageModelFeaturizer
- name: "CRFEntityExtractor"
- name: "EmbeddingIntentClassifier"

[sara-tagger]

Thanks for raising this issue, @tabergma will get back to you about it soon✨

Please also check out the docs and the forum in case your issue was raised there too 🤗

[tabergma]

@MajaRolevski Thanks for raising the issue. How small is your "very small dataset"? And how does your "normal" dataset look like that resolves in this issue?

[MajaRolevski]

@tabergma Thank you for the answer. My "very small dataset" consists of 19 classes (intents); each class has 10-30 sentences, except one that has 347 sentences and the other with 1897. On the other hand, the "normal" dataset contains 19 classes, where each class has on average 70-100 sentences, except one with 1565 and the other with 41564 sentences.

[dakshvar22]

@MajaRolevski Looks like your dataset has some sentences which are very long(>512 tokens). We'll add a fix soon so that the training doesn't break because of this but I would also suggest you to sanitize your data and possibly clean up such sentences because they are unusual and could be very noisy for training.

[hotzenklotz]

We are facing the same issue. Our response selector has some rather long answers, so we run into the BERT token limit as well. When I shorten my answers, the model trains fine again. It is hard to find a good cut off length for the answers though with the different tokenizer and libs involved.

I am looking forward to fix for this :-)

[hotzenklotz]

@dakshvar22 Thanks for fixing this. Will the fix be available for Rasa version 1.10.x or only Rasa v2?

[thanhphi0401]

any news on it? i'm facing with this problem

[zr-smile]

i'm facing with this problem too，

tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[3453,0] = 1780522202 is not in [0, 120000)
----[[node net_input/embedding/embedding_lookup_55 (defined at ./check_ctr_v1.py:410) ]]

I define tf.saved_model.loader.load(sess, [tf.saved_model.tag_constants.SERVING], self.saved_model_dir) in check_ctr_v1.py:410

[SamMohel]

Did someone solve it , please ?

[zr-smile]

您好，您的邮件已收到，我会尽快查看，

[MohanKrishna-RC]

What could be the possible solution, or just a try except situation to handle.