How to handle Index Error Issue

[ShauryaUppal-1Mg]

Should I trim the string to 512 or I can increase the maximum sequence length or something.

[ShauryaUppal-1Mg]

@Tiiiger

[felixgwu]

Hi @ShauryaUppal-1Mg,

Thank you for using our BERTScore.
The reason for this error is that both BERT and RoBERTa are trained with sentences with at most 512 tokens. Unlike the original Transformers, BERT and RoBERTa use learned positional embedding whose size is set during training.
BERTScore is commonly computed between a pair of sentences.
We would suggest that you split the documents into sentences before feeding them to BERT.
If there happen to be some sentences with more than 512 tokens, you can:

1. train a BERT model using longer sentences
2. cut sentences into multiple chunks and design a better way to aggregate them. This is one of the future directions to extend BERTScore, but we haven't studied it yet.
3. use XLNets which support longer inputs. However, they perform worse in our experiments.

Best,
Felix

[ShauryaUppal-1Mg]

Will if work will it for sentence having length > 512, I remove stopwords and common words?

[ShauryaUppal-1Mg]

But BERT as a service allows user to make max_sequence= ignore
Can't we do something with that?

https://github.com/hanxiao/bert-as-service

[felixgwu]

As far as I know, they just trim down the sequence. Please see:
https://github.com/hanxiao/bert-as-service/blob/85690491d66fd1ca0d03924f8c9ead3d1cad90b1/server/bert_serving/server/__init__.py#L414-L422

We would like to follow huggingface's transformers and just raise an error instead.
We encourage users to deal with their own special cases.

Thank you for raising this issue. We will update the README to remind other users.