Issues with dangling zshot.linker.LinkerRegen

[louisguitton]

Hi, I am trying to use the library and I am running into issues with zshot.linker.LinkerRegen.

if I try to run a pipeline on a document that is "too long", the pipeline stays dangling with no error message.
I see from the docs that there is a max_input_len parameter.

Name | Type | Description | Default
-- | -- | -- | --
max_input_len |   | Max length of input | 384

What happens if the input text is longer than this?

[marmg]

Hello. The max_input_len is the maximum number of tokens the model can use as input. When we create the input sentences, we need to add special tokens to specify which are the entities to classify, we create one different sentence for each entity in the text.

In the case that the input text is longer than the max_input_len, we truncate the input text, keeping both left and right context of the entity, with the same number of tokens in both left and right side.

The linker will see all the mentions as we are creating one input sentence for each mention, but the context will be limited to that number of tokens. If the text is too long and there are too many mentions in it, this will result in running inference n times, one per each mentions, which may result in a really long time.

I hope this answers your question.