DictionaryAnnotator fails to annotate at the end of a sentence

[jkirsch]

Given a dictionary of

• John Silver

And a test sentence which shows the match at the end

I am John Silver

The dictionary annotator fails to match.

Sample test that shows the error, adapted from de.tudarmstadt.ukp.dkpro.core.dictionaryannotator.DictionaryAnnotatorTest

@Test
public void testEndOfSentence() throws Exception
{
    AnalysisEngine ae = createEngine(DictionaryAnnotator.class,
            DictionaryAnnotator.PARAM_ANNOTATION_TYPE, NamedEntity.class,
            DictionaryAnnotator.PARAM_VALUE, "PERSON",
            DictionaryAnnotator.PARAM_MODEL_LOCATION, "src/test/resources/persons.txt");

    JCas jcas = JCasFactory.createJCas();
    TokenBuilder<Token, Sentence> tb = new TokenBuilder<>(Token.class, Sentence.class);
    tb.buildTokens(jcas, "I am John Silver");

    ae.process(jcas);

    NamedEntity ne = selectSingle(jcas, NamedEntity.class);
    assertEquals("PERSON", ne.getValue());
    assertEquals("John Silver", ne.getCoveredText());
}

[jkirsch]

The fix might be to change

List<Token> tokensToSentenceEnd = tokens.subList(i, tokens.size() - 1);

to

List<Token> tokensToSentenceEnd = tokens.subList(i, tokens.size());

in DictionaryAnnotator. But I'm not sure if this has been put there for a reason

[reckart]

@jkirsch Could be that the author of the code assumed that a sentence always ends in punctuation token and for some reason wanted to exclude that - no idea. Would you like to do a PR with our fix and unit test?