Different POS tags for same sentence repeated in paragraph

[garyspatterson]

I am seeing odd behavior with regards to fine-grained POS tags for a text with identical repeated sentences: 'The cactus also bears fruit. The cactus also bears fruit.' For the first sentence, the 'cactus' token is tagged as NN, whereas in the second sentence, it is NNS. If you take away the 'also' in the second sentence, the tag is correctly 'NN'. I had assumed that POS tagging was done at the sentence level of analysis, so I'm curious why this is happening. Thanks!

for t in sent:
print t, t.tag_, t.dep_

The DT det
cactus NN nsubj
also RB advmod
bears VBZ ROOT
fruit NN dobj
. . punct
The DT det
cactus NNS nsubj
also RB advmod
bears VBZ ROOT
fruit NN dobj
. . punct

Your Environment

spaCy 1.6
spyder 3.0.2
Error also replicates on displacy.

[honnibal]

The POS tagger has always been document level -- it's the parser that decides the sentence boundaries (relying heavily on POS tag features).

This specific example is interesting though. I wouldn't have predicted this, especially across the previous model as well.

[garyspatterson]

OK cool thanks for the reply.

[ines]

Closing this and making #1057 the master issue – work in progress for spaCy v2.0!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.