Mismatch in synonym analysis between ngram and phrase analyzers

[missinglink]

dev ticket to fix problems noted in pelias/pelias#211

the bug effects two 'classes' of tokens (street suffix synonyms/compass directional synonyms), in both cases it is triggered when the final token of the search text is a synonym. The result is that 0 results are returned:

/v1/autocomplete?text=world trade center      # last token 'center' has a synonym 'ctr'
/v1/autocomplete?text=hackney road            # last token 'road' has a synonym 'rd'
/v1/autocomplete?text=30 west                 # last token 'west' has a synonym 'w'

... all return 0 results

.. however it is not triggered when adding a comma and then specifying an 'admin' component:

/v1/autocomplete?text=30 west, new york

... returns >0 results

The reason this is happening is due to a 'mismatch' between how the 'ngrams' analyzer handles synonyms and how the 'phrase' analyzer handles them.

Since the query is split up in to 'finished' tokens and 'unfinished tokens', these different 'types' of tokens get analyzed in different ways.

Eg. 'world trade center', we know that 'world' and 'trade' are finished (the user is done typing them) but the last term 'center' we are not yet sure if this is a partial word or a complete word.

So the first two tokens get sent to the 'phrase' analyzer which is super efficient while the last token has some tricky analysis applied to it.

Since we don't know if it's complete yet we have to check it against the ngrams index; however we have a performance 'hack' in place which uses the phrase analyzer to produce a single token, so instead of using the ngrams analyzer to produce [ 'c', 'ce', 'cen', 'cent', 'center' ] we just produce [ 'center ], this results in a bit of a performance boost as searching the other prefixes adds no value.

The issue with this is that using the peliasPhrase analyzer against an index created using peliasTwoEdgeGram analysis will not work properly because they handle synonyms differently, in the example above the token created is [ 'ctr' ] not [ 'center' ] as expected, it can't find any docs with the ngram 'ctr' and no results are returned.

in progress, more to come.

Connected to pelias/pelias#211

[missinglink]

[edit] I also noticed another issue in Germany:

When specifying the 'short' version of a street name such as Grolmanstr. vs. Grolmanstraße then the phrase analyzer and phrase index fail to match any returns.

eg:

• http://pelias.github.io/compare/#/v1/autocomplete%3Ftext=grolmanstr%2051
• http://pelias.github.io/compare/#/v1/autocomplete%3Ftext=51%20grolmanstr

[missinglink]

acceptance tests: pelias/acceptance-tests@a92393c

[dianashk]

All the acceptance tests pass on production.