I agree that shingle filter needs to have some options here but as a workaround you can specify a lucene_version with your stop filter to still set the enable_position_increment

OK thanks for the tip - that works for now. Presumably if position increment disabling has been removed from the Lucene core then the only workaround is to modify the shingle filter. I had a look at the Lucene source and it looks there is currently no method to override the filler token to an empty string which is currently hardcoded as an underscore.

@MrHash yes that is correct. I think there needs to be one. I hope I will be able to open an issue and fix that in lucene soon. Feel free to beat me!

here is an issue https://issues.apache.org/jira/browse/LUCENE-5353

Nice. I noticed also that the remove_trailing=false option of the stop filter also generates a filler token, despite enable_position_increments being set to false. Hopefully this update should also take care of this issue.

with lucene 4.7 we will be able to make the filler token configurable.

Good stuff.

By using filler_token = "" you may end up with duplicate tokens in the token stream. This is not the same behavior as it was by using enable_position_increment. Is there any way to bypass this problem?

+1

@alup Could you open a new issue and provide more detail of the problem please?

Ok, when I find some time, I will come up with an example to denote the problem.

@alup Did you find the work around for this? I did post a question here here as well.

@apanimesh061 could you open a new issue for this, and include a full JSON recreation?