Add keep_types for filtering by token type

[rmuir]

For example, you can use this combined with UAXURLTokenizer to extract urls from text.

This patch works, but I am unsure about the naming of the filter and the parameters.
In fact this filter in lucene supports two modes "stop" and "keep".

[jpountz]

LGTM

[dweinstein]

👍

[dweinstein]

Will this work for custom patterns? I thought it was great that there existed a UAXURLTokenizer but what if one isn't so lucky in the future and must write one using a custom pattern. What would be the type of those tokens?

[rmuir]

Well this is related to token type, which is like a "tag" for the token that is produced by the analysis chain. Typically the lucene tokenizers provide tags if they recognize different types of tokens, but there is nothing limiting it to that. For example it could contain part-of-speech or whatever is useful.

This filter just filters by tag, it doesnt do tagging itself. It couldn't meet all the possible use cases for token types :)

So to tag by "pattern", we would just need a filter that does that. Its separate from what action to do with the actual tags...

[rmuir]

Adding review just for another opinion on the API before pushing it.

[mikemccand]

LGTM, except minor naming nit: it's called "keep_types" publicly (I like this name) but in the code it's KeepType (without the s).

[clintongormley]

the id of the section should be

[[analysis-keep-types-tokenfilter]]

[rmuir]

Thanks @clintongormley and @mikemccand

I updated the PR.

[mikemccand]

LGTM

[mikemccand]

LGTM, thanks for the rename!