Implement token blacklist filtering

[c-w]

Built without using the keyword extractor so that we can offer more predictable behavior to the user: events will only get filtered out if tokens match exactly what the user specified.

Resolves #34

[kevinhartman]

The keyword extractor will only return exact matches. Can you explain how this is different?

There's also a requirement for conjunctive blacklist term filtering, where we filter a post only if it contains a tupleN of blacklisted terms.

I was imagining that this would be implemented by retrieving the set of blacklisted terms present using the keyword extractor, and then determining which conjunctive filter tuples were satisfied by it.

[c-w]

The keyword extractor is case insensitive.

[kevinhartman]

Why should the blacklist be case sensitive, but not the whitelist?

[kevinhartman]

Ideally, this method should be added to the Analyzer interface, and this implementation should exist in AnalysisDefaults. The type of the details parameter there should be changed to accept ExtendedDetails[T] to give each concrete Analyzer the chance to override it, and blacklist will need to be added as a parameter as well.

[c-w]

Is there a scenario in which we do not want to blacklist terms? Also: we have a bunch of other filters that are currently not in the Analyzer trait (hasKeywords, isLanguageSupported). Do you want to include all of those too? If so: this should be a follow-up PR.

[c-w]

Done in 94bf88b. Not convinced that this will or shold be ever over-written though.

[kevinhartman]

Imagine if a pipeline had yet another field that they wanted to blacklist terms from.

I think hasKeywords and isLanguageSupported belong where they are in Pipeline.scala. We extract the keywords and language from the Analyzer since we need those results for our analysis. We don't care what the blacklisted terms were, so we don't need them as an output from the Analyzer.

[c-w]

I'd assert that it's worse to erroneously filter out an event than to erroneously include it. Ergo, we don't want any surprises for blacklist terms. For example, I may want to blacklist "Trump" (as in "the president") but not "trump" (as in "to best").

Additionally, unless I'm missing something, re-using the keyword extractor will
(1) couple us to that implementation so we won't be able to make changes to the keyword code unless they also work for the blacklist code
(2) make the code somewhat less time efficient because we have to re-tokenize the sentence once per blacklist
(3) make the code less memory efficient as we'll store the blacklist words twice
(4) make the code a little bit more complicated to understand

At the end of the day, this implementation also works, but I'm not convinced it's superior:

class Blacklist(blacklist: Seq[Set[String]]) {
  private lazy val extractors = blacklist.map(new KeywordExtractor(_))

  def matches(text: String): Boolean = {
    extractors.zip(blacklist).exists(kv => kv._1.extractKeywords(text).size == kv._2.size)
  }
}

[kevinhartman]

LGTM

[kevinhartman]

What about EnableBlacklist or something that starts with Enable for consistency?

[c-w]

9d38b87

[kevinhartman]

I like this since we can choose to implement this differently later on :)

[kevinhartman]

Is this one still needed after your latest commits?

[c-w]

d5b30ba