Guidance for Blacklisting and Watching

Blacklisting or watching a keyword or a web site address cause the spam detection bot SmokeDetector to trigger an alert whenever that keyword or web site address appears in a post. In other words, it basically says that any post containing this expression is spam, or at least suspicious.

The Website Blacklist

The website blacklist consists of a list of websites associated with known spam that automatically raise suspicion when posted anywhere on Stack Exchange.

Blacklisting a website makes SmokeDetector report every post that is posted or modified with a link to the website (formatted as a link or otherwise) in its text.

The website blacklist is maintained in the SmokeDetector GitHub repository, specifically in the file blacklisted_websites.txt.

The Keyword Blacklist

The keyword blacklist consists of a list of regular expressions which are frequently seen in spam, and only very rarely outside of spam posts.

Blacklisting a "keyword" (which can actually be a regular expression matching a phrase or a more-complex expression with alternatives, like find (?:true )?love which matches either of "find love" or "find true love") causes any post which matches it to be reported as probable spam by SmokeDetector. Matches are not reported in the middle of a word; the keyword expression "dog" does not match "doggone" or "endogenous".

The keyword blacklist is maintained in the SmokeDetector GitHub repository, specifically in the file bad_keywords.txt.

Watch Expressions

"Watching" an expression causes SmokeDetector to report it just like a blacklist expression, but the rule weight is kept at one, in order to prevent matches from triggering autoflagging. The watchlist is considered an "experimental" detection reason. Posts which are detected for only "experimental" reasons are reported into only a limited set of chat rooms, because each room has to opt-in to receiving reports with only experimental detections. All reports are always sent to Charcoal HQ. That means you can use !!/watch to try out different patterns experimentally, just to get an idea of what sorts of posts match a particular expression.

The list of watched expressions is maintained in the SmokeDetector GitHub repository, specifically in the file watched_keywords.txt. The format is slightly different from the other similar files; each entry is a tab-delimited record which includes a date stamp (expressed as Unix epoch, i.e. seconds since midnight Jan 1 1970 UTC), the user name of the person who added the expression, and the regular expression to watch for.

Rules for Blacklisting and Watching

We have established the following rules for watching and blacklisting.

• Blacklisted websites: reserved for sites which we are highly confident that are used only in spam. You may add a site to this list if one of the following is true.
  • The site has at least five hits in metasmoke, with no false positives, and at least one of them is below the default autoflagging threshold (currently, 280) and no older than six months.
  • The site has at least three hits in metasmoke, with no false positives, and it is part of a preexisting campaign. (See below.)
  • There are more than twenty hits in the last six months, and no false positives.
  • There are recent hits, and more than 30 hits overall, and no false positives.
• Blacklisted keywords: reserved for phrases which we are highly confident that are used only in spam. You may add a phrase to this list if the following is true.
  • The phrase has been used repeatedly in recent spam, has no false positives in metasmoke, and searching on Stack Exchange indicates that it is not a common phrase on any site in the network. When searching on Stack Exchange, you need to search in text, in URLs, and in code. It must also meet similar true positive and false positive requirements as are in place for blacklisted websites.
• Watched keywords: anything is game, but be prepared to have it removed if circumstances require it.
  • We will be removing patterns periodically; you can reduce the risk of having useful patterns removed by proactively removing patterns you no longer are interested in, or which produce very uncertain value.
  • Autoflagging weight for this reason is technically forced to stay at 1.
  • SmokeDetector will regard these rules as "experimental"; it will not alert in rooms other than Charcoal HQ and rooms that have specifically opted to receive reports for "experimental" reasons, if there are hits solely from this set of rules.

Preexisting Campaign

The criteria for blacklisting a web site is relaxed when the site is promoted in spam posts which we identify as being part of a "preexisting campaign". This means the spam posts containing the web site can be directly tied to other recent spam posts which contain campaign-specific blacklisted web sites which already fulfill the stricter blacklisting criteria (i.e. not just generic blacklisted terms like "keto"). In practice, this helps us trigger blacklisting early for sites which are clearly part of a series of spam from the same, or closely associated, spammers, where we can be reasonably sure that the only purpose of the site is to have a different URL than the other sites used in the preexisting campaign. (This is called "snowshoe spamming" -- the tactic is to spread your footprint across many sites so as to evade trivial duplicate detection.)

How to Blacklist or Watch Something

You will want to test that the expression you want to blacklist or watch isn't already covered by one of the existing patterns.

You can use !!/bisect <string to test> to test to see if the text is matched by an existing watch or blacklist entry, and find out what that entry is. !!/bisect only tests against the patterns contained in the lists maintained with the !!/watch and !!/blacklist-* commands. You can test the string for matching all detections by using the !!/test <string to test> command (or !!/test-a <string to test> to test as an answer). The !!/test commands will also test against some blacklist and watchlist entries which are contained in findspam.py.

Everyone with SmokeDetector privileges (if you don't have those and would like them, read up on how to get them) can blacklist a website, though this will need to be approved by someone with code privileges if you don't already have them. Additions to the blacklist must be valid regular expressions (regex). In reality that means for largely exact matches (like the website blacklist) that you ensure that special characters (like .) are escaped. (Example: thisisspam\.com)

There are two methods to add a website to a watch list or blacklist:

• Propose a change to the relevant file on GitHub and create a pull request specifying why you want to blacklist the website.

• Use the pertinent chat command in any of the Smoke Detector chatrooms. This will create a pull request containing your change for you.

  • !!/blacklist-website <regex> to blacklist a web site
  • !!/blacklist-keyword <regex> to blacklist a keyword expression
  • !!/blacklist-number <number> to blacklist a phone number (not things like URL path fragments, which can't be obfuscated; use !!/blacklist-keyword for those)
  • !!/blacklist-username <regex> to blacklist a username (display name)
  • !!/watch <regex> to add something to the watch list (keywords, web sites, and usernames)
  • !!/watch-number <number> to add a phone number to the watch list (not things like URL path fragments, which can't be obfuscated; use !!/watch for those)

If you're blacklisting or watching a complex regex to match a whole bunch of different stuff, it's probably better off in the pattern-matching section of findspam.py. You'll need to propose a change to the file on GitHub for this; ask for help if you're unsure what to do. However, if you thing something might be appropriate for the pattern-matching section of findspam.py, it's reasonable to put it on the watchlist first, so we can collect data as to how accurate it is, without giving extra weight to something that might catch a lot of false positive posts.