New blacklist module. Fixes #48 #100

Zalgo2462 · 2017-06-26T21:19:19Z

The previous blacklist module possessed some issues which involved mixing up ips, urls, and hostnames. Additionally, it had a hard time distinguishing between blacklisted hosts which initiated connections and received them. These kludges lead to difficulty in expanding the blacklisted module.

This new blacklist module accompanies a new rita blacklist backend. The new backend is meant to handle various types of blacklists, enforce the database abstraction, and provide support for blacklists that are managed by outside software. This has allowed the removal of the google safebrowsing code from RITA and the addition of the code to rita-bl. (Fixes issue #48)

The manner in which user defined lists has changed, as has the RITA config file.
User defined blacklists are expected to be simple line separated lists. These simple files are then registered with RITA in the config file.
Here is an example of a new blacklisted configuration section.

BlackListed:
    Database: rita-blacklist
    # These are blacklists built into rita-blacklist. Set these to false
    # to disable checks against them.
    myIP.ms: true
    MalwareDomains.com: true
    MalwareDomainList.com: true

    # Google SafeBrowsing requires an api key and a file to cache the results in.
    # If either APIKey or Database is an empty string, Google SafeBrowsing will 
    # not be queried.
    SafeBrowsing:
        APIKey: "YOUR API KEY"
        Database: $HOME/.rita/safebrowsing

    # These are custom blacklists that you may define. They are lists of either
    # file paths or urls. These custom blacklists are expected to be simple,
    # line separated text documents containing a list of blacklisted entries.

    # Example: CustomIPBlacklists: ["$HOME/.rita/myIPBlacklist.txt"]
    # myIPBlacklist.txt would look like this:
    # 192.168.0.1
    # 10.10.174.1

    #Lists containing both IPv4 and IPv6 addresses are acceptable
    CustomIPBlacklists: []
    #Lists containing hostnames, domain names, and FQDNs are acceptable
    CustomHostnameBlacklists: ["$HOME/.rita/hostnamesBL.txt"]
    #URLs must each contain a protocol, a host, and a resource
    #Ex: http://google.com/
    #Ex: ftp://myftpserver.com/a/file/over/here.txt
    CustomURLBlacklists: ["$HOME/.rita/urlsBL.txt"]

    # Table names
    SourceIPsTable: blSourceIPs
    DestIPsTable: blDestIPs
    HostnamesTable: blHostnames
    UrlsTable: blUrls

The new blacklist module brings with it four new commands.

     show-bl-hostnames       Print blacklisted hostnames which recieved connections
     show-bl-source-ips        Print blacklisted IPs which initiated connections
     show-bl-dest-ips            Print blacklisted IPs which recieved connections
     show-bl-urls                   Print blacklisted URLs which were visited

The html output has been updated with a page for each of these commands.
Finally, crossref has been simplified to source and destination collections rather than internal/ external. The new blacklist information has been integrated into this new crossref configuration.

Zalgo2462 · 2017-06-30T18:47:27Z

analysis/blacklist/blacklist.go

+	mgo "gopkg.in/mgo.v2"
+)
+
+type resultsChan chan map[string][]blDB.BlacklistResult


We chunk up the checks against RITA-BL. RITA-BL returns maps of (hosts or ip addresses or urls) to their results. The channel is used for communication.

Zalgo2462 · 2017-06-30T18:48:44Z

analysis/blacklist/blacklist.go

+		newList := lists.NewLineSeperatedList(
+			entryType,
+			path,
+			86400,


cache time, could use a comment

comment would be very helpful :)

Zalgo2462 · 2017-06-30T18:51:25Z

analysis/blacklist/blacklist.go

+
+//provide a closure over path to read the file into a line separated blacklist
+func tryOpenFileThenURL(path string) func() (io.ReadCloser, error) {
+	return func() (io.ReadCloser, error) {


Rita-bl expects a function which returns (io.ReadCloser, error) to get the data from.

Zalgo2462 · 2017-06-30T18:52:24Z

analysis/blacklist/blacklist.go

+	ritaBL.SetRPCs(buildBlacklistRPCS(res)...)
+
+	//update the lists
+	ritaBL.Update()


This line forces ritaBL to update its cache.

Zalgo2462 · 2017-06-30T18:53:29Z

analysis/blacklist/blacklist.go

+	//create the data
+	//TODO: refactor these into modules
+	bufferSize := 1000
+	buildBlacklistedIPs(uniqueSourceIter, res, ritaBL, bufferSize, true)


true is for source ip addresses, false if for destination ip addresses.

Zalgo2462 · 2017-06-30T18:56:48Z

config/config.go

@@ -176,6 +183,12 @@ func expandConfig(reflected reflect.Value) {
 			expandConfig(f)
 		} else if f.Kind() == reflect.String {
 			f.SetString(os.ExpandEnv(f.String()))
+		} else if f.Kind() == reflect.Slice && f.Type().Elem().Kind() == reflect.String {


expand lists

…n collections

lisaSW · 2017-07-06T18:02:12Z

analysis/blacklist/blacklist.go

+package blacklist
+
+import (
+	bl "github.com/ocmdev/rita-blacklist2"


At some point, we can probably just rename this rita-blacklist and get rid of the other one...

lisaSW · 2017-07-06T18:05:42Z

analysis/blacklist/blacklist.go

+	//get our data sources
+	ssn := res.DB.Session.Copy()
+	defer ssn.Close()
+	uniqueSourcesAggregation := getUniqueIPFromUconnPipeline("src")


Is there a chance that the "src" and "dst" strings could ever change? Should they be variables?

lisaSW · 2017-07-06T18:08:06Z

analysis/blacklist/blacklist.go

+		res.System.UrlsConfig.UrlsTable,
+	).Find(nil).Iter()
+
+	bufferSize := 1000


Is this something that could or should be in the config file?

lisaSW · 2017-07-06T18:23:13Z

analysis/blacklist/blacklist.go

+
+	//create the data
+	//TODO: refactor these into modules
+	bufferSize := 1000


Could or should this be a config option?

This can be a config option, but I don't see any reason why it should be changed. It is the amount of entries to check against rita-bl at a time. The chunking is mainly for rita-bl RPC's such as google safebrowsing. Some RPC's can take advantage of this chunked behavior when querying the internet. The proper thing to do here is probably change it to a constant in the blacklist package.

Yep, please make this a constant

lisaSW · 2017-07-06T18:27:15Z

analysis/blacklist/blacklist.go

+
+	buildBlacklistedHostnames(hostnamesIter, res, ritaBL, bufferSize)
+
+	buildBlacklistedURLs(urlIter, res, ritaBL, bufferSize, "http://")


What happens in the case of a different url prefix?

So yeah. This is something I really struggled with. RITA-BL expects a prefix for its urls, and bro-ids does not, particularly since it places urls in files separated by protocol. Right now, we only pull urls from the http.log. The buildBlacklistedURLS function, however, is set up to take in a generic url iterator and protocol prefix. If we start reading the ftp log for instance, we can use buildBlacklistURLs(ftpIter, res, ritaBL, buffersize, "ftp://")

sounds good.

lisaSW · 2017-07-06T18:39:21Z

analysis/blacklist/ips.go

+			//So we can safely view buff as an array of strings using a
+			//reinterpret cast. Then, we can dereference the pointer to the array
+			//and use the variadic syntax to pass the array to CheckEntries.
+			indexesArray := (*[]string)(unsafe.Pointer(&buff))


This structure is being treated as an array of strings, but if that ever changes, this will break. Will we ever need to add fields to this structure?

See comment above

lisaSW · 2017-07-06T18:40:09Z

analysis/blacklist/hostnames.go

+	for hostnames.Next(&buff[i]) {
+		if i == bufferSize-1 {
+			//see comment in checkRitaBlacklistIPs
+			indexesArray := (*[]string)(unsafe.Pointer(&buff))


This structure is being treated as an array of strings, but if that ever changes, this will break. Will we ever need to add fields to this structure?

I don't foresee having to change this structure. In this piece of the code we are only checking hostnames. Perhaps hostnames may need to be represented by their individual labels for some reason in the future, but this line can be altered to match that.
Overall since this piece of code deals specifically with hostnames, I felt comfortable going with this solution.
(Same answer for ip addresses since RITA-BL treats both ip v4 and v6 addresses as strings)

sounds good.

lisaSW · 2017-07-06T18:59:59Z

etc/rita.yaml

+    CustomHostnameBlacklists: []
+    # URLs must each contain a protocol, a host, and a resource
+    # Ex: http://google.com/
+    # Ex: ftp://myftpserver.com/a/file/over/here.txt


On this line, it looks like the url prefix is hardcoded to http... Will other prefixes still work and if so, how?

Currently, we will never pull an ftp url out of the bro logs. In the future, we may read the bro ftp.log. We would prepend ftp:// to the urls in that file so as to match these blacklist entries.

sounds good.

lisaSW

I've got a few inline comments up, @Zalgo2462 could you please take a look and reply to my questions?

Zalgo2462 changed the title ~~New blacklist module.~~ New blacklist module. Fixes #48 Jun 26, 2017

This was referenced Jun 30, 2017

RITA does not support TLS encryption / MongoDB authentication #105

Closed

RITA does not display column headers for CSV output #107

Closed

Separate internal mongo settings from config.yaml #103

Closed

Zalgo2462 commented Jun 30, 2017

View reviewed changes

Zalgo2462 force-pushed the new-blacklist branch 2 times, most recently from a862e7d to 221487e Compare June 30, 2017 20:51

logan lembke and others added 15 commits July 3, 2017 10:22

Remove old blacklist module

935e7c7

Fetch data and check it against rita-bl

4b058a2

Integrated rita-blacklist with rita

5ea7240

tested bl ip

336c8c8

Index bl collections

3c5f620

Test hostnames. Fixed urls and tested them

8e76dca

Add custom bl's. Alter config file. Fix urls

6f055e8

Add which lists produced the results

98a03e9

write initial show-blacklisted

89d4467

Rudimentary printing for all bl types

f71ad81

Console output finished

2536c98

Added html reporting for new blacklist

0f14901

Add blacklisted back to xref, simplify xref to source and destionatio…

12fdd42

…n collections

Rename rita-blacklist2 to rita-bl

e53f02d

Update PR to match tls compatible rita-bl

112fd88

Zalgo2462 force-pushed the new-blacklist branch from 221487e to 112fd88 Compare July 3, 2017 16:35

lisaSW self-assigned this Jul 6, 2017

lisaSW self-requested a review July 6, 2017 17:53

lisaSW removed their assignment Jul 6, 2017

lisaSW reviewed Jul 6, 2017

View reviewed changes

change bufferSize to a constant, added a comment to ritabl interaction

27e468c

lisaSW merged commit 73dc84a into activecm:dev Jul 7, 2017

bekirk mentioned this pull request Aug 7, 2020

csv output? #580

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New blacklist module. Fixes #48 #100

New blacklist module. Fixes #48 #100

Zalgo2462 commented Jun 26, 2017 •

edited

Zalgo2462 Jun 30, 2017

Zalgo2462 Jun 30, 2017

lisaSW Jul 6, 2017

Zalgo2462 Jun 30, 2017

Zalgo2462 Jun 30, 2017

Zalgo2462 Jun 30, 2017

Zalgo2462 Jun 30, 2017

lisaSW Jul 6, 2017

lisaSW Jul 6, 2017

lisaSW Jul 6, 2017

lisaSW Jul 6, 2017

Zalgo2462 Jul 6, 2017

lisaSW Jul 7, 2017

lisaSW Jul 6, 2017

Zalgo2462 Jul 6, 2017

lisaSW Jul 7, 2017

lisaSW Jul 6, 2017

Zalgo2462 Jul 6, 2017

lisaSW Jul 6, 2017

Zalgo2462 Jul 6, 2017

lisaSW Jul 7, 2017

lisaSW Jul 6, 2017

Zalgo2462 Jul 7, 2017

lisaSW Jul 7, 2017

lisaSW left a comment


		buildBlacklistedHostnames(hostnamesIter, res, ritaBL, bufferSize)

		buildBlacklistedURLs(urlIter, res, ritaBL, bufferSize, "http://")

New blacklist module. Fixes #48 #100

New blacklist module. Fixes #48 #100

Conversation

Zalgo2462 commented Jun 26, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lisaSW left a comment

Choose a reason for hiding this comment

Zalgo2462 commented Jun 26, 2017 •

edited