Google Safebrowsing API v2 for Scala and Java

This project implements the Google Safebrowsing API v2 and the Google Safebrowsing Lookup API in Scala. It also works with Yandex Safebrowsing API.

See Zscaler blog post for other implementations and notes

Usage in Scala

Safe Browsing API

val apikey = "123456"
val dburl = "jdbc:mysql://localhost:3306/googlesafebrowsing2"
val tablePrefix = "gsb2_"
val storage = new MySQL(LiteDataSource.driverManager(dburl, "user", "pass"), tablePrefix)
val sb2 = new SafeBrowsing2(apikey, storage)
// uses Google database by default, you can use Yandex like this:
// val sb2 = new SafeBrowsing(apikey, storage, "yandex")

// update database
val forceUpdate = false
val useMac = false
val secondsToWaitBeforeNextUpdate = sb2.update("", forceUpdate, useMac)

// lookup url in database
val list = "" // lookup in all lists
val listMatch = sb2.lookup("http://ianfette.org", list, useMac)
listMatch match {
  case Some(list) => println("Match found in list: " + list)
  case None => println("No match found")
}

Lookup API

val resp = new Lookup(apikey).lookup(Array("http://www.google.com/", "http://ianfette.org/"))
resp.foreach(a => {
  println(a._1 + " -> " + a._2)
})

Outputs:

http://www.google.com/ -> ok
http://ianfette.org/ -> malware

Usage in Java

Safe Browsing API

String apikey = "123456";
String url = "jdbc:mysql://localhost:3306/googlesafebrowsing2";
String tablePrefix = "gsb2_";

Storage storage = JavaHelper.buildStorageMySQL(url, "user", "password", tablePrefix);
SafeBrowsing2 sb2 = new SafeBrowsing2(apikey, storage);

// update the database
boolean forceUpdate = false;
boolean useMac = false;
String[] updateLists = null; // update all lists
int wait = sb2.update(updateLists, forceUpdate, useMac);
System.out.println("Seconds to wait before next update: " + wait);

// lookup url in database
String[] lookupLists = null; // lookup in all lists
String match = sb2.jlookup("http://ianfette.org", lookupLists, useMac);
if (match != null)
	System.out.println("Match found in list: " + match);

Lookup API

Lookup lookup = new Lookup(apikey, "appname");
Map<String, String> r = lookup.jlookup(new String[]{"http://ianfette.org"}, 0);
for (String key : r.keySet()) {
	System.out.println(key + " -> " + r.get(key));
}

Test URLs

The follow URLs are used to test Safe Browsing API:

Malware

• http://www.ianfette.org
• http://www.ianfette.org/test/123
• http://ianfette.org
• http://ianfette.org/test/123
• http://www.gumblar.cn
• http://www.gumblar.cn/test/123
• http://gumblar.cn
• http://gumblar.cn/test/123
• http://malware.testing.google.test/testing/malware/

Phishing

• http://freepaccek.w.interia.pl
• http://freepaccek.w.interia.pl/test/123

Good

• http://www.google.com
• http://google.com
• http://malware.testing.google.test
• http://malware.testing.google.test/testing/malware

Database support

The following databases are currently supported:

• MySQL
• MS SQL
• HSQLDB

New Storage classes can be added by extending the net.google.safebrowsing2.db.DBI class or the net.google.safebrowsing2.db.Storage trait.

Dependencies

• slf4j-api-1.6
• joda-time-1.6.2
• scala-http-client-1.0
• Apache httpclient-4.1
• Apache httpcore-4.1
• Apache commons-codec-1.4
• scala-library-2.9.1

Attributions

• The is based off Julien Sobrier's Net-Google-SafeBrowsing2 perl module.
• URL Canonicalization by Dave Shanley as part of jgooglesafebrowsing.
  • Subsequently heavily modified by Edward Chu
• ExpressionGenerator inspired by the google-safe-browsing python module.
• JDBC API by Stepan Koltsov
• Edward Chu
• Alexey Lapitsky