It's time to benchmark sast-scan

[prabhu]

Let's benchmark this project using OWASP/Benchmark

As you might know that Find Sec Bugs has a great detection rate of 39% as of last test.

https://github.com/OWASP/Benchmark/blob/master/scorecard/benchmark_comparison.png

[prabhu]

Happy to announce that sast-scan has scored a whopping 42% thanks to the bundled Find Security Bugs plugin version 1.10.1

Benchmark_v1.2_Scorecard_for_SBwFindSecBugs_v1.10.1.csv.txt

False positives

SQL and XPath injection

In BenchmarkTest00104.java the following code snippet is used.

int num = 86;
if ( (7*42) - num > 200 )
   bar = "This_should_always_happen"; 
else bar = param;


try {
    String sql = "SELECT * from USERS where USERNAME='foo' and PASSWORD='" + bar + "'";
 ...

Find-Sec-Bugs cannot evaluate this if condition being a SAST product and hence cannot determine the fact that bar would always get set to This_should_always_happen.

In BenchmarkTest00116.java the code tries to trick the scanner by using an HashMap

String bar = "safe!";
java.util.HashMap<String,Object> map51005 = new java.util.HashMap<String,Object>();
map51005.put("keyA-51005", "a_Value"); // put some stuff in the collection
map51005.put("keyB-51005", param); // put it in a collection
map51005.put("keyC", "another_Value"); // put some stuff in the collection
bar = (String)map51005.get("keyB-51005"); // get it back out
bar = (String)map51005.get("keyA-51005"); // get safe value back out

The bar value would be set to a safe value in this case so there is no XPath injection possibility here

String expression = "/Employees/Employee[@emplid='"+bar+"']";

I think the Benchmark project in general is a fun project to look at. I'm glad that sast-scan powered by the best oss tools fared quite well in this.