Bandit Baseline with severity filtering is broken

[ericwb]

If the baseline feature is used along with severity filtering the results end up broken.

Basically, the current baseline routine is to check if there are different results in the currently checked file than what is present in the baseline. If so, all the issues from the current file are inputted and then filtered. To make this clear let's say we have original file a.py:

import subprocess
do_something_benign()
os.system('/tmp/something_bad')

We'll get a low severity finding for subprocess usage, but if we examine this file with a -ll filter, this will be suppressed from the baseline output. We also get a higher severity error which for /tmp.

Now we run a baseline comparison against it, the first action that is performed is checking the results against the previous level. Since the previous level had the LOW severity finding filtered out, it only has the MEDIUM. Since no filtering has been done yet, the current version of the file includes the low severity issue. Basically the baseline has the low severity issue filtered out and the current version doesn't. So they don't match and all of the issues from the file get added to the results.

At this point filtering is done so the LOW severity issue is removed, but the point is even though nothing changed in this file Bandit thinks it did.

The correct behavior is to filter severity first.

[ericwb]

Launchpad bug to Github issue import error