More granularity for thresholds

[xrigau]

It'd be nice if I could choose to give each analysis type specific thresholds instead of having global ones. I'd like to do something like this:

staticAnalysis {
    checkstyle {
        penalty {
            maxErrors = 0
            maxWarnings = 15
        }
    }
    pmd {
        penalty {
            maxErrors = 4
            maxWarnings = 10
        }
    }
    findbugs {
        penalty {
            maxErrors = 0
            maxWarnings = 0
        }
    }
}

This way I would have more granularity on what needs fixing and what's ok.

Thoughts?

[mr-archano]

You know, we discussed this when penalty was introduced, but we figured out there will be a full set of scenarios that would need to be addressed if we want to keep the global thresholds.
For instance we will probably be tempted to add a simple validation of the cumulative thresholds, something like:

staticAnalysis.penalty.maxErrors >= checkstyle.penalty.maxErrors + pmd.penalty.maxErrors + findbugs.penalty.maxErrors
staticAnalysis.penalty.maxWarnings >= checkstyle.penalty.maxWarnings + pmd.penalty.maxWarnings + findbugs.penalty.maxWarnings

But what if you don't specify penalties for one of the tools while you have global ones? Are the specific ones to be considered simple overrides of the global ones? Would that be straightforward enough for consumers?

[xrigau]

IMO there shouldn't be a global scope penalty configuration since I don't see it could add any value, I'd say each analysis would have the thresholds set to 0 by default and it is up to the developer to increase those values if needed by passing in a penalty configuration object

[mr-archano]

Right now we are using as default zero tolerance for errors, max tolerance for warnings. I'm not sure we should remove the global settings (could just be used as default for tools without specific thresholds), but I see the value of imposing a full zero tolerance policy and then let users tweak that to fit their needs.
I'd like to collect more thought around this: /cc @blundell @devisnik

[rock3r]

I'd be for the zero-tolerance for anything as default, and then allow to specify per-tool, as @xrigau suggests. I see two scenarios:

• Green field project. This starts at 0 for everything and the restrictive defaults would encourage staying there. If need be (tech debt, etc) those can be overridden further on, while being clearly visible as something that needs addressing.
• Pre-existing project. In this case you'd start with the lowest allowances that make the build green, but make it painfully obvious that things need to be addressed.

I see zero tolerance with the ability to "get a pass" when needed the best possible approach.