Recommend term deletion/modification once a specific TP/FP ratio has been reached

[ArcticEcho]

For example, whenever someone FP/TPs a report, Pham could quickly check each found blacklist term's TP/FP ratio to determine whether it is returning an unusually high number of FPs. If, say the term's ratio is <1:5, Pham would suggest that the term needs editing (to try to improve its "accuracy"). And if the term's ratio is above 1:10, Pham would suggest term deletion.

Any thoughts?

[Unihedro]

So we'll have Recommend Review: term (term here) message?

[ArcticEcho]

I was kinda thinking of The following term(s): (term(s) here), attract a high number of FPs. Term review is recommended. and The following term(s): (term(s) here), attract a *very* high number of FPs. Term deletion is recommended. Come to think of it, a ratio of 1 to 5, and 1 to 10 are probably a little too high. Perhaps 1 to 3, and 1 to 6, instead?

[honnza]

The correct ratios depend on the amount of spam during the measurement
period. A better solution would be based on sensitivity and specificity
(say, sens + spec < 100%?)

On Wed, Nov 5, 2014 at 11:19 AM, Sam notifications@github.com wrote:

I was kinda thinking of The following term(s): (term(s) here), attract a
high number of FPs. Term review is recommended. and The following
term(s): (term(s) here), attract a very high number of FPs. Term deletion
is recommended. Come to think of it, a ratio of 1 to 5, and 1 to 10 are
probably a little too high. Perhaps 1 to 3, and 1 to 6, instead?

—
Reply to this email directly or view it on GitHub
#31 (comment)
.

[ArcticEcho]

Yes, using the sens/spec metrics would seem to be a much better solution. The exact ratios may need some adjusting, shall we start with <25% = recommend deletion, <50% = recommend review?

[Unihedro]

<offtopic>

How come what you wrote in your comment was different from what I received through mail?

</offtopic>

[honnza]

spec + sens is always <= 200%

.. or should be, if the estimates were at least somewhat sensible, which
I'm not quite sure of, yet. Let's fix the stats first before implementing a
stat-based term review system

On Wed, Nov 5, 2014 at 12:04 PM, Sam notifications@github.com wrote:

Yes, using the sens/spec metrics would seem to be a much better solution.
The exact ratios may need some adjusting, but shall we start with <100% =
recommend deletion, <200% = recommend review?

—
Reply to this email directly or view it on GitHub
#31 (comment)
.

[ArcticEcho]

@Vincentyification I edited my comment after discovering a bug with the sens/spec calculations.

[ArcticEcho]

@honnza Agreed.

[ghost]

Would it be a good idea to keep a list of removed terms? This way, if a user attempts to re-add a term on said list, Pham would reply with how it was removed, the term's stats (preserved from time of deletion), and listen for a y/n command whether to add it back.

[ArcticEcho]

Good idea. I'll add that too.