Notebook to compute stats about label bot

[jlewi]

We need metrics about how many issues were labeled by the label bot and whether they are accurate or not.

The label bot adds a comment to each issue when it adds a label. It also leaves a comment if it can't predict labels with high enough accuracy.

If the label bot is wrong we can detect this by looking to see if someone removed the label applied by the label bot.

We should write some code to compute accuracy of the label bot.

The issue_triage project provides some code that could be a good starting point
We can modify the code below to download recent issues for a specific repository

code-intelligence/py/issue_triage/triage.py

Line 394 in 23ff8f6

def download_issues(self, repo, output, issue_filter=None):

We can then iterate over comments and events to look for label bot comments

[issue-label-bot]

Issue-Label Bot is automatically applying the label kind/feature to this issue, with a confidence of 0.91. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.