Skip to content
This repository has been archived by the owner. It is now read-only.

Notebook to compute stats about label bot #71

Open
jlewi opened this issue Nov 27, 2019 · 1 comment
Open

Notebook to compute stats about label bot #71

jlewi opened this issue Nov 27, 2019 · 1 comment
Labels
help wanted kind/feature

Comments

@jlewi
Copy link
Contributor

@jlewi jlewi commented Nov 27, 2019

We need metrics about how many issues were labeled by the label bot and whether they are accurate or not.

The label bot adds a comment to each issue when it adds a label. It also leaves a comment if it can't predict labels with high enough accuracy.

If the label bot is wrong we can detect this by looking to see if someone removed the label applied by the label bot.

We should write some code to compute accuracy of the label bot.

The issue_triage project provides some code that could be a good starting point
We can modify the code below to download recent issues for a specific repository

def download_issues(self, repo, output, issue_filter=None):

We can then iterate over comments and events to look for label bot comments

@issue-label-bot
Copy link

@issue-label-bot issue-label-bot bot commented Nov 27, 2019

Issue-Label Bot is automatically applying the label kind/feature to this issue, with a confidence of 0.91. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

@jlewi jlewi added the help wanted label Nov 27, 2019
@jlewi jlewi changed the title Notebook to compute states about label bot Notebook to compute stats about label bot Dec 10, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
help wanted kind/feature
Projects
None yet
Development

No branches or pull requests

1 participant