Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Notebook to compute stats about label bot #71

Open
jlewi opened this issue Nov 27, 2019 · 1 comment
Open

Notebook to compute stats about label bot #71

jlewi opened this issue Nov 27, 2019 · 1 comment

Comments

@jlewi
Copy link
Contributor

@jlewi jlewi commented Nov 27, 2019

We need metrics about how many issues were labeled by the label bot and whether they are accurate or not.

The label bot adds a comment to each issue when it adds a label. It also leaves a comment if it can't predict labels with high enough accuracy.

If the label bot is wrong we can detect this by looking to see if someone removed the label applied by the label bot.

We should write some code to compute accuracy of the label bot.

The issue_triage project provides some code that could be a good starting point
We can modify the code below to download recent issues for a specific repository

def download_issues(self, repo, output, issue_filter=None):

We can then iterate over comments and events to look for label bot comments

@issue-label-bot

This comment has been minimized.

Copy link

@issue-label-bot issue-label-bot bot commented Nov 27, 2019

Issue-Label Bot is automatically applying the label kind/feature to this issue, with a confidence of 0.91. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

@jlewi jlewi added the help wanted label Nov 27, 2019
@jlewi jlewi changed the title Notebook to compute states about label bot Notebook to compute stats about label bot Dec 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.