Automatically add and remove the "info needed" label #20970
Conversation
karrtikr
added
the
debt
| runs-on: ubuntu-latest | ||
| if: contains(github.event.issue.labels.*.name, 'triage-needed') && !contains(github.event.issue.labels.*.name, 'info-needed') | ||
| env: | ||
| KEYWORDS: '["\\?", "please", "kindly", "let me know", "try", "inform", "can you", "could you", "would you", "may I", "ask", "provide", "tell me", "give me", "send me", "what", "when", "where", "why", "how"]' |
There was a problem hiding this comment.
These are used to detect when we ask a user for more information. Let me know if there're any more keywords you use. cc/ @microsoft/pyvsc
There was a problem hiding this comment.
I would use a tool that would tell you if a question was asked. I can see this easily getting wrong.
But for key word triggers you can add /info-needed and /more-info
There was a problem hiding this comment.
I initially thought to use ChatGPT itself, but I don't think we have official team credentials on OpenAI for that. Can we create a team account on OpenAI for this? https://openai.com/ cc/ @brettcannon
There was a problem hiding this comment.
But for key word triggers you can add /info-needed and /more-info
I don't see the point of adding this as it would probably be easier to add "info-needed" label itself.
There was a problem hiding this comment.
VS Code core uses this format to add labels, or to trigger bot responses.
There was a problem hiding this comment.
This doesn't need the complexity of AI to decide whether to add or remove a label.
I personally think a slash command with an "argument" of whom we want more info from is the simplest. For instance, /more-info @karrtikr has the benefits of being unambiguous to automation as well as implicitly @ mentioning the person whom we want more data from.
There was a problem hiding this comment.
I think it would be worth considering if false positives or false negatives are worse since this could help determine if we want a smaller or bigger funnel. I think I would find it hard to put "more info" in my post / maybe would just think a label is easier. I think i use "let me know" very often and if I knew it was a trigger word could use it even more so that one feels the most intuitive to me.
There was a problem hiding this comment.
@brettcannon While it technically solves the problem of exactly when to add and remove labels, I personally don't feel like typing /more-info out as I would just add the label instead.
The code as it currently is doing a pretty accurate job of removing labels, so I don't feel we need to be that exact. The problem comes when adding labels.
There was a problem hiding this comment.
I think we want to avoid false positives when adding labels, i.e don't add a label unless we're sure a question is being asked.
There was a problem hiding this comment.
I wrote a Python script to find out the top 10 most frequent keywords in the second last comment across all closed issues with "info-needed" label (excluding stop words, nouns, and pronouns):
- provide: 8 occurrences
- try: 7 occurrences
- using: 6 occurrences
- Can: 4 occurrences
- use: 4 occurrences
- would: 4 occurrences
- mentioned: 3 occurrences
- perform: 3 occurrences
- reproduce: 3 occurrences
- providing: 3 occurrences
Second last because generally the last comment is where we close the issue. Based on this I will remove the words which for sure doesn't indicate that it's a question.
There was a problem hiding this comment.
@karthiknadig Based on what we talked about, it mostly matches with the current list.
| runs-on: ubuntu-latest | ||
| if: contains(github.event.issue.labels.*.name, 'triage-needed') && !contains(github.event.issue.labels.*.name, 'info-needed') | ||
| env: | ||
| KEYWORDS: '["\\?", "please", "kindly", "let me know", "try", "inform", "can you", "could you", "would you", "may I", "ask", "provide", "tell me", "give me", "send me", "what", "when", "where", "why", "how"]' |
There was a problem hiding this comment.
I wrote a Python script to find out the top 10 most frequent keywords in the second last comment across all closed issues with "info-needed" label (excluding stop words, nouns, and pronouns):
- provide: 8 occurrences
- try: 7 occurrences
- using: 6 occurrences
- Can: 4 occurrences
- use: 4 occurrences
- would: 4 occurrences
- mentioned: 3 occurrences
- perform: 3 occurrences
- reproduce: 3 occurrences
- providing: 3 occurrences
Second last because generally the last comment is where we close the issue. Based on this I will remove the words which for sure doesn't indicate that it's a question.
|
Merging this but feel free to let me know of any feedback you may have. My inkling is to try it first. |
Closes #20944