Add template validation bot

[chadwhitacre]

Auto close issues and PRs not adhering to the templates

Example: https://github.com/seferov/pr-lint-action

Todo

• better log message than full echo template
• auto-close on fail, with comment
• write real comment
• deploy to onpremise
• close hole: Remove injection vector self-hosted#822
• look into \r bug: Add template validation bot #8 (comment)
• relax match constraint: Make issue validation less aggressive #15
• harden against additional ISSUE_TEMPLATE situations: Account for different ISSUE_TEMPLATE situations #26
• make log download more robust (seeing off-by-ones, probably race condition?) → Test suite occasionally fails with issue number mismatch #30
• probably convert to an action
• deploy to onpremise
• deploy to sentry-docs
• deploy to sentry

[chadwhitacre]

I want to review our templates as part of this.

I also want to consider automation that nudges people to reveal more info vs. hard-closing right off the bat. There was this early AI bot called Dr. Sbaitso that would basically just rubber duck. "How do you feel? Tell me more about your problems." That kind of thing. I find this approach can be an effective way to turn a bad first post into an interesting conversation. "What version are you running? What have you done to debug? What hypotheses do you have?" I'd like to explore automating that vs. automating "No soup for you!"

[chadwhitacre]

Like maybe having a bot watch for the Needs More Info label and following up with a comment?

[chadwhitacre]

Fire on issues, need to compare against all possible template types. Want to use single source of truth for template definition. Easy heuristic would be to compare headings in description and template, roughly:

grep '^#' template > expected
grep '^#' issue-description > actual
diff -rub expected actual

[chadwhitacre]

No-op GHA that fires on issue open:

https://github.com/chadwhitacre/chadwhitacre-testing-nothing-to-see-here/actions/runs/481292243

Takes 18 seconds to run. Seems like a lot to run on every single issue open. 🐭

[chadwhitacre]

actual: https://github.com/chadwhitacre/chadwhitacre-testing-nothing-to-see-here/runs/1691291348?check_suite_focus=true#step:2:6

next step is to pull expected ... clone repo and read .github/ISSUE_TEMPLATE? do I already have a repo clone?

[chadwhitacre]

MVP: https://github.com/chadwhitacre/chadwhitacre-testing-nothing-to-see-here/issues/20#issuecomment-759751070

[chadwhitacre]

Choking on \r?

https://github.com/getsentry/onpremise/runs/1708876463?check_suite_focus=true

2021-01-15T13:48:33.7472458Z /home/runner/work/_temp/b836d143-90e1-4150-bbcb-6cc9d82121f5.sh: line 54: $'\r': command not found
2021-01-15T13:48:33.7481004Z /home/runner/work/_temp/b836d143-90e1-4150-bbcb-6cc9d82121f5.sh: line 55: TypeError: command not found
2021-01-15T13:48:33.7487873Z /home/runner/work/_temp/b836d143-90e1-4150-bbcb-6cc9d82121f5.sh: command substitution: line 56: syntax error near unexpected token `('
2021-01-15T13:48:33.7490627Z /home/runner/work/_temp/b836d143-90e1-4150-bbcb-6cc9d82121f5.sh: command substitution: line 56: `r.takeRecords is not a function. (In 'r.takeRecords()', 'r.takeRecords' is undefined)
2021-01-15T13:48:33.7492405Z '
2021-01-15T13:48:33.7504446Z /home/runner/work/_temp/b836d143-90e1-4150-bbcb-6cc9d82121f5.sh: line 54: $'\r': command not found
2021-01-15T13:48:33.7513927Z /home/runner/work/_temp/b836d143-90e1-4150-bbcb-6cc9d82121f5.sh: line 55: TypeError: command not found
2021-01-15T13:48:33.7520816Z /home/runner/work/_temp/b836d143-90e1-4150-bbcb-6cc9d82121f5.sh: command substitution: line 56: syntax error near unexpected token `('
2021-01-15T13:48:33.7523228Z /home/runner/work/_temp/b836d143-90e1-4150-bbcb-6cc9d82121f5.sh: command substitution: line 56: `r.takeRecords is not a function. (In 'r.takeRecords()', 'r.takeRecords' is undefined)
2021-01-15T13:48:33.7524920Z '
2021-01-15T13:48:33.7534951Z /home/runner/work/_temp/b836d143-90e1-4150-bbcb-6cc9d82121f5.sh: line 54: $'\r': command not found
2021-01-15T13:48:33.7542001Z /home/runner/work/_temp/b836d143-90e1-4150-bbcb-6cc9d82121f5.sh: line 55: TypeError: command not found
2021-01-15T13:48:33.7546351Z /home/runner/work/_temp/b836d143-90e1-4150-bbcb-6cc9d82121f5.sh: command substitution: line 56: syntax error near unexpected token `('
2021-01-15T13:48:33.7548485Z /home/runner/work/_temp/b836d143-90e1-4150-bbcb-6cc9d82121f5.sh: command substitution: line 56: `r.takeRecords is not a function. (In 'r.takeRecords()', 'r.takeRecords' is undefined)
2021-01-15T13:48:33.7549707Z '
20

[chadwhitacre]

Choking on \r?

Nope. Dutifully performing command substitution. This is another demonstration of the vuln class in getsentry/self-hosted#822, and a confirmation that the first approach to fixing it was indeed insufficient:

$ BAR="`head -n3 /etc/passwd`"
$ echo "$BAR"
##
# User Database
# 
$

[chadwhitacre]

Done and deployed! 💃

getsentry/self-hosted#845
getsentry/sentry-docs#2988
getsentry/sentry#23493