Implement a build bisect script

[bsclifton]

Description

Goal is to have a script that automates downloading/installing versions of browser for your platform

• fetches all versions
• will attempt to find good/bad for start
• downloads binary from GitHub

WIP checklist

TODOs before it's ready:

• need to get ALL releases (I believe it's only getting first page from GitHub API)
• search is not specific enough (stops after first result is different than last)
• need to implement install
• need to implement execute
• display all the commits for a given version (to help narrow search for what broke)
• be able to provide a parameter for profile to use
• documentation (edit: available at https://github.com/brave/brave-browser/wiki/Build-Bisect-Tool)
• schedule a demo to folks (edit: scheduled for March 14th, 2019)

Reviewer notes

• Currently requires a GitHub token. This would be defined in your .npmrc with an entry like BRAVE_GITHUB_TOKEN=<TOKEN-GOES-HERE>
• Currently only implemented for macOS!
• Close all instances of Brave before trying this!
• Temporary directories are created for the profiles. No need to worry about having your data overwritten (unless you pass the --real-profile option!)
• You can start the process by running the script (Python 2 required):
  python ./script/build-bisect.py
• You can provide a GOOD or BAD version via the --good= and --bad= parameters
• --verbose will show where in the binary search you are
• Script will download and install versions for you, from the GitHub releases page (does not include drafts)
• After downloading + installing, the browser will be launched so you can manually test the feature
• After verifying problem happens/doesn't happen, you can close the browser and answer either y or n to the prompt
• Search will continue until a definitive version is found

[bsclifton]

Script will now find/print all commits unique to the release where issue was found:

[bsclifton]

Feature requests

I'll make sure to create an issue for these 😄 These were items that came from folks during our company brown-bag presentation of the tool

• Can the tool pre-download all (or some) of the releases?
• Can the tool cache the releases? (perhaps it can check SHA in GitHub against local disk, for example)
• For use-cases like rewards, consider using staging environment by default (to prevent creating lots of wallets)
• Tie this into an issue
  • potentially create an issue; prompt user for title/desc and then include the identified release / commits
  • if issue already exists, add a comment after tool is ran to provide the release / commits where problem was identified
• Be able to pass Chromium feature flags in!
• New non-interactive mode flag: Spit out all versions, organized by release channels (ex: --show-versions)
• Be able to use a LOCAL profile also, instead of specifying a ZIP
• Consider tying this into the Chromium bisect tool (for finding bugs in Chromium)

Refactoring notes

Areas where the code could be cleaner

• token is used, but is honestly not needed
• use util methods where possible
• create an npm command for this!

[bsclifton]

Ready for review - needs token for use, but otherwise is documented and ready to go 😄

[mbacchi]

love it!