Piotr Kaminski edited this page Dec 15, 2016 · 54 revisions
Clone this wiki locally

Why do you ask for write access to my repos!?

Sorry about that—we swear we'll never muck around with your code. We do need read access to your repo to compute all those lovely diffs, though, and unfortunately GitHub's permission scopes don't distinguish between read and read/write access. Also, we post pull request comments on your behalf, which GitHub considers as "writing to the repo", and give you the option to merge a pull request once the review is complete.

Do you keep my private source code secure?

You bet! First, we never store your source code on our servers—the client fetches it directly from GitHub into the browser over a secure HTTPS connection. The only data we store on our servers are comments (including drafts), pull request metadata (filenames, commit and file SHAs, etc.), basic account data (id, username, email, etc.), and the access token you authorized. Access permissions are inherited from GitHub and rechecked every half hour and all data is always transmitted across secure connections. The access token is kept private at all times and used only to access GitHub on your behalf. We will never use the token to peek at your repository without your explicit written authorization (e.g., to debug a thorny issue you're seeing).

If you need more details about our security architecture or have any other concerns we can address, please get in touch.

What do you do with each of the authorizations you ask for?

Here's how we use the authorizations you grant us:

  • public_repo: We use it to post pull request comments on your behalf, merge pull requests and delete branches when you ask us to, and pin revision commits in the repo so they don't disappear if you rebase.
  • repo: We use it to list your private repos and organizations, read source code (only in the browser), read pull request metadata and comments, post pull request comments on your behalf, merge pull requests and delete branches when you ask us to, and pin revision commits in the repo so they don't disappear if you rebase.
  • admin:repo_hook: We use it to add or remove an event hook when connecting or disconnecting a repo, so that Reviewable is notified of relevant events (e.g., pull request updates).
  • admin:org_hook: We use it to add or remove an event hook when subscribing to or unsubscribing from a plan, so that Reviewable is notified of relevant events (e.g., organization membership changes).
  • read:org: We use it to get a list of the organizations that you're a member of, even if your membership is private. It also allows us to get a list of all the repos you're granted access to as an organization member.
  • user:email: We may use it to contact you directly in exceptional circumstances (e.g., something wrong with your account) or if you requested to be notified (e.g., of an upcoming feature). All of Reviewable's day-to-day email is sent via GitHub and can be controlled via its notification settings.

GitHub's authorization scopes are pretty coarse, and cannot be constrained to specific organizations—sorry! GitHub also has some docs on authorization scopes you can look at. You can revoke all of these permissions at any time via your authorized applications settings page on GitHub. If you decide to do that, we'd appreciate it if you would disconnect your repos first, though.

How do subscriptions work?

A subscription is necessary to create reviews in private organizational repos. (Public repos and private personal repos don't need a subscription.) You can change or cancel a subscription at any time with immediate effect, but we will not refund or prorate fees (either up or down). If you cancel, previously created reviews will continue to be accessible and synchronized with GitHub but no new reviews will be created. Every subscription gets a 30 day free trial with no credit card required. We use Stripe to store your credit card details and process payments, so your financial information is secure.

Each plan has a maximum number of monthly users. Rather than forcing you to maintain yet another user list, we count each distinct PR author as a user at the time a review is created and linked to the PR. This means that once a review has been created, any number of people can view it and participate. If a new PR author causes a subscription to exceed its user cap, we'll notify you immediately. If the plan has the flexible overage feature, we'll allow any number of extra users as long as you didn't also exceed your cap in the previous billing cycle. This should give you time to upgrade your subscription without disruption—or, if the overage is temporary and you know you'll be back under the cap next month, you can just ignore it. If your plan doesn't have the flexible overage feature or you did exceed the cap last month, we won't create the review until you either upgrade your subscription or the user count resets at your next billing date. (We'll continue updating all previously created reviews, though, and keep creating reviews for PR authors that were already counted this month.)

By default, a subscription is organization-wide and allows anybody to create reviews that count towards it, either by connecting a repo or opening a review from the Reviews page. If your organization is large and you're worried that developers on other teams will cause your subscription to exceed its cap by abusing this privilege, you can reserve this ability for yourself in the subscription panel. If you do that, only you will be able to connect repos and create reviews—if others want to do so as well, they'll need to start their own subscription. Note that even if you exercise this option and connect just one repo, it won't count as "1 user"—we'll still be counting the number of distinct PR authors towards your cap.

If I connect a repo, do I have to use Reviewable for all the PRs?

No, not at all. While every pull request will automatically get a button linking it to a Reviewable review, you can just ignore it and conduct the review in GitHub (e.g., if the change is very small). Reviewable will close the review when you close the PR. Note, however, that if it's a private organizational repo covered by a subscription, every PR's author will count against your user cap whether you use the review or not.

Can I mix using GitHub and Reviewable on a PR?

Yes, but the experience won't be optimal. While comments posted in one will also appear in the other, line comments will turn into top-level comments due to the systems' irreconcilable approaches to line numbering. Also, Reviewable expects that all participants are using its system for tracking unreviewed files and unresolved comments, so if some people are using GitHub those counters probably won't be right.

How does connecting my PRs in all repos work?

If you toggle on "My PRs in any public repo" (or "My PRs in any private repo"), Reviewable will regularly scan your public (or private) PRs and create reviews for them. This can be useful to connect reviews to all your open source contributions, or if only a subset of the users of an organization's private repo want to use Reviewable.

One big caveat about connecting your PRs in private repos: this will only work if the relevant repo has an active subscription at the time the PR is created. If no subscription is found, you'll get a warning email, the review won't be created, and Reviewable will not try again. If a subscription is added later you'll need to create the review manually by clicking on the PR in your dashboard.

What's the difference between connecting a repo and connecting all my own PRs, or not connecting anything?

Besides the obvious—connecting a repo links all PRs in that repo, and connecting your PRs links all PRs you author across all repos—there are some technical differences as well. If you connect all your own PRs or do ad-hoc reviews then Reviewable doesn't get write access to the repo, so:

  • New commits, GitHub comments, labels, and the PR assignee aren't synced into the review immediately when they're updated, but only after somebody loads the review to look at it. This is especially noticeable in the review dashboard, since it will show stale data for those reviews. Comments posted via Reviewable are still propagated immediately, though.
  • Assignee and label directives in GitHub and emailed comments won't be applied until somebody loads the review to look at it.
  • Review status checks are not posted to the PR, since Reviewable isn't subscribed to repo events and couldn't make timely updates.
  • We may not be able to pin revision commits, so if you use git rebase and git push --force some of them may get garbage collected and will no longer be accessible in the review. They'll usually get pinned when the reviewer (with push authority) accesses the review.
  • Reviewable may not be able to reliably detect and apply your branch protection settings.

Overall, these are minor differences, but if you have that option it's better to connect a repo directly.

Does Reviewable support a git rebase workflow?

Yes! You can rebase and force-push, and Reviewable will correctly diff against past revisions even if their commits are no longer in the branch history. To make sure that those orphaned commits don't eventually get garbage-collected, Reviewable creates git refs for each commit that was snapshotted as a revision in the review. (This works best if you connect a repo, see the question above.)

Can I just review a commit directly?

No. Because Reviewable doesn't actually clone your repo it relies on a bunch of GitHub APIs that are only available for pull requests, so it can't operate directly on raw commits.

Why don't Reviewable comments show up inline in GitHub and vice-versa?

To be blunt: GitHub's model for comment placement is poor, and conforming to it would invalidate many of Reviewable's best features.

Specifically, GitHub has two types of inline comments: pull request (PR) comments and commit comments. PR comments are assigned to lines based on the raw git diffs between two commits. Reviewable doesn't use these diffs, making mapping locations hard (see this issue), and can place comments on lines that don't appear in the diffs. Also, comments placed on negative delta lines in GitHub don't actually record sufficient information to figure out what line they're really on! Commit comments share most of those limitations, and also disappear from the PR if the branch is rebased. Finally, it's impossible to post either kind of comment without triggering a notification email for each and every comment, which would defeat Reviewable's comment batching feature.

Can I reply to Reviewable comments by email?

Yes, Reviewable will do its best to parse incoming messages and split them up into the appropriate comment threads. This only works well if you leave all the separators and comment links in place, though, so be sure not to mangle the quoted message too much when replying. Quoted chunks will be shown in Reviewable if they're intermixed with your reply, and elided otherwise (whether top- or bottom-quoted). Any parts of the message that can't be conclusively tied to a specific discussion will show up in a top-level comment instead.

You can also reply with the single word acknowledge (or ack) to simulate clicking the acknowledge button in Reviewable, or include an updated disposition in your reply (discussing, satisfied, or blocking, on a line by itself) -- these are all case-insensitive. Reviewable's label and assignee directives (+label, -label, +@username, -@username) will also work, but there's no autocomplete so be careful to spell things correctly! You'll get a warning email for misspelled usernames, but not for labels/milestones since false positives are too common there.

Note that if the repo is not connected then directives won't take effect until somebody visits the review in Reviewable, and that editing directives in a previously sent message won't work either.

Can I customize how the source code is displayed?

Sure! The following options are available:

  • You can resize the window to get either a two column (side-by-side) or single column (unified) diff.
  • You can adjust the margin used to wrap the code by dragging the small triangle near the top right of every file. The wrapped portion of a line will be italicized.
  • You can set a custom code font in your account settings. The font must be monospace, already be installed on your machine, and the name must match precisely. The font size should be specified in CSS format, e.g. 10pt, 1.2em, or 90%.
  • You can set a custom syntax highlighting stylesheet in your account settings. We use highlight.js, so you can either pick one of their predesigned styles from the styles directory or create your own. For example, to use the GitHub style, set the URL to https://cdnjs.cloudflare.com/ajax/libs/highlight.js/8.2/styles/github.min.css. Make sure the stylesheet is served as text/css—you might need to use https://rawgit.com. We also have a few extra styles used to highlight tabs, trailing whitespace, and lack of final newline that you can customize:
.tab, .no-newline, .trailing {
  color: #D95C5C;

.tab:before {
  content: "\00bb\00a0\00a0\00a0";

.no-newline:before {
  content: "\00a0\20e0\23ce";

If you'd like diff lines with edits to be highlighted in two-column mode (by default, only the deltas are highlighted), you can add the styles from this gist, or just set your account's custom stylesheet URL to https://rawgit.com/pkaminski/2922da3d58f76a8ed7bf/raw/highlight_lines_in_two_columns.css.

If your favorite language is not one of the 120+ that have syntax highlighting, you can open an issue with highlight.js or even contribute a language-specific module to the project yourself.

Can I jump directly from the review to the right place in my favorite editor?

You can, as long as your favorite editor supports a custom URL scheme for linking to files. In your account settings you can set a custom template for the line links that appear in the upper-right corner of each discussion. Some sample templates that assume /directory/with/repos is the absolute path to the directory where your git repos live, and each repo directory is named the same as its repo:

  • GitHub: https://github.com/{{owner}}/{{repo}}/blob/{{viewSha}}/{{path}}#L{{viewLine}} is the built-in default that opens GitHub on the exact version of the file you're looking at with the line highlighted.
  • TextMate: txmt://open?url=file:///directory/with/repos/{{repo}}/{{path}}&line={{line}}.
  • Sublime Text: subl://open?url=file:///directory/with/repos/{{repo}}/{{path}}&line={{line}} if you install an appropriate URL handler.
  • Emacs: emacs://open?url=file:///directory/with/repos/{{repo}}/{{path}}&line={{line}} if you install the Emacs URL Handler for OS X. If you know of solutions for Linux or Windows please let us know!
  • Atom: atm://open?url=file:///directory/with/repos/{{repo}}/{{path}}&line={{line}} if you install Atom Handler on OS X.
  • Eclipse: openineclipse://open?url=file:///directory/with/repos/{{repo}}/{{path}}&line={{line}} if you install the OpenInEclipse script on OS X or follow these instructions on Linux. If you know of solutions for Windows please let us know!
  • IntelliJ IDEA, Android Studio, PyCharm, PHPStorm: idea://open?file=/directory/with/repos/{{repo}}/{{path}}&line={{line}} (or replace idea: with pycharm:, phpstorm:—check your docs for the scheme specific to your editor flavor). Should work on OS X, not sure about other platforms.

Your custom URL template can use the following variables:

  • {{owner}}: the repo owner (or organization) username
  • {{repo}}: the repo name
  • {{pr}}: the pull request number
  • {{path}}: the full path to the file (does not start with /)
  • {{sha}}: the commit sha of the latest revision in this review
  • {{line}}: the line number in the latest revision of the file that best represents the original context in which the comment was made
  • {{viewSha}}: the commit sha of the revision in which the discussion is currently being displayed
  • {{viewLine}}: the line number in the revision in which the discussion is currently being displayed; this is the number that's displayed in the top-right corner of the discussion

Typically you'll want to edit the latest version of the file and that's what you'll have loaded in the editor, so you probably want to use {{line}} rather than {{viewLine}}. If you need any other variables for your template, please let us know.

Can I customize the key bindings?

Sure can. Grab a copy of the default bindings file and put it somewhere accessible on the web (like in a gist). Edit to taste: all available commands are listed in the file (though not all are bound by default), and you can use any key combos supported by Mousetrap. If your favorite command isn't listed, please open an issue so we can add it.

When you're done, point Reviewable to your custom bindings file through the corresponding field in your account settings drop-down. (If you're using a gist, make sure to get the "raw" URL, and consider removing the commit SHA from the path to always point to the latest version.) Your bindings will be loaded and applied right away, and the cheatsheet updated to reflect them; be sure to check the console if they don't appear to work, as any errors will be printed there. To load any updates to your file, either reload the page or make a no-op edit to the URL field.

How do I tell Reviewable that a file is generated and should not be reviewed?

Reviewable will automatically detect many kinds of generated files and hide the diff by default. If you'd like to add a new common pattern (based either on the filename or the contents of the file), please let us know. If your generated files are specific to your project, you have two options:

  1. You can tweak your build to insert the exact text GENERATED FILE DO NOT EDIT as a comment on a line by itself into the generated file. (There can be up to 4 additional characters on the line before and after the marker text. We have this restriction so that Reviewable doesn't accidentally consider your build script as generated itself!)

  2. You can use .gitattributes files checked into your repo to control diffing. These files hold glob patterns like .gitignore, but you can also add space-separated attributes on each line. Reviewable recognized the attributes -diff (suppress diffing), diff (force diffing), and diff=<language> (force diffing with the given language highlighting, or suppress diffing if language not supported).

Can I customize what makes a review complete?

Yup, you can—quite extensively at that. If you're a repo admin you can click on the small gear icon next to the current state of any review (in the "checks" dropdown) and you'll be able to write a code snippet to implement your custom rule.

You'll see code that replicates Reviewable's default algorithm and you can select among a few other examples from a dropdown menu (including one that enforces "explicit LGTM" semantics). Your code will be given the current state of the review and return an object that looks like this:

  completed: false,                      // whether review is complete
  description: '2 of 5 files reviewed',  // full description of current status
  shortDescription: '2 files left',      // optional short description (to fit into 50 chars)
  // optionally, users whose action is needed to advance the review; will be appended to description
  pendingReviewers: [ {username: 'pkaminski'} ],
  // optionally, per-file per-revision flags to override reviewed status (defaults to reviewers.length >= 1)
  // (it's OK to just augment the input files structured with reviewed flags and return the whole thing here)
  files: [
    {path: 'LICENSE', revisions: [key: 'r1', reviewed: true]}

The review variable is predefined, and holds a bunch of information about the state of the review. All timestamps are in milliseconds since the epoch, and all lists are ordered chronologically (when meaningful). It looks like this:

  summary: {
    lastRevision: 'r1',             // The key of the last revision
    numUnresolvedDiscussions: 1,    // The number of unresolved discussions
    numFiles: 1,                    // Total number of active files in the review
    numUnreviewedFiles: 1,          // Number of files not reviewed by anyone at latest revision
    numFilesReviewedByAtLeast: [1]  // Number of files reviewed by at least N people (as index)
      // e.g., numFilesReviewedByAtLeast[2] is the number of file reviewed by at least 2 people
  pullRequest: {
    author: {username: 'pkaminski'},
    assignees: [{username: 'pkaminski-test'}, {username: 'mdevs5531'}],
    number: 44,
    numCommits: 3,
    target: {owner: 'pkaminski', repo: 'sample', branch: 'work'},
    source: {owner: 'pkaminski', repo: 'sample', branch: 'pkaminski-patch-9'}
  revisions: [  // List of all revisions, in chronological order
      key: 'r1',
      snapshotTimestamp: 1436825047000,  // When this revision was snapshotted (missing if provisional)
      commitSha: '435ae39a89e6992c9ed72fd154bc3c45290d8a97',
      baseCommitSha: '3cd017d236fe9174ab22b4a80fefb323dbefb50f',
      commits: [
        {sha: '435ae39a89e6992c9ed72fd154bc3c45290d8a97', timestamp: 1436825047000}
  labels: [  // List of all labels applied to the pull request
    'Comments only in Reviewable'
  sentiments: [  // List of sentiments (currently just emojis) extracted from comments
    {username: 'pkaminski', emojis: ['lgtm', 'shipit'], timestamp: 1449045103897}
  discussions: [  // List of the discussions in the review (metadata only)
      numMessages: 1,
      resolved: false,  // Whether the overall discussion is resolved
      participants: [
          username: 'pkaminski',
          disposition: 'discussing',  // Participant's current disposition
          resolved: true  // False if this participant is blocking resolution
      target: {  // Target file location; the top-level discussion doesn't have a target
        file: 'LICENSE', revision: 'r1', base: false, line: 4
  files: [  // List of files in the review
      path: 'LICENSE',
      revisions: [  // List of the revisions where this file was changed
          key: 'r1',
          reviewers: [  // List of users who marked file as reviewed at this revision
            {username: 'somebody'}

If you find that you'd like more data please ask and we'll see what we can do.

Your code will be executed on AWS Lambda in NodeJS 4.3.x with Lodash 3.x bound to _, and isolated on a per-repo basis to ensure safety and security.

I have a question you didn't answer.

Please drop by the chatroom or email us and we'll get it answered right away!