Summary
Calling RemoveReviewers when you want to only remove teams elicits a 422, due to how omitempty prevents the golang client from serializing the empty reviewers slice. This field is required on the server, so the golang client needed to send reviewers: [].
Workaround
Work around this by exploiting delete idempotency =) Just pass it a bogus username alongside real team slugs.