I created a client and set the site to my Authorization host, then spent an hour trying to figure out how to change the site on the AccessToken object. DUH!
Set the client's site to the API server's hostname. If the authorization comes from a different server use an absolute URL for your authorize_url.
I do wonder about your names: you create a Client, then use that to get a Token. But in this case the Token has #get and #post etc. methods. It might be clearer if you used the Client to get an AuthorizedClient, which can access protected resources.
Anyway, perhaps you could add some examples to the README showing :authorize_url on one host and :site somewhere else.