gdrive: support Google Service accounts

[maxhora]

iterative/PyDrive2#7 should be merged first

Introduces gdrive_service_account_email, gdrive_service_account_user_email and gdrive_service_account_p12_file_path DVC config params to support authorization with Google Service accounts.

Introduced config params are independent from client_id and client_secret config params

Update:

iterative/PyDrive2#11 should be merged first

Introduces gdrive_user_service_account DVC config param disabled by default.

[codecov]

Codecov Report

Merging #3269 into master will increase coverage by 0.00%.
The diff coverage is 89.47%.

@@           Coverage Diff           @@
##           master    #3269   +/-   ##
=======================================
  Coverage   91.30%   91.30%           
=======================================
  Files         141      141           
  Lines        8582     8594   +12     
=======================================
+ Hits         7836     7847   +11     
- Misses        746      747    +1     

Impacted Files	Coverage Δ
dvc/config.py	96.81% <ø> (ø)
dvc/remote/gdrive.py	86.36% <89.47%> (+0.30%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ea981ae...8441ca5. Read the comment docs.

[efiop]

For the record: PyDrive2 1.4.4 has been released.

[efiop]

@Maxris Any progress on this?

[maxhora]

@efiop I believe, it should be evaluated/tested by @shcheklein
Also it is turned out that refresh token is not generated for Service accounts. Since PyDrive supports only old p12 keys to do Service account auth, Travis setup will require additional changes (bundling of p12 cert with non-default password; maybe limiting GDrive API scope only to drive.appdata).

[shcheklein]

@Maxris @efiop we add a lot of code, but I'm not convinced it solves the testing. At least for now. It does not solve the limits issue, unless I'm missing something? What else am I missing?

[maxhora]

@shcheklein yes, it seems it doesn't help to solve limits issues, since Service Account impersonate some another user in domain and the limits are the same.
These changes might be considered as "New functionality" to support login with Service Account. While it is not ready to be automated on Travis, it works perfectly locally.

[maxhora]

These changes might be considered as "New functionality" to support login with Service Account. While it is not ready to be automated on Travis, it works perfectly locally.

Locally user specified path to p12 key and PyDrive automatically retrieves new access_token for Service Account after old one is expired. The difference to the auth for normal Google account is that in addition to an access_token a refresh_token is obtained, and PyDrive automatically retrieves a new access_token by using a refresh_token value.

[shcheklein]

@Maxris not sure I understood the last part about refresh tokens and differences, could you please elaborate a bit?

These changes might be considered as "New functionality" to support login with Service Account. While it is not ready to be automated on Travis, it works perfectly locally.

what would the use case for this? automation? like in CI/CD or something?

While it is not ready to be automated on Travis, it works perfectly locally.

what are we missing here?

[maxhora]

@shcheklein

what would the use case for this? automation? like in CI/CD or something?

Yes, use case is to run tests on Travis with Service Account.

what are we missing here?

p12 key with custom password to do auth with Service Account from Iterative's G Suite and secure way (probably, one another env var) to pass custom password into PyDrive.ServiceAuth()

[shcheklein]

@Maxris do service accounts have their own GDrive space?

could you please elaborate on that difference in the token refresh mechanism, please? I still don't get it.

[maxhora]

@shcheklein

@Maxris do service accounts have their own GDrive space?

No, Service account is a "resource" which has permissions to access other resources.

This PR was tested with following flow https://developers.google.com/identity/protocols/OAuth2ServiceAccount#delegatingauthority . As I understand, Service account receives access to any domain's data on behalf of any domain's user, but the access scope for this Service account can be limited to list of desired access scopes (for example, only https://www.googleapis.com/auth/drive ).

could you please elaborate on that difference in the token refresh mechanism, please? I still don't get it.

For usual Google drive user generated gdrive-user-credentials.json contains both access_token and refresh_token. Just content of gdrive-user-credentials.json is enough to have unlimited in time access to the user's Google Drive.

For Service account generated gdrive-user-credentials.json contains only access_token which is valid only for 1 hour. After access_token is expired, it is supposed that p12 key file is available around to ask Google API to generate new access_token.

btw, in that Google's doc they warn:

Always discourage developers from checking keys into the source code or leaving them in the Downloads directory of their workstation.

[shcheklein]

@Maxris Ok, makes sense. I checked the page and it looks like there is an option to not involve user into this 2LO vs 3LO. Don't you think for automation and testing purposes it would be easier to use 2LO from that doc? I don't see a need to impersonate someone through a service account, at least it does not look superior to the regular service account. Unless I'm missing something again.

[maxhora]

@Maxris Ok, makes sense. I checked the page and it looks like there is an option to not involve user into this 2LO vs 3LO. Don't you think for automation and testing purposes it would be easier to use 2LO from that doc? I don't see a need to impersonate someone through a service account, at least it does not look superior to the regular service account. Unless I'm missing something again.

Good point!
So far, I was not able to find any evidences that in case of 2LO a Service account can access Google Drive storage (Google Drive storage, seems, should belong to the Google Project, but not to specific user, but I was not able to find the technical way to achieve that so far)

Another even more important thing is that PyDrive expects all 3 following params to be available https://github.com/iterative/PyDrive2/blob/master/pydrive2/auth.py#L146
And if, for example, client_user_email (which is used to impersonate real user) is not provided, https://github.com/iterative/PyDrive2/blob/master/pydrive2/auth.py#L416 throws "Insufficient service config in settings". I afraid that PyDrive for the moment doesn't support 2LO since providing of client_user_email always expected.

[maxhora]

Once iterative/PyDrive2#11 is merged, it should be sufficient to set secured GDRIVE_USER_CREDENTIALS_DATA env var in Travis settings for service account and specify service account email only.

[shcheklein]

@shcheklein I've rebased the branch to master already,
Since config doesn't have global constants anymore, I have defined self.GDRIVE_SERVICE_ACCOUNT_EMAIL locally to reference it twice from inside RemoteGDrive. Is it a restriction now to use constants to reference some key from config?

yeah, if you put them in the schema, no need to define upper case constants anymore, just use strings, check similar cases in the gdrive.py

[shcheklein]

please, see my comment - no need to do separate constants

[shcheklein]

looks great overall, check the only comment regarding the constants

[maxhora]

@shcheklein this PR is not yet ready, need to setup Service Account and set Travis env var.

[maxhora]

@shcheklein should be ready to be merged

[shcheklein]

@Maxris travis failed, PTAL.

[maxhora]

@shcheklein thanks! Have fixed value of Travis env var and GDrive tests are passing now. There are still random failures of S3 tests (happen on all DVC Travis builds).

[efiop]

S3 issues fixed, restarted the tests for this PR. Let's see.

[shcheklein]

@efiop I think should be good to merge! :)