Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Google will revoke Gmvault access March 31, 2019 #335

Open
matthewhelmke opened this issue Mar 21, 2019 · 44 comments

Comments

Projects
None yet
@matthewhelmke
Copy link

commented Mar 21, 2019

FYI, I just received this email. I've been using Gmvault for a few years assuming it was abandoned, since there have been no commits since 2016, but thought I would create this issue anyway, just in case...

Hello,

Although you don’t need to take any action, we wanted to let you know that the following third-party apps will no longer be able to access some data in your Google Account, including your Gmail content. This change will go into effect starting March 31, 2019.

Gmvault

We are making this change as part of ongoing efforts to make sure your data is protected and private. These apps haven’t yet complied with our updated data privacy requirements announced on October 8, 2018.

You can always view, manage and remove apps you’ve given access to your account by visiting your Google Account.

Thanks,
The Google Accounts team

@tofurky

This comment has been minimized.

Copy link

commented Mar 21, 2019

nice. remove our ability to archive our emails to help protect it 😂

@joshstrange

This comment has been minimized.

Copy link

commented Mar 21, 2019

I’m hoping this can worked around by replacing an api token somewhere in the source and/or requiring users to register their own app with google. It’s not ideal but it’s easy enough and free for our level of usage I would guess.

I looked a little in the code but I’m on mobile so I’ll have to look later when I’m on my laptop.

@tofurky

This comment has been minimized.

Copy link

commented Mar 21, 2019

see

def generate_permission_url():
"""Generates the URL for authorizing access.
This uses the "OAuth2 for Installed Applications" flow described at
https://developers.google.com/accounts/docs/OAuth2InstalledApp
Args:
client_id: Client ID obtained by registering your app.
scope: scope for access token, e.g. 'https://mail.google.com'
Returns:
A URL that the user should visit in their browser.
"""
params = {}
params['client_id'] = gmvault_utils.get_conf_defaults().get("GoogleOauth2", "gmvault_client_id", "1070918343777-0eecradokiu8i77qfo8e3stbi0mkrtog.apps.googleusercontent.com")
params['redirect_uri'] = gmvault_utils.get_conf_defaults().get("GoogleOauth2", "redirect_uri", 'urn:ietf:wg:oauth:2.0:oob')
params['scope'] = gmvault_utils.get_conf_defaults().get("GoogleOauth2","scope",'https://mail.google.com/')
params['response_type'] = 'code'
account_base_url = gmvault_utils.get_conf_defaults().get("GoogleOauth2", "google_accounts_base_url", 'https://accounts.google.com')
return '%s/%s?%s' % (account_base_url, 'o/oauth2/auth', gmvault_utils.format_url_params(params))

@joshstrange

This comment has been minimized.

Copy link

commented Mar 21, 2019

@tofurky Wow, I was in the file and even scrolled over but didn’t scroll far enough to see the “default values”. I even dug in gmvaul_utils a little after that before I gave up on my phone lol.

That gives me hope that we can work around this, potentially, if I’m scanning that code right, with just a config/defaults file?

@johanatan

This comment has been minimized.

Copy link

commented Mar 21, 2019

I am interested in whatever solution is ultimately reached here.

@fortran77

This comment has been minimized.

Copy link

commented Mar 21, 2019

Would somebody please find a procedure for users to get their own token, and post the steps required? Then each person doesn't have to figure it out separately.

@joshstrange

This comment has been minimized.

Copy link

commented Mar 21, 2019

I started generating new credentials and this was interesting. I'm guessing this the option that gmvault uses:

Option 3: Manual copy/paste

Important: The custom URI scheme and loopback IP address options now provide more reliable, secure, and user-friendly ways to obtain user authorization. This option may be discontinued in the future and should only be used if the choices above are not viable.

I'm still reading but:

Option 2: Loopback IP address (macOS, Linux, Windows desktop)

To receive the authorization code using this URL, your application must be listening on the local web server. That is possible on many, but not all, platforms. However, if your platform supports it, this is the recommended mechanism for obtaining the authorization code.

When your app receives the authorization response, for best usability it should respond by displaying an HTML page that instructs the user to close the browser and return to your app.

This looks like a realistic option for us if option 3 is ever removed. I think it will be but I also think we have more time. Right now we can focus on getting to to work past March 31st.

@gboudreau

This comment has been minimized.

Copy link

commented Mar 21, 2019

The client_id and secret are in $HOME/.gmvault/gmvault_defaults.conf
Swapping them there should do the trick without needing to modify the code.

@joshstrange

This comment has been minimized.

Copy link

commented Mar 21, 2019

Ok this should be as easy as replacing 2 lines in you config (~/.gmvault/gmvault_defaults.conf)

[GoogleOauth2]
scope=https://mail.google.com/
# The URL root for accessing Google Accounts.
google_accounts_base_url=https://accounts.google.com
# Hardcoded dummy redirect URI for non-web apps.
redirect_uri=urn:ietf:wg:oauth:2.0:oob
#identifier and secret in app mode for gmvault
gmvault_client_id=1070918343777-0eecradokiu8i77qfo8e3stbi0mkrtog.apps.googleusercontent.com
gmvault_client_secret=IVkl_pglv5cXzugpmnRNqtT7

Those last 2 lines are what we are going to need to change. I generated 2 new ones but it will only allow for 100 "Sensitive logins" unless my "app" is approved. I don't really want to be the one responsible for this going forward and maybe it's best instead of publishing new keys we write a guide on how to generate your own? It literally takes less than 5 minutes.

@joshstrange

This comment has been minimized.

Copy link

commented Mar 21, 2019

It literally takes less than 5 minutes.

Note: I already had an account setup, I just had to create a new "Project" so I don't know how long it would take for someone who had never used the google API console or if they will have to setup billing info (My account does have billing on file).

@tofurky

This comment has been minimized.

Copy link

commented Mar 21, 2019

falling back to generating an app password and using that is an option. i think that should bypass any of the oauth stuff.
example:

$ gmvault sync --store-passwd user@gmail.com
<enter app password>
^C
$ gmvault sync -p user@gmail.com
...

see https://support.google.com/accounts/answer/185833?hl=en for details on app passwords.
not ideal, though.

@joshstrange

This comment has been minimized.

Copy link

commented Mar 21, 2019

@tofurky Yeah, I saw that mentioned in a different issue. That might be the easier long-term solution.

I wish github made it easy to see if any of the 200+ forks on this saw any real development. I'd switch to a fork of this if I knew there was at least some housekeeping going on.

@brechmos

This comment has been minimized.

Copy link

commented Mar 21, 2019

It sounds like there is some direction on fixes for this.

When there is an acceptable solution could someone post in the Wiki and then post back here of the "accepted" solution?

(Glad there are lots thinking and working on this!)

[Edit, Hah, for some reason I was able to post in the future...]
image

@gboudreau

This comment has been minimized.

Copy link

commented Mar 21, 2019

Procedure to get yourself a new client ID & secret:

  • Start here: https://console.developers.google.com
  • Accept conditions, if you never used the Google API Console (after reading all the terms and conditions, evidently)
  • Create a new project (at the top); of choose one you already have, if that makes sense
  • Go to https://console.developers.google.com/apis ; click + ENABLED APIS AND SERVICES at the top
  • Search for Gmail; select it; click Enable
  • Go to https://console.developers.google.com/apis/credentials; choose OAuth Consent Screen tab
  • Fill name (enter anything), click Add scope
  • Select the checkbox on the https://mail.google.com/ line; click Add
  • Click Save
  • Click Create Credentials; choose OAuth client ID
  • Application Type: Other; give it a name (anything; I suggest gmvault, since this credential will be used by gmvault)
  • Click Create
  • You will get a client ID and secret. Those two values needs to replace the existing ones in $HOME/.gmvault/gmvault_defaults.conf : gmvault_client_id=... and gmvault_client_secret=...
  • If you are running gmvault 1.9.1, make sure conf_version=1.9 in that same conf file, and NOT conf_version=1.9.1, otherwise, gmvault will overwrite it every time it runs. (This is a bug in 1.9.1, fixed in 1.9.2; so you do not need to change conf_version if you are running 1.9.2-beta-1 or higher.)
  • Finally, obtain a new OAuth token using the following command:
    gmvault check --renew-oauth2-tok your_email_address@gmail.com
@Pectojin

This comment has been minimized.

Copy link

commented Mar 21, 2019

FYI, it seems google is pretty sticky with handing out the API access, so it may take some time to get this working.
Screenshot 2019-03-22 at 00 23 00
Screenshot 2019-03-22 at 00 32 13

@brechmos

This comment has been minimized.

Copy link

commented Mar 21, 2019

There is a "Submit for verification" beside the "save" button. Not sure if that speeds up the process at all...

@DavoOZ

This comment has been minimized.

Copy link

commented Mar 22, 2019

Questions:

  1. Does anyone have any idea of how often one's access credentials would need to be updated (using the manual method described above)?
    https://github.com/gaubert/gmvault/issues/335#issuecomment-475437988

  2. My reading of the new Google API conditions suggests that the actual app may need to be accredited for access, as well as individual users - are I correct in this?

@chbug

This comment has been minimized.

Copy link

commented Mar 22, 2019

I’m hoping this can worked around by replacing an api token somewhere in the source and/or requiring users to register their own app with google. It’s not ideal but it’s easy enough and free for our level of usage I would guess.

I looked a little in the code but I’m on mobile so I’ll have to look later when I’m on my laptop.

I'm confused: why is there a need to work around it instead of getting the current key reviewed once and for all? gmvault doesn't move user data to a cloud storage or anything where the data of multiple users gets centralized, so it should be straightforward, no?

@daneroo

This comment has been minimized.

Copy link

commented Mar 22, 2019

I successfully created App credentials with @gboudreau 's great instructions above.
Thanks for taking the time to write that up, (and being so thorough!)

@dan20047

This comment has been minimized.

Copy link

commented Mar 22, 2019

Daneroo - did you get confirmation that you won't have access issues or will you have to wait until after March 31st?

Thanks

@daneroo

This comment has been minimized.

Copy link

commented Mar 22, 2019

Not sure @dan20047 ,

  • It's working right now.
  • I'm don't know what 100 "Sensitive logins" means
  • I did not submit the OAuth consent screen for verification, since it's already working, we'll see how it goes...
@fortran77

This comment has been minimized.

Copy link

commented Mar 23, 2019

@joshstrange and others interested, here is a list of forks with a last commit date in 2017 and later. Now you can search for ones with the most current development. Anything useful should preferably be merged back here if possible.

2019:

https://github.com/Anmol-Singh-Jaggi/gmvault
https://github.com/mbarczak/gmvault

2018:

https://github.com/fossabot/gmvault
https://github.com/hengy2017/gmvault

2017:

https://github.com/andriusadamonis/gmvault
https://github.com/asez73/gmvault
https://github.com/csirmaz/gmvault
https://github.com/eliask/gmvault

@dygordon

This comment has been minimized.

Copy link

commented Mar 24, 2019

Answers re: the consent screen/verification here: https://support.google.com/cloud/answer/7454865

In brief:

  1. you can only mark an app "internal" if you have G Suite and you are creating it within your organization, in which case it is internal by default.

  2. 100 sensitive logins is defined as 100 new users, so this limit shouldn't matter anyway if it's just you. I think. Not sure if it will see each use of the OAuth credentials by gmvault as a new user or not. But some rclone forums setting up G Drive scopes suggest no.

EDIT:

Well, now that I actually try the login, I get the same "Sign in with Google temporarily disabled" screen some have reported despite not being anywhere close to the 100 new user cap.

Looking at the URL and then the file, looks like gmvault is reverting the changes to $HOME/.gmvault/gmvault_defaults.conf and putting its own app keys back in. I guess this is the known issue at #245 and #273 and some of you may be running the beta version as I am on 1.9.1.

@gboudreau @daneroo you may want to check if your changes to the conf file actually did stick or you perhaps just reauthed using the standard gmvault keys (client_id=1070918343777-0eecradokiu8i77qfo8e3stbi0mkrtog.apps.googleusercontent.com).

@seanlane

This comment has been minimized.

Copy link

commented Mar 24, 2019

In regards to the issue with $HOME/.gmvault/gmvault_defaults.conf being overwritten, one quick patch is to change the line conf_version=1.9.1 to conf_version=1.9, per issue #273. I wasn't able to find any documentation on the differences between config files of the two versions, but some initial testing hasn't found problems. Feel free to correct me if this would actually cause any issues though.

@gboudreau

This comment has been minimized.

Copy link

commented Mar 24, 2019

@dygordon Yes, my changes did stick. But while I am using version 1.9.1 of gmvault, my .conf shows 1.9 (because it was created when I was running 1.9 I would guess), so like @seanlane pointed out, that seems to prevent the overwriting problem from happening. I'll add that to the guide I posted above.

@yesrod

This comment has been minimized.

Copy link

commented Mar 26, 2019

I overlooked a critical step at first; I needed to move or delete the old oauth2 token and repeat the sign-in process.

I'm using the docker image from https://hub.docker.com/r/aubertg/gmvault-docker so for me the process was:

  1. Follow the instructions in gboudreau's comment to generate and install new credentials
  2. cd to the directory mounted as /data in the container
  3. move email@gmail.com.oauth2 to email@gmail.com.oauth2.old
  4. attach to the existing container with docker exec -it gmvault-docker /bin/bash
  5. run su -c 'gmvault sync -d /data ${GMVAULT_EMAIL_ADDRESS}' gmvault and follow the instructions
@jaskerr

This comment has been minimized.

Copy link

commented Mar 26, 2019

Followed @gboudreau instructions for new client ID and secret.

Google's Oauth2 endpoint: https://accounts.google.com/o/oauth2/token returned Error: HTTP Error 401: Unauthorized.

Had to remove the old Oauth2 token and generate a new one (similar to @yesrod).

Might want to update instructions to include the generation of new tokens.

@benreic

This comment has been minimized.

Copy link

commented Mar 27, 2019

Just used @gboudreau instructions and worked perfectly. They've been updated with the feedback from the successive comments so the instructions are comprehensive.

@tidhub

This comment has been minimized.

Copy link

commented Mar 28, 2019

Thank you @gboudreau, could you please clarify what the name should be:

  • Fill name, click Add scope <-- is name = Gmvault, or can I give it a name?
  • Select the checkbox on the https://mail.google.com/ line; click Add
  • Click Save
  • Click Create Credentials; choose OAuth client ID
  • Application Type: Other; give it a name <-- is this name the same as the one above?
@gboudreau

This comment has been minimized.

Copy link

commented Mar 28, 2019

@tidhub Both names can be anything you'd like.

@tidhub

This comment has been minimized.

Copy link

commented Mar 28, 2019

Great thank you @gboudreau

@jezzabart

This comment has been minimized.

Copy link

commented Apr 1, 2019

Many thanks to @gboudreau for your very clear instructions. I have no idea what they do, but they worked perfectly.

@dalin-

This comment has been minimized.

Copy link

commented Apr 5, 2019

I ran @gboudreau 's instructions:
#335 (comment)

But when I ran gmvault I got the error:

Error: Problems when trying to connect to Google oauth2 endpoint: https://accounts.google.com/o/oauth2/token.

Error: HTTP Error 400: Bad Request.

=== Exception traceback ===
Traceback (most recent call last):
  File "/Users/gaubert/Dev/projects/gmvault-1.8.2/build/gmvault/out00-PYZ.pyz/gmv.gmv_cmd", line 743, in run
  File "/Users/gaubert/Dev/projects/gmvault-1.8.2/build/gmvault/out00-PYZ.pyz/gmv.credential_utils", line 235, in get_credential
  File "/Users/gaubert/Dev/projects/gmvault-1.8.2/build/gmvault/out00-PYZ.pyz/gmv.credential_utils", line 411, in get_oauth2_credential
  File "/Users/gaubert/Dev/projects/gmvault-1.8.2/build/gmvault/out00-PYZ.pyz/gmv.credential_utils", line 267, in _get_oauth2_acc_tok_from_ref_tok
HTTPError: HTTP Error 400: Bad Request

The solution was related to
@yesrod's instructions
#335 (comment)
but a bit different since I'm not using Docker:

  1. cd ~/.gmvault/
  2. mv <my-email>.oath2 <my-email>.oath2.bak
  3. Re-run Gmvault, follow the instructions about authenticating.
@medmunds

This comment has been minimized.

Copy link

commented Apr 6, 2019

Note that if you need to back up both GSuite accounts and actual @gmail.com addresses, you must create your gmvault project using your @gmail.com Google account, not a GSuite account. (In the first step of @gboudreau's instructions, check the Google account switcher at top right of https://console.developers.google.com/.)

It seems that, for unpublished client ids:

  • A client id owned by an @gmail.com account can be authorized to back up that Gmail account, and any other GSuite account
  • A client id owned by a GSuite account cannot be authorized to back up an @gmail.com account: you'll get the error @Pectojin saw when you try to issue a token for your Gmail address: "Sign in with Google temporarily disabled for this app. This app has not been verified yet by Google in order to use Google Sign In."

BTW, after you've updated your gmdefaults.conf, an easy way to obtain new auth tokens is gmvault check --renew-oauth2-tok $YOUR_EMAIL_ADDRESS

@limes007

This comment has been minimized.

Copy link

commented Apr 7, 2019

The procedure of @gboudreau has worked, together with the "renew" command of @medmunds. Thank you very much.
I wonder if or how long my Google project will be free of charge (and maybe how long this will work at all). As far as I understand, its a free trail only. Does anyone know for sure?

@cornasdf

This comment has been minimized.

Copy link

commented Apr 9, 2019

Hi all,
Thanks for the write ups, i am back online with this.

FYI, when at the 'Create OAuth client ID' step, if you choose web application, you will end up with the error below when trying to do the web page redirect/auth flow. I was successful when I chose 'Other' as the application type at https://console.developers.google.com/apis/credentials/oauthclient

Error: redirect_uri_mismatch

The redirect URI in the request, urn:ietf:wg:oauth:2.0:oob, can only be used by a Client ID for native application. It is not allowed for the WEB client type. You can create a Client ID for native application at https://console.developers.google.com/apis/credentials/oauthclient

@dsl101

This comment has been minimized.

Copy link

commented Apr 12, 2019

I'm using 1.9.2-beta-1 on windows, and in the end I had to make gmvault_defaults.conf unwriteable (Properties→Security→Edit, Select User, click Write→Deny) to stop gmvault putting it's own keys back in. Once I'd done that, I was able to authenticate.

@lexelby

This comment has been minimized.

Copy link

commented Apr 12, 2019

This looks like a workable replacement for gmvault, including backing up based on a gmail search: https://github.com/jay0lee/got-your-back/

They seem to have run into the same authentication issue, and they've partially automated and fully streamlined the process of setting up your google project to get auth working.

@mhsmith

This comment has been minimized.

Copy link

commented Apr 16, 2019

Like @dsl101 above, I'm also using 1.9.2-beta-1 on Windows. However, making gmvault_defaults.conf unwriteable did not work for me. It did stop the file from being overwritten, but gmvault still considered it to be out of date and ignored it, so my client ID and secret weren't used.

Instead, I simply left conf_version at its previous value of 1.9.2-beta-1. Aside from that, I was able to follow the instructions without change.

@dsl101

This comment has been minimized.

Copy link

commented Apr 16, 2019

Just to clarify, I didn't change conf_version—also left at 1.9.2-beta-1.

@gitasong

This comment has been minimized.

Copy link

commented May 5, 2019

I'm going through @gboudreau's instructions. When I try to save the config file with my new client ID and secret, I get Unable to save file: Permission denied '/.gmvault'. I've never modified a Unix executable before and have no idea how to properly do so. Can anyone advise?

@gitasong

This comment has been minimized.

Copy link

commented May 5, 2019

Found it. For Mac users, the config file is located at /Users/yourname/.gmvault/gmvault_defaults.conf. It'll give you a No Storage DB in /Users/nicolefreed/gmvault-db. Create it. message if you try to run the OAuth update command gmvault check --renew-oauth2-tok your_email_address@gmail.com, but just ignore it and start backing up e-mails with ./gmvault sync myuser@gmail.com. Gmvault should work fine at that point.

@mfonville

This comment has been minimized.

Copy link

commented May 12, 2019

Would anyone please fork the project and put out a new (maybe renamed) release that incorporates setting your own gmvault_client_id and gmvault_client_secret during the setup process? and also to directly tackle #273 :-)

@samadore

This comment has been minimized.

Copy link

commented May 12, 2019

Thank @gboudreau for much needed step-by-step, but when I run last command to get new token it returns "command not found". Clearly, I'm doing something wrong. Any idea what?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.