Change Google Drive backend to Google Drive API + OAuth2

[richieforeman]

Currently, it would appear that Duplicati uses the Google DocsList v3 API + Client Login for the Google Docs Backend. These interfaces have been deprecated since April 2012 will go dark in April 2015.

The Google Drive backend should be updated to use the Google Drive API and OAuth2 for authentication.

The Google .NET Client Library can be found here: https://github.com/peleyal/google-api-dotnet-client

[miriampl]

Will it work ?

[richieforeman]

Yeah, it might be advisable that each installation has their own Google Developer project for quota purposes, but yes -- it will work.

[TechGuy2000]

I agree that the Google Drive backend should be updated to use the Google Drive API with OAuth2.

I would also like the scope of the authorization to be limited to the Application Data Folder (see: https://developers.google.com/drive/web/scopes and https://developers.google.com/drive/web/appdata), or at least to allow the user to select that option during the authorization process. My Google Drive setup involves terabytes of data of various kinds, including backups. I do not want to authorize any apps to have full access to the contents of my Google Drive. The Application Data scope seems appropriate because Duplicati 2.0's backups are not usable by applications other than Duplicati 2.0.

Another alternative would be the scope "https://www.googleapis.com/auth/drive.file", but it's not clear to me how that scope limits the ability of the app to open files not created by the app.

[JoeSchmuck]

Yes, it did happen a few days ago where Google Drive Authentication now fails and they have went to the new V3 libraries and OAuth2.0.

[rryk]

Is there a way to star an issue on GitHub? My backup started failing since Monday (25.05.2015). Also +1 to what TechGuy2000 said about limiting authorization to Application Data Folder.

[caldude321]

It appears google has undone their removal of basic authentication. Duplicati is now working normally again; since 6/11.

[rryk]

Indeed it seems to work for me. Although I wouldn't count on Google keeping it working for too long. It's a deprecated API after all. If only I had time I'd gladly contribute myself and implement a new backend. Btw, I would be willing to give a small donation (10-20 EUR) to whoever would add such support.

[kenkendk]

I am working on the Google Drive implementation for OAuth, and making some good progress.
I think the correct scope would be "https://www.googleapis.com/auth/drive.file", that way files processed by Duplicati will not interfere with other documents.

However, for those currently using the Google Docs backend to store data, they will need to use the full "https://www.googleapis.com/auth/drive".

From what I can read about the AppData folder, it prevents users from seeing the contents, which would make it hard to recover the files if Duplicati's OAuth connector goes broken at some point. Also, files stored with AppData will be delete if the user removes the app (de-authorizes it?), which might give a nasty surprise.

[rryk]

Considering that my current backup is over 30 days old I'd be willing to
drop it and create a new one instead. So drive.file is just fine.

Perhaps for best user experience it should be possible to specify the type
of access in the extended settings. The default may be drive.file.

On Mon, Jun 22, 2015, 11:22 Kenneth Skovhede notifications@github.com
wrote:

I am working on the Google Drive implementation for OAuth, and making some
good progress.
I think the correct scope would be "
https://www.googleapis.com/auth/drive.file", that way files processed by
Duplicati will not interfere with other documents.

However, for those currently using the Google Docs backend to store data,
they will need to use the full "https://www.googleapis.com/auth/drive".

From what I can read about the AppData folder, it prevents users from
seeing the contents, which would make it hard to recover the files if
Duplicati's OAuth connector goes broken at some point. Also, files stored
with AppData will be delete if the user removes the app (de-authorizes
it?), which might give a nasty surprise.

—
Reply to this email directly or view it on GitHub
#1347 (comment)
.

[kenkendk]

I think I can just add another login button, and it should work for both access levels.

The implementation is now complete, I am running tests for the rest of the day, and if all works, it should be comitted by tonight.

[MalachiC]

Not trying to sound ungrateful, love your work, SUPER appreciate it.... but..... any chance of pushing out a preview_2.0.0.87 (or similar) with this fix in it? This is a show stopper on a lot of backups for me personally.

[rryk]

What MalachiC said.

[rryk]

Just downloaded latest version and compiled from sources. After reading some code, I've registered on https://duplicati-oauth-handler.appspot.com/ with Google Drive login button and manually entered the following connection URI: googledrive://DuplicatiBackup?authid=my-authid.

This didn't work and web GUI crashed with "Error while running DriveD: Backend not supported: googledrive://DuplicatiBackup?authid=my-authid." error. I must be doing something wrong...

[kenkendk]

@rryk It sounds like the Google Drive dll is not being picked up. Make sure the Duplicat.Library.Backend.GoogleDrive.dll is in the folder where you are running Duplicati from.
@MalachiC Yes, I will make a build soon (in the weekend I think)

[rryk]

I've copied manually Duplicat.Library.Backend.GoogleDrive.dll and Duplicati.Library.OAuthHelper.dll into my current installation, restarted Duplicati but it still didn't work.

[kenkendk]

You also need the dll's from the thirdparty/GoogleAPI folder.

Are you on windows? I have only tested on OSX.

[rryk]

Yes, I am on Windows. Tried with command line util:

D:\Projects\duplicati\Duplicati\GUI\Duplicati.GUI.TrayIcon\bin\Release>Duplicati.CommandLine.exe backup googledrive://DuplicatiBackup?authid=my-authid D:\Books --snapshot-policy=required
Backup started at 25.6.15 11:03:12

Enter encryption passphrase: ************

Confirm encryption passphrase: ************
Fatal error => Attempted to perform an unauthorized operation.

Attempted to perform an unauthorized operation.
Update "2.0.0.85_preview_2015-03-07" detected

[rryk]

Copying thirdparty/GoogleAPI next to the Duplicati.CommandLine.exe didn't help.

[kenkendk]

Try adding --verbose --debug-output

[rryk]

Sorry, I was being stupid. I actually ran it in the project folder. Here is the output with --verbose and --debug-output from the production version (with needed libs copied):

D:\Software\Duplicati2>Duplicati.CommandLine.exe backup googledrive://DuplicatiBackup?authid=my-authid D:\Books --snapshot-policy=required --verbose --debug-output
Input command: backup
Input arguments:
        googledrive://DuplicatiBackup?authid=my-authid
        D:\Books

Input options:
snapshot-policy: required
verbose:
debug-output:

Backup started at 25.6.15 11:10:32

Enter encryption passphrase: ************

Confirm encryption passphrase: ************
The supplied option --authid is not supported and will be ignored

System.Exception:
The database has version 3 but the largest supported version is 2.

This is likely caused by upgrading to a newer version and then downgrading.
If this is the case, there is likely a backup file of the previous database version in the folder C:\Users\Rryk\AppData\Roaming\Duplicati.
   at Duplicati.Library.SQLiteHelper.DatabaseUpgrader.UpgradeDatabase(IDbConnection connection, String sourcefile, String schema, IList`1 versions)
   at Duplicati.Library.SQLiteHelper.DatabaseUpgrader.UpgradeDatabase(IDbConnection connection, String sourcefile, Type eltype)
   at Duplicati.Library.Main.Database.LocalDatabase.CreateConnection(String path)
   at Duplicati.Library.Main.Database.LocalBackupDatabase..ctor(String path, Options options)
   at Duplicati.Library.Main.Operation.BackupHandler.Run(String[] sources, IFilter filter)
   at Duplicati.Library.Main.Controller.<Backup>c__AnonStorey0.<>m__0(BackupResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, Action`1 method)
   at Duplicati.Library.Main.Controller.Backup(String[] inputsources, IFilter filter)
   at Duplicati.CommandLine.Commands.Backup(List`1 args, Dictionary`2 options, IFilter filter)
   at Duplicati.CommandLine.Program.RealMain(String[] args)

[rryk]

Perhaps we can continue this discussion on email rryk.ua [at] gmail.com to avoid spamming all the people who are watching this issue.

[rryk]

Finally managed to make the latest version work from sources:

1. Add Google APIs Client Library to the Duplicati.Library.Backend.GoogleDrive project via nuGet
2. Build entire solution in Release mode
3. Create a deployment folder
4. Copy the content of Duplicati/GUI/Duplicati.GUI.TrayIcon/bin/Release into this folder
5. Copy Duplicati/Server/webroot to the deployment folder

Without 1st step, it will complain about missing libraries. Copying thirdparty/GoogleAPIs to deployment folder didn't help. Without 5th step, the web interface won't load. I started the backup and left it run over night. Hopefully it will complete successfully by tomorrow.

[rryk]

Got an error after stopping after upload and restarting again:

System.OutOfMemoryException: Array dimensions exceeded supported range.
   at Duplicati.Library.Main.BackendManager.List()
   at Duplicati.Library.Main.Operation.FilelistProcessor.RemoteListAnalysis(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log)
   at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log)
   at Duplicati.Library.Main.Operation.BackupHandler.Run(String[] sources, IFilter filter)

I wonder why did I get OutOfMemoryException. I have 8 GB of memory and just 147 files on the backend. FileEntry is not a very large datastructure.

[rryk]

Synched to master and all worked. Nevermind.

[kenkendk]

This was fixed with fed160f and will be part of the next preview release

[kenkendk]

If you want to test the preview that is not yet released for automatic update, you can find it here:
http://updates.duplicati.com/preview/duplicati-2.0.0.86_preview_2015-07-02.zip