Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Any plan to add support to Google Photos? #369

Open
lssong99 opened this issue Feb 24, 2016 · 166 comments

Comments

@lssong99
Copy link

commented Feb 24, 2016

If possible, please add a support to upload photo/video files to Google Photos directly!

Although it's possible to add a "Google Photos" folder in Google Drive, and all your Google Photos will be there (organized by date folder), However, photos uploaded into this folder does not seems to reflect into Google Photos.

Also, if we upload in "High Quality" than there will be unlimited storage size for photos and Video. I am not sure the "down-sizing" is done locally or remotely by Google Photos server, however...

I realize Google Photo is not a good place to organize photos but it's a good place to share photos with others. And with a stock of 300k+ photos I really don't want to have my PC running for God-knows-how-long for the upload.... It's the job of RPi!

@ncw ncw added the enhancement label Feb 25, 2016

@ncw

This comment has been minimized.

Copy link
Collaborator

commented Feb 25, 2016

Thanks for the suggestion.

There seems to be a comprehensive API here

https://developers.google.com/picasa-web/docs/2.0/developers_guide_protocol

@lssong99

This comment has been minimized.

Copy link
Author

commented Feb 25, 2016

Yes, but I believe Picasa and it's API will be depreciated from May/1...

@ncw

This comment has been minimized.

Copy link
Collaborator

commented Feb 25, 2016

Foo! Is there a different API for google photos that you can find?

@buchno

This comment has been minimized.

Copy link

commented Mar 1, 2016

The Picasa API will continue to support uploading images:

Beginning May 1st, 2016, we’ll start rolling out changes to the Picasa Web Albums Data API and no longer support the following functionality:

  • Flash support
  • Community search
  • Mutation operations other than uploads
  • All support for tags, comments, and contacts

The API will still support other functions, including reading photos, reading albums, reading photos in albums, and uploading new photos. Although these operations will continue to be supported and the protocol will remain the same, the content included in the responses and the operation behavior may change.

https://developers.google.com/picasa-web/

Edit: Judging by this answer, it will however count towards the storage quota.

@lssong99

This comment has been minimized.

Copy link
Author

commented Mar 1, 2016

Even if it count towards the storage quota, it is still possible to use "reclaim space" in the setting of Photos.google.com which will re-compress (or just convert?) it into non-quota related "high resolution" photos.

Thus before Google comes up with new Photos API (God knows when...), do you think it's the best solution we could get so far?

@kiddick

This comment has been minimized.

@calisro

This comment has been minimized.

Copy link
Contributor

commented Mar 3, 2016

I've been trying to accomplish this for a long time. I have a huge local store of photos that I want to store in google photos at 'high resolution' as to have them NOT count towards the quotas but instead only use it to be able to see these photos across google devices. If we upload to drive, we get hit with quotas. It would be absolutely awesome to be able to sync them with rclone at a 'under the quota' limit to G-Photos.

@lv913

This comment has been minimized.

Copy link

commented Apr 26, 2016

+1

Another useful functionality could be to arrange photos/videos in albums, i.e. using the folder name for that.
Nick, do you think that it would be possible to have Google Photos in roadmap soon or later?

Thanks!

@BenoitDuffez

This comment has been minimized.

Copy link

commented May 16, 2016

I was able to sync photos using this option from Google Drive:

Google Drive Settings

This created a folder called Google Photos which I could sync using:

rclone --exclude '*.{dng,CR2}' --include '{IMG,PANO,VID}_*.{jpg,mp4}' sync 'MyRemoteName:/Google Photos/2016' /mnt/raid0/GooglePhotos/2016/

This examples removes the RAW images because they come from my DSLR, which is already backed up. I'm only interested in the pics uploaded from mobile.

@ncw

This comment has been minimized.

Copy link
Collaborator

commented May 16, 2016

@BenoitDuffez nice writeup thanks - a great way of downloading your pics from google photos.

It doesn't solve the upload part though which is what this issue is about.

@kiddick

This comment has been minimized.

Copy link

commented May 16, 2016

@ncw that's strange that Google still doesn't have official API for that..

@ncw ncw added this to the Unplanned / Help Wanted milestone Jul 15, 2016

@leocrawford

This comment has been minimized.

Copy link

commented Aug 8, 2016

I'm the author of picasawebsync and believe that the picasaweb API is the only one that currently works for reasons that baffle me. However the binding libraries don't support OAUTH2 which is the only one supported by the REST API, so you have to make a manual workaround.

A quick look though seems to imply that google now does offer a two way sync between drive and photos (https://support.google.com/photos/answer/6156103?co=GENIE.Platform%3DAndroid&hl=en-GB) though I can't readily test it as I'm a google apps user

@TimJWatts

This comment has been minimized.

Copy link

commented Aug 22, 2016

2 comments based on tests I just did:

a) You can upload to Google Drive and have them displayed in Google Photos if you turn the setting on as someone may have said above. With the exception that Panasonic RW2 raw files render in GDrive but do not show in GPhotos.

b) GDrive is quite a nice medium to display and share photos. But does not have all the polish of GPhotos. And does not have a free option for JPEGs (won't affect me - I want to archive my RW2s and processed TIFFs/DNGs).

But it would be nice to have rclone support if it can be figured out :)

@TioBorracho

This comment has been minimized.

Copy link
Contributor

commented Aug 22, 2016

Sadly the reclaim space feature wouldn't work since the photos are uploaded via google drive according to this

@gbvaz

This comment has been minimized.

Copy link

commented Nov 21, 2016

Hi, I know this is not a new topic, but is there any solution to the problem @TioBorracho was talking about? I'm facing the same situation, as related on Google Photos support forum topic.

Any help is appreciated!

@Webreaper

This comment has been minimized.

Copy link
Contributor

commented Feb 1, 2017

Just seen this thread. You may be interested in the little app I put together to solve this:
https://github.com/Webreaper/GooglePhotoSync/
It was primarily designed for running on a Mac desktop (it sits in the menu bar, like the Google Drive app, and just uploads/downloads photos to GPhotos. But I also introduced a 'headless' mode, so you can run it on a box without a GUI. I have it running on my Synology NAS, and used it for a few months to upload pictures to the cloud.
The app currently uploads at full res but once I remember the setting I plan to make it an option to upload at the 16MP setting (which is free and doesn't count against your quota) - the idea being that I'll use my unlimited Amazon Cloud storage for proper backup (via rclone) and then have a low-res copy in GPhotos to take advantage of their advanced image search (at least until Amazon bring their intelligent search, aka Family Vault, to the UK).

You could also run leocrawford's excellent sync tool, which would work too.

As has been discussed, the 'two-way' sync between drive and GPhotos is massively limited:

  • Despite uploading all photos into named, organised folders (albums) into GPhotos, if you make them available through GDrive they just appear in month/year folders, which is really annoying.
  • As has been mentioned, uploading to GDrive and then making the pictures available in GPhotos will not upload them as 'free' resolution. However, you could periodically click the 'Recover storage' option after doing this, and it'll resize them for you. I did this earlier this week (having moved all my full-res images to Amazon Drive) and it resized 1.2TB of images (around 200k x 5MB-9MB images) in about 36 hours.

Perhaps a neat solution would be to see if there's an API for 'Recover Storage' - you could then run rclone to upload to Gdrive, and then automatically trigger a 'recover storage' process once it's uploaded. :D

@lv913

This comment has been minimized.

Copy link

commented Feb 28, 2017

@Webreaper

This comment has been minimized.

Copy link
Contributor

commented Feb 28, 2017

@gerardo-junior

This comment has been minimized.

Copy link

commented Dec 18, 2017

Any prediction for this feature?

@dzg

This comment has been minimized.

Copy link

commented Dec 24, 2017

I would also love to be able to rclone to/from Google Photos.

Many items which are in my Google Photos are not showing up or syncing via the Google Photos folder in my Google Drive as they're supposed to.

The "Create a Google Photos folder" option in Drive is apparently unreliable; it does not show all items from Google Photos.

@Webreaper

This comment has been minimized.

Copy link
Contributor

commented Dec 24, 2017

It's a bit pointless requesting this feature when Google have shut down the API that allows you to write to GPhotos. Also, I can tell you from my experience of writing a client back when the API was still available, it's in no way rich enough or functional enough to be able to implement with rclone.

If you're putting photos in Gdrive and they're not showing up in GPhotos, you should raise a support ticket with Google. I have a quarter of a million photos in Gdrive and they all show up just fine in Photos.

@dzg

This comment has been minimized.

Copy link

commented Dec 24, 2017

@Webreaper Actually I'm more concerned with GP > GD ... My phone auto uploads all items to GP, then I use GD to sync them to my desktop. I noticed today that many items in my GP are not showing up in my GD.

@Webreaper

This comment has been minimized.

Copy link
Contributor

commented Dec 24, 2017

Ah, yes, we'll, given the lack of support for GPhotos, that doesn't surprise me at all. I had all sorts of problems with GPhotos not backing up photos, and so on. In the end I've moved to an entirely Gdrive based workflow.

  1. Turn off Google photos backup
  2. Install this: https://play.google.com/store/apps/details?id=com.ttxapps.drivesync
  3. Configure to sync to a particular folder when any new pics are taken
  4. Use rclone to sync from Gdrive back down to my pc.
    Works a treat, and a billion times more reliable (and faster too) than using Google's backup and autosync.
@lv913

This comment has been minimized.

Copy link

commented Dec 25, 2017

@Webreaper

This comment has been minimized.

Copy link
Contributor

commented Dec 25, 2017

If you upload photos to either GDrive or GPhotos and they're less than 16Mp, then they don't count against your storage tarifff. However, I'm using this as a primary offsite backup, so need photos stored in full ('original', as Google calls it) resolution, so yes, it's not free.

@dzg

This comment has been minimized.

Copy link

commented Dec 25, 2017

@Webreaper -- That sync app looks cool; but in the past I've always had issues with 3rd party syncers, mainly their not running until I launch them manually. Does this reliably upload in the background immediately upon file creation? THanks

@Peregrinox

This comment has been minimized.

Copy link

commented Jul 3, 2019

Hello,

I'd just made some tests with Beta 1.48.56,

rclone copy test gphotos:Album\test

If the "test" dir/Album has more than 1 image, the above command creates an album for each image, all with same name "test". At least, this is what I observe in the google photos android App

@ncw

This comment has been minimized.

Copy link
Collaborator

commented Jul 3, 2019

Hello,

I'd just made some tests with Beta 1.48.56,

rclone copy test gphotos:Album\test

If the "test" dir/Album has more than 1 image, the above command creates an album for each image, all with same name "test". At least, this is what I observe in the google photos android App

I've reproduced this - thanks for reporting :-) A work-around is to create the album first with rclone mkdir. I'll post a fix shortly.

@ncw

This comment has been minimized.

Copy link
Collaborator

commented Jul 4, 2019

I've merged a fix for the problem reported by @Peregrinox . That will be in the latest beta in 15-30 mins with version v1.48.0-063-gd7016866

@Peregrinox

This comment has been minimized.

Copy link

commented Jul 5, 2019

@ncw, many thanks for the fix.

After some more experimentation, now with beta 063, I founded that creation of Album with 'rclone mkdir gphotos:/album/test' before adding images to it, works well if there is some delay between the two comands.

In the bash script:

ALBUM= some dir with multiples images
rclone mkdir gphotos:album/"$ALBUM"
# sleep 6
rclone copy "$ALBUM" gphotos:album/"$ALBUM"

Without the intermediate "sleep" delay, the "copy" causes the same behaviour, as described in my previous post, of one album per image in the ALBUM folder. It seems that album creation is not inmediatly avaiable to the API just after the "mkdir".

Of course, typing the comands in a shell, causes the same effect as the "sleep X" delay if you dont't have superfasts fingers

@ncw

This comment has been minimized.

Copy link
Collaborator

commented Jul 5, 2019

Without the intermediate "sleep" delay, the "copy" causes the same behaviour, as described in my previous post, of one album per image in the ALBUM folder. It seems that album creation is not inmediatly avaiable to the API just after the "mkdir".

Hmm, you shouldn't need the rclone mkdir now at all. What happens if you try without it?

@Peregrinox

This comment has been minimized.

Copy link

commented Jul 5, 2019

It´s behaving nice now. I think that my tests with some old duplicated album names were the source of my troubles. I tried again with some fresh album names and images.

Sorry about the confusion.

There is no need to mkdir at all. My albumns are upped great with copy or move

@ncw

This comment has been minimized.

Copy link
Collaborator

commented Jul 5, 2019

There is no need to mkdir at all. My albumns are upped great with copy or move

Great :-)

@badzz

This comment has been minimized.

Copy link

commented Jul 8, 2019

Hi, Thanks a lot for the hard work
If I want to do a full backup of my gphotos and keep the albums structure (some photos are not in albums),should I do
rclone sync remote:media/all /mnt/backup/media
and
rclone sync remote:album /mnt/backup/album
And I will have twice my photos that are in albums , or there is a better way ?
Thanks
PS: There seems to be a bug with album names containing / character . the name stops before the /

@calisro

This comment has been minimized.

Copy link
Contributor

commented Jul 11, 2019

Im a little confused. I understand that videos aren't being downloaded at full resolution but what about pictures? I took a look at the limitations and it only talks about EXIF data. To me it looks like the downloads of images are being scaled so I wanted to confirm. If so, that should be added to the limitations page.

@badzz

This comment has been minimized.

Copy link

commented Jul 11, 2019

@calisro

This comment has been minimized.

Copy link
Contributor

commented Jul 11, 2019

(Wasn't sure if I should open a new ISSUE. I can if you'd like @ncw )

I was testing this today. I'm using this to upload photos rather than downloading them for this test.

I'm finding that I am seeing a lot of corrupted on transfers. I'm not sure why but it is consistent for many photos/vids. I thought there may be the case where the file(s) already existed in GPhotos uploaded by a different method but that doesn't appear to be the case as I've searched both in the web interface and the rclone cli to see if it exists.

This is one example:

root@xxxx:/data/bin# rclone copy test robgs-photos:album -vv --tpslimit=1 --include="2019/05/Emerson and Maddy Joke Talen1 show.jpg"
2019/07/11 13:48:17 DEBUG : rclone: Version "v1.48.0-066-g5433021e-beta" starting with parameters ["rclone" "copy" "test" "robgs-photos:album" "-vv" "--tpslimit=1" "--include=2019/05/Emerson and Maddy Joke Talen1 show.jpg"]
2019/07/11 13:48:17 DEBUG : Using config file from "/root/.rclone.conf"
2019/07/11 13:48:17 INFO  : Starting HTTP transaction limiter: max 1 transactions/s with burst 1
2019/07/11 13:48:17 DEBUG : Google Photos path "album": List: dir=""
2019/07/11 13:48:18 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/11 13:48:18 DEBUG : Google Photos path "album": List: dir="2019"
2019/07/11 13:48:18 DEBUG : Google Photos path "album": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2019/07/11 13:48:18 DEBUG : Google Photos path "album": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2019/07/11 13:48:18 DEBUG : Google Photos path "album": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2019/07/11 13:48:18 DEBUG : Google Photos path "album": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2019/07/11 13:48:18 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/11 13:48:18 DEBUG : 2019/07: Excluded
2019/07/11 13:48:18 DEBUG : 2019/06: Excluded
2019/07/11 13:48:18 DEBUG : 2019/01: Excluded
2019/07/11 13:48:18 DEBUG : 2019/03: Excluded
2019/07/11 13:48:18 DEBUG : 2019/04: Excluded
2019/07/11 13:48:18 DEBUG : Google Photos path "album": List: dir="2019/05"
2019/07/11 13:48:18 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/11 13:48:18 DEBUG : Google Photos path "album": Put: src=2019/05/Emerson and Maddy Joke Talen1 show.jpg
2019/07/11 13:48:18 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Update: src=2019/05/Emerson and Maddy Joke Talen1 show.jpg
2019/07/11 13:48:18 INFO  : Google Photos path "album": Waiting for checks to finish
2019/07/11 13:48:18 INFO  : Google Photos path "album": Waiting for transfers to finish
2019/07/11 13:48:21 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: >Update: err=<nil>
2019/07/11 13:48:21 DEBUG : Google Photos path "album": >Put: 
2019/07/11 13:48:21 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Size: 
2019/07/11 13:48:21 DEBUG : Google Photos path "album": List: dir="2019/05"
2019/07/11 13:48:22 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/11 13:48:23 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: >Size: 
2019/07/11 13:48:23 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Size: 
2019/07/11 13:48:23 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: >Size: 
2019/07/11 13:48:23 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Size: 
2019/07/11 13:48:23 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: >Size: 
2019/07/11 13:48:23 ERROR : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: corrupted on transfer: sizes differ 85040 vs 67499
2019/07/11 13:48:23 INFO  : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Removing failed copy
2019/07/11 13:48:24 ERROR : Attempt 1/3 failed with 2 errors and: corrupted on transfer: sizes differ 85040 vs 67499
2019/07/11 13:48:24 DEBUG : Google Photos path "album": List: dir=""
2019/07/11 13:48:24 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/11 13:48:24 DEBUG : Google Photos path "album": List: dir="2019"
2019/07/11 13:48:24 DEBUG : Google Photos path "album": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2019/07/11 13:48:24 DEBUG : Google Photos path "album": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2019/07/11 13:48:24 DEBUG : Google Photos path "album": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2019/07/11 13:48:24 DEBUG : Google Photos path "album": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2019/07/11 13:48:24 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/11 13:48:24 DEBUG : 2019/07: Excluded
2019/07/11 13:48:24 DEBUG : 2019/06: Excluded
2019/07/11 13:48:24 DEBUG : 2019/01: Excluded
2019/07/11 13:48:24 DEBUG : 2019/03: Excluded
2019/07/11 13:48:24 DEBUG : 2019/04: Excluded
2019/07/11 13:48:24 DEBUG : Google Photos path "album": List: dir="2019/05"
2019/07/11 13:48:25 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/11 13:48:25 DEBUG : Google Photos path "album": Put: src=2019/05/Emerson and Maddy Joke Talen1 show.jpg
2019/07/11 13:48:25 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Update: src=2019/05/Emerson and Maddy Joke Talen1 show.jpg
2019/07/11 13:48:25 INFO  : Google Photos path "album": Waiting for checks to finish
2019/07/11 13:48:25 INFO  : Google Photos path "album": Waiting for transfers to finish

EDIT: Well this is interesting. If I upload it to the /upload path instead of albums, it works. Now its in the GUI too. and if I rerun the album upload after it still fails with the exact same message. I've tried changing the album names just to see but that didn't help.

root@cloudc:~# rclone copy robgs:Local-Pictures robgs-photos:upload -vv --tpslimit=1 --include="2019/05/Emerson and Maddy Joke Talen1 show.jpg"
2019/07/12 10:46:37 DEBUG : rclone: Version "v1.48.0-066-g5433021e-beta" starting with parameters ["rclone" "copy" "robgs:Local-Pictures" "robgs-photos:upload" "-vv" "--tpslimit=1" "--include=2019/05/Emerson and Maddy Joke Talen1 show.jpg"]
2019/07/12 10:46:37 DEBUG : Using config file from "/root/.rclone.conf"
2019/07/12 10:46:37 INFO  : Starting HTTP transaction limiter: max 1 transactions/s with burst 1
2019/07/12 10:46:38 DEBUG : Google Photos path "upload": List: dir=""
2019/07/12 10:46:38 DEBUG : Google Photos path "upload": >List: err=directory not found
..
.. <snip>
..
2019/07/12 10:46:46 INFO  : Google Photos path "upload": Waiting for checks to finish
2019/07/12 10:46:46 INFO  : Google Photos path "upload": Waiting for transfers to finish
2019/07/12 10:46:46 DEBUG : Google Photos path "upload": Put: src=2019/05/Emerson and Maddy Joke Talen1 show.jpg
2019/07/12 10:46:46 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Update: src=2019/05/Emerson and Maddy Joke Talen1 show.jpg
2019/07/12 10:46:49 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: >Update: err=<nil>
2019/07/12 10:46:49 DEBUG : Google Photos path "upload": >Put: 
2019/07/12 10:46:49 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Size: 
2019/07/12 10:46:49 DEBUG : Google Photos path "upload": List: dir="2019/05"
2019/07/12 10:46:49 DEBUG : Google Photos path "upload": >List: err=directory not found
2019/07/12 10:46:49 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Size: Failed to read metadata: object not found
2019/07/12 10:46:49 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: >Size: 
2019/07/12 10:46:49 INFO  : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Copied (new)
@ncw

This comment has been minimized.

Copy link
Collaborator

commented Jul 24, 2019

Apologies for the delay in responding, I've been on vacation.

@calisro @badzz group reply :-)

If I want to do a full backup of my gphotos and keep the albums structure (some photos are not in albums),should I do
rclone sync remote:media/all /mnt/backup/media

I'd recommend using media:by-month probably as the directory will get a bit big.

and
rclone sync remote:album /mnt/backup/album
And I will have twice my photos that are in albums , or there is a better way ?

There isn't a better way at the moment :-( Though you could just list the albums rclone lsf -R gphotos:albums which would give you enough info to reconstruct them - you could do this with a script and a bit of hardlinking...

PS: There seems to be a bug with album names containing / character . the name stops before the /

What rclone should have done is made a sub directory so if the album
is called photos from mum/dad you'll get a directory called
photos from mum with a directory called dad inside it.

Im a little confused. I understand that videos aren't being downloaded at full resolution but what about pictures? I took a look at the limitations and it only talks about EXIF data. To me it looks like the downloads of images are being scaled so I wanted to confirm. If so, that should be added to the limitations page.

Hi the bug clearly states that the sizes are different as well. My test showsthat indeed as well rclone doc does not say it, but it is indeed the case (plus EXIF location missing) https://issuetracker.google.com/issues/112096115 du -sh web.jpg api.jpg 2.4M web.jpg 616K api.jpg

In my tests I uploaded a picture then downloaded it and the only difference was the GPS info snipped from the EXIF data. So my re-download was a few bytes shorter only.

How did you do your upload? Could it have been one done at reduced resolution?

2019/07/11 13:48:23 ERROR : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: corrupted on transfer: sizes differ 85040 vs 67499

That looks like a bug.

This doesn't happen for all files does it?

Is it something special about that file? If so any chance you could send it to me either attach it here or email it to nick@craig-wood.com

@vitobotta

This comment has been minimized.

Copy link

commented Jul 24, 2019

Hi! I read the doc but I am confused. If I just want to periodically back up my photos from Google Photos to a local disk, which commands should I use? Thanks

@badzz

This comment has been minimized.

Copy link

commented Jul 24, 2019

@badzz

This comment has been minimized.

Copy link

commented Jul 24, 2019

@vitobotta

This comment has been minimized.

Copy link

commented Jul 24, 2019

On Wed, Jul 24, 2019, 18:10 Vito Botta @.***> wrote: Hi! I read the doc but I am confused. If I just want to periodically back up my photos from Google Photos to a local disk, which commands should I use? Thanks
rclone sync remote:media/by-month /mnt/backup/media —

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#369?email_source=notifications&email_token=ABR5UYJR6ESGLX7OKY3MOIDQBBWGJA5CNFSM4B4HRKVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2WUVPY#issuecomment-514673343>, or mute the thread https://github.com/notifications/unsubscribe-auth/ABR5UYJZ64SWD422CA4SJITQBBWGJANCNFSM4B4HRKVA .

Hi, what happens if I run it multiple times, will it skip the files already downloaded or? Thanks!

@badzz

This comment has been minimized.

Copy link

commented Jul 24, 2019

@vitobotta

This comment has been minimized.

Copy link

commented Jul 24, 2019

.
Hi, what happens if I run it multiple times, will it skip the files already downloaded or? Thanks!
Sure, yes

Wonderful, thanks a lot!

@gedl

This comment has been minimized.

Copy link

commented Jul 27, 2019

Where do the "magic" Assistant creations end up in? Is there a way to retrieve only those?

My use case would be to backup from local into google photos and then download the auto creations to somewhere. Not sure if into the same tree or somewhere else.

I plan to use Google Photos to view my photos - they are already backed up somewhere else too.

@calisro

This comment has been minimized.

Copy link
Contributor

commented Jul 28, 2019

How did you do your upload? Could it have been one done at reduced resolution?

2019/07/11 13:48:23 ERROR : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: corrupted on transfer: sizes differ 85040 vs 67499

That looks like a bug.

This doesn't happen for all files does it?

Is it something special about that file? If so any chance you could send it to me either attach it here or email it to nick@craig-wood.com

@ncw It happens for the vast majority of the files. Ill send you one. But it's consistent across a lot of them. I was simply uploading to an album. (Doesn't matter if I create a new one or an existing)

This is an example:
rclone copy robgs:Local-Pictures robgs-photos:album -vv --tpslimit=1 --include="2019/05/Emerson and Maddy Joke Talen1 show.jpg"

I've also tried from a local drive rather than Gdrive to gdrive.

root@cloudc:~/2019/05# rclone copy . robgs-photos:album/ -vv --tpslimit=1 
2019/07/28 18:14:00 DEBUG : rclone: Version "v1.48.0-066-g5433021e-beta" starting with parameters ["rclone" "copy" "." "robgs-photos:album/" "-vv" "--tpslimit=1"]
2019/07/28 18:14:00 DEBUG : Using config file from "/root/.rclone.conf"
2019/07/28 18:14:00 INFO  : Starting HTTP transaction limiter: max 1 transactions/s with burst 1
2019/07/28 18:14:00 DEBUG : Google Photos path "album": List: dir=""
2019/07/28 18:14:01 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/28 18:14:01 DEBUG : Google Photos path "album": Put: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:01 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: Update: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:01 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: >Update: err=can't upload files here
2019/07/28 18:14:01 DEBUG : Google Photos path "album": >Put: 
2019/07/28 18:14:01 ERROR : Emerson and Maddy Joke Talen1 show.jpg: Failed to copy: can't upload files here
2019/07/28 18:14:01 INFO  : Google Photos path "album": Waiting for checks to finish
2019/07/28 18:14:01 INFO  : Google Photos path "album": Waiting for transfers to finish
2019/07/28 18:14:01 ERROR : Attempt 1/3 failed with 2 errors and: can't upload files here
2019/07/28 18:14:01 DEBUG : Google Photos path "album": List: dir=""
2019/07/28 18:14:01 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/28 18:14:01 DEBUG : Google Photos path "album": Put: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:01 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: Update: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:01 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: >Update: err=can't upload files here
2019/07/28 18:14:01 DEBUG : Google Photos path "album": >Put: 
2019/07/28 18:14:01 ERROR : Emerson and Maddy Joke Talen1 show.jpg: Failed to copy: can't upload files here
2019/07/28 18:14:01 INFO  : Google Photos path "album": Waiting for checks to finish
2019/07/28 18:14:01 INFO  : Google Photos path "album": Waiting for transfers to finish
2019/07/28 18:14:01 ERROR : Attempt 2/3 failed with 2 errors and: can't upload files here
2019/07/28 18:14:01 DEBUG : Google Photos path "album": List: dir=""
2019/07/28 18:14:01 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/28 18:14:01 INFO  : Google Photos path "album": Waiting for checks to finish
2019/07/28 18:14:01 INFO  : Google Photos path "album": Waiting for transfers to finish
2019/07/28 18:14:01 DEBUG : Google Photos path "album": Put: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:01 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: Update: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:01 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: >Update: err=can't upload files here
2019/07/28 18:14:01 DEBUG : Google Photos path "album": >Put: 
2019/07/28 18:14:01 ERROR : Emerson and Maddy Joke Talen1 show.jpg: Failed to copy: can't upload files here
2019/07/28 18:14:01 ERROR : Attempt 3/3 failed with 2 errors and: can't upload files here
2019/07/28 18:14:01 Failed to copy with 2 errors: last error was: can't upload files here
root@cloudc:~/2019/05# rclone copy . robgs-photos:album/ad -vv --tpslimit=1 
2019/07/28 18:14:07 DEBUG : rclone: Version "v1.48.0-066-g5433021e-beta" starting with parameters ["rclone" "copy" "." "robgs-photos:album/ad" "-vv" "--tpslimit=1"]
2019/07/28 18:14:07 DEBUG : Using config file from "/root/.rclone.conf"
2019/07/28 18:14:07 INFO  : Starting HTTP transaction limiter: max 1 transactions/s with burst 1
2019/07/28 18:14:07 DEBUG : Google Photos path "album/ad": List: dir=""
2019/07/28 18:14:07 DEBUG : Google Photos path "album/ad": >List: err=directory not found
2019/07/28 18:14:07 DEBUG : Google Photos path "album/ad": Put: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:07 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: Update: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:07 INFO  : Google Photos path "album/ad": Waiting for checks to finish
2019/07/28 18:14:07 INFO  : Google Photos path "album/ad": Waiting for transfers to finish
2019/07/28 18:14:12 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: >Update: err=<nil>
2019/07/28 18:14:12 DEBUG : Google Photos path "album/ad": >Put: 
2019/07/28 18:14:12 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: Size: 
2019/07/28 18:14:12 DEBUG : Google Photos path "album/ad": List: dir=""
2019/07/28 18:14:12 DEBUG : Google Photos path "album/ad": >List: err=<nil>
2019/07/28 18:14:13 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: >Size: 
2019/07/28 18:14:13 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: Size: 
2019/07/28 18:14:13 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: >Size: 
2019/07/28 18:14:13 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: Size: 
2019/07/28 18:14:13 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: >Size: 
2019/07/28 18:14:13 ERROR : Emerson and Maddy Joke Talen1 show.jpg: corrupted on transfer: sizes differ 85040 vs 67499
2019/07/28 18:14:13 INFO  : Emerson and Maddy Joke Talen1 show.jpg: Removing failed copy
2019/07/28 18:14:15 ERROR : Attempt 1/3 failed with 2 errors and: corrupted on transfer: sizes differ 85040 vs 67499
2019/07/28 18:14:15 DEBUG : Google Photos path "album/ad": List: dir=""
2019/07/28 18:14:16 DEBUG : Google Photos path "album/ad": >List: err=<nil>
2019/07/28 18:14:16 INFO  : Google Photos path "album/ad": Waiting for checks to finish
2019/07/28 18:14:16 INFO  : Google Photos path "album/ad": Waiting for transfers to finish
2019/07/28 18:14:16 DEBUG : Google Photos path "album/ad": Put: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:16 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: Update: src=Emerson and Maddy Joke Talen1 show.jpg

@andreask1

This comment has been minimized.

Copy link

commented Aug 4, 2019

@ncw I just did some testing with the latest beta version (v1.48.0-103-g6a3e3013-beta) using sync remote:media/by-month and ran into the following issues:

  • No photos before Jan 1st, 2000 were synced
  • No archived photos were synced (the old Drive sync included them in their original folder)
@Conroman16

This comment has been minimized.

Copy link

commented Aug 6, 2019

This is looking pretty good. That said, I did some testing on the current beta (rclone v1.48.0-131-ge502be47-beta) and I too see that my archived photos are not showing up. I see a reference to it here however it doesn't appear that param has been implemented yet. Are there any plans to do so at this time? The ability to work with the archived media is a must in my use case.

@lssong99

This comment has been minimized.

Copy link
Author

commented Aug 10, 2019

IT's been two years since I opened this thread... Thank you guys just make this work in time for GDrive and GPhoto's desync. (Nobody could predict then...)

The beta version still have some quarks to remove but really a good progress...

Looking at ls output, it seems the duplicated files are getting some random filename tails to make them unique... Really missed the dedupe command here.... Before Google cut tie between GDrive and GPhoto, all I need to do is dedupe the GDrive MyPhoto folder and duplicates are eliminated... Not sure if it's possible with rclone in GPhoto... (Tried with latest beta version but doesn't seems to work due to error....)

2019/08/10 19:27:10 Failed to dedupe with 20 errors: last error was: google photos API only implements removing files from albums

Any way to do this from non-album way...?

Edit even album way no work... The command doesn't return anything by exit after quite a bit time without any output...

-1 PANO_20190714_181128.vr {ANZQEuahFdXObFCdxl7GbrcwKbzxKWkxbVAf80dfadsfasdflka13ai1-wcBwa2fe3KbKrNp18k45g_XHOcfeGspQ1Zw}.jpg -1 PANO_20190714_181128.vr {ANZQEubpau27dgeUZf;lkasd;flkasdflj-F7r3dWMdaxlwtWQky-vNyjvEuVy83xrk5dLuJ4rMZE6g2umC2OkaIUNaWdtqOIE7_VOJYA}.jpg -1 PANO_20190714_200502.vr {ANZQEua-JRSn8zMFbMpp9Src6T95HMSPcpmmX6AHwFJHPgTiOGkfa;lskdf;laskflkjasefNzsov9mlzwVM0Jp8sii32DjHC4g}.jpg -1 PANO_20190714_200502.vr {ANZQEubh0LFwOem4iuM8YJayhjafslkjlskadjf;laksjf;alsfCMEuL2s09ZaUSNlQSn_PZgWmCGb1muYSZ4b-Zg}.jpg

@ncw

This comment has been minimized.

Copy link
Collaborator

commented Aug 11, 2019

IT's been two years since I opened this thread... Thank you guys just make this work in time for GDrive and GPhoto's desync. (Nobody could predict then...)

The beta version still have some quarks to remove but really a good progress...

:-)

I've almost got it approved by google, a few more hurdles to jump!

Looking at ls output, it seems the duplicated files are getting some random filename tails to make them unique... Really missed the dedupe command here.... Before Google cut tie between GDrive and GPhoto, all I need to do is dedupe the GDrive MyPhoto folder and duplicates are eliminated... Not sure if it's possible with rclone in GPhoto... (Tried with latest beta version but doesn't seems to work due to error....)

2019/08/10 19:27:10 Failed to dedupe with 20 errors: last error was: google photos API only implements removing files from albums

Unfortunately the API doesn't support renaming media items. Yes the API is really limited which is annoying :-(

So rclone can't support dedupe.

Any way to do this from non-album way...?

The API pretty much limits you to using albums if you want to read and write. You can read the media items but you can't rename/delete them.

-1 PANO_20190714_181128.vr {ANZQEuahFdXObFCdxl7GbrcwKbzxKWkxbVAf80dfadsfasdflka13ai1-wcBwa2fe3KbKrNp18k45g_XHOcfeGspQ1Zw}.jpg -1 PANO_20190714_181128.vr {ANZQEubpau27dgeUZf;lkasd;flkasdflj-F7r3dWMdaxlwtWQky-vNyjvEuVy83xrk5dLuJ4rMZE6g2umC2OkaIUNaWdtqOIE7_VOJYA}.jpg -1 PANO_20190714_200502.vr {ANZQEua-JRSn8zMFbMpp9Src6T95HMSPcpmmX6AHwFJHPgTiOGkfa;lskdf;laskflkjasefNzsov9mlzwVM0Jp8sii32DjHC4g}.jpg -1 PANO_20190714_200502.vr {ANZQEubh0LFwOem4iuM8YJayhjafslkjlskadjf;laksjf;alsfCMEuL2s09ZaUSNlQSn_PZgWmCGb1muYSZ4b-Zg}.jpg

Indeed!

The {xxxx} are the unique IDs of the media items.

@lssong99

This comment has been minimized.

Copy link
Author

commented Aug 12, 2019

IT's been two years since I opened this thread... Thank you guys just make this work in time for GDrive and GPhoto's desync. (Nobody could predict then...)
The beta version still have some quarks to remove but really a good progress...

:-)

I've almost got it approved by google, a few more hurdles to jump!

Looking at ls output, it seems the duplicated files are getting some random filename tails to make them unique... Really missed the dedupe command here.... Before Google cut tie between GDrive and GPhoto, all I need to do is dedupe the GDrive MyPhoto folder and duplicates are eliminated... Not sure if it's possible with rclone in GPhoto... (Tried with latest beta version but doesn't seems to work due to error....)
2019/08/10 19:27:10 Failed to dedupe with 20 errors: last error was: google photos API only implements removing files from albums

Unfortunately the API doesn't support renaming media items. Yes the API is really limited which is annoying :-(

So rclone can't support dedupe.

Any way to do this from non-album way...?

The API pretty much limits you to using albums if you want to read and write. You can read the media items but you can't rename/delete them.

-1 PANO_20190714_181128.vr {ANZQEuahFdXObFCdxl7GbrcwKbzxKWkxbVAf80dfadsfasdflka13ai1-wcBwa2fe3KbKrNp18k45g_XHOcfeGspQ1Zw}.jpg -1 PANO_20190714_181128.vr {ANZQEubpau27dgeUZf;lkasd;flkasdflj-F7r3dWMdaxlwtWQky-vNyjvEuVy83xrk5dLuJ4rMZE6g2umC2OkaIUNaWdtqOIE7_VOJYA}.jpg -1 PANO_20190714_200502.vr {ANZQEua-JRSn8zMFbMpp9Src6T95HMSPcpmmX6AHwFJHPgTiOGkfa;lskdf;laskflkjasefNzsov9mlzwVM0Jp8sii32DjHC4g}.jpg -1 PANO_20190714_200502.vr {ANZQEubh0LFwOem4iuM8YJayhjafslkjlskadjf;laksjf;alsfCMEuL2s09ZaUSNlQSn_PZgWmCGb1muYSZ4b-Zg}.jpg

Indeed!

The {xxxx} are the unique IDs of the media items.

However, it seems is lsd a album, nothing shows up....

@badzz

This comment has been minimized.

Copy link

commented Aug 12, 2019

@lssong99 : normal lsd list folders. There isn't any in an album. Try lsf

@lssong99

This comment has been minimized.

Copy link
Author

commented Aug 12, 2019

@lssong99 : normal lsd list folders. There isn't any in an album. Try lsf

Yes! Got it! How did I miss such a simple thing!?

It seems the size of photo are always -1 (no matter lsf or ls)... Anyway to get the size?

If I could get the size or the upload date, then it's possible to write an external script to simulate --dedupe-mode newest/largest etc.....

Also, by looking at GPhoto API page, it states
Note that you can only remove media items that your application has added to an album or that have been created in an album as part of an upload. For albums that are shared, you can only remove items added by other collaborators if you are acting on behalf of the owner of the album.

Does this means we rclone will only manage (delete) photos uploaded by rclone?

@ncw

This comment has been minimized.

Copy link
Collaborator

commented Aug 13, 2019

It seems the size of photo are always -1 (no matter lsf or ls)... Anyway to get the size?

You can use --gphotos-read-size. However it will really slow things down and I don't recommend it.

If I could get the size or the upload date, then it's possible to write an external script to simulate --dedupe-mode newest/largest etc.....

The dates on the photos are the EXIF created dates and they should be reliable. Gphotos doesn't support reading the upload date.

Also, by looking at GPhoto API page, it states
Note that you can only remove media items that your application has added to an album or that have been created in an album as part of an upload. For albums that are shared, you can only remove items added by other collaborators if you are acting on behalf of the owner of the album.

Does this means we rclone will only manage (delete) photos uploaded by rclone?

That is correct. Check out the limitations section in the docs: https://tip.rclone.org/googlephotos/#limitations

@ncw ncw modified the milestones: v1.49, v1.50 Aug 27, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.