No new photos

[daboynb]

Hi,
I have a problem.
I am running this app on a raspberry pi 4. When I run it the first time it downloads the photos but there is a problem :

• If I upload some photos to gphoto and run the program it tells me that all photos are already downloaded

I tested the issue on another account too.

How can I fix that? Thanks

Btw this tool is awesome 😎

Testing on 4 accounts.

I uploaded some photos but nothing....

[daboynb]

27/10/2023 The issue is back.

28/10/2023 New photos are synced now.

So, the problem is the delay, it can takes one day or two to sync the new photos.

The script run every hour through cron .

[gilesknap]

@daboynb please send the command line that you are running in the cron. Thanks.

[daboynb]

Sure, this is the script.

I saw the log this morning, the first account got synced with the new photos but not the other three.

#!/bin/bash

# usereone
python3 -m venv /home/pico/ssd/g.usereone/venv

source /home/pico/ssd/g.usereone/venv/bin/activate

gphotos-sync --secret /home/pico/ssd/g.usereone/bkdir/client_secret.json /home/pico/ssd/g.usereone/bkdir/

deactivate

# usertwo
python3 -m venv /home/pico/ssd/g.usertwo/venv

source /home/pico/ssd/g.usertwo/venv/bin/activate

gphotos-sync --secret /home/pico/ssd/g.usertwo/bkdir/client_secret.json /home/pico/ssd/g.usertwo/bkdir/

deactivate

# userthree
python3 -m venv /home/pico/ssd/g.userthree/venv

source /home/pico/ssd/g.userthree/venv/bin/activate

gphotos-sync --secret /home/pico/ssd/g.userthree/bkdir/client_secret.json /home/pico/ssd/g.userthree/bkdir/

# userfour

python3 -m venv /home/pico/ssd/g.userfour/venv

source /home/pico/ssd/g.userfour/venv/bin/activate

gphotos-sync --secret /home/pico/ssd/g.userfour/bkdir/client_secret.json /home/pico/ssd/g.userfour/bkdir/

deactivate

Thank you.

[gilesknap]

This looks fine to me. I wonder if you are subject to a timezone problem.

Please could you temporarily set timezone to GMT and see if the issue goes away?

We did have timezone issues with token generation as discussed here: #413

I just noticed that I have not made a release since the above fix was merged. However this won't help you as your issue would be to do with timezone info when choosing media search filters.

[daboynb]

I tried with :

sudo timedatectl set-timezone GMT

               Local time: Wed 2023-11-01 11:48:28 GMT
           Universal time: Wed 2023-11-01 11:48:28 UTC
                 RTC time: n/a
                Time zone: GMT (GMT, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Sadly, it still says no new photos.

[gilesknap]

Hmmm. Well in that case its hard to pin the problem on gphotos-sync and it's back to blaming the Google API I'm afraid.

However if you want to do more analysis please add --log-level trace to your command line and post the trace and log files that appear in your gphotos-sync root folder. Hopefully they won't be too big we are not finding any photos to download.

[daboynb]

This is the output.

11-01 22:40:35 WARNING  gphotos-sync 3.1.3 2023-11-01 22:40:35.924339
11-01 22:40:35 DEBUG    MINIMUM_DATE = 1800-01-01 00:00:00
11-01 22:40:35 INFO     Target filesystem /home/pico/ssd/g.userone/bkdir is ext4
11-01 22:40:35 DEBUG    Checking if is filesystem supports symbolic links...
11-01 22:40:35 DEBUG    attempting to symlink /home/pico/ssd/g.userone/bkdir/test_src_537777397 to /home/pico/ssd/g.userone/bkdir/test_dst_514065428
11-01 22:40:35 DEBUG    Checking if File system supports unicode filenames...
11-01 22:40:35 INFO     Filesystem supports Unicode filenames
11-01 22:40:35 DEBUG    Checking if File system is case insensitive...
11-01 22:40:35 INFO     Case sensitive file system found
11-01 22:40:35 INFO     Max Path Length: 4096
11-01 22:40:35 INFO     Max filename length: 255
11-01 22:40:35 INFO     version: 3.1.3, database schema version 5.7
11-01 22:40:36 DEBUG    bad_ids file, loaded 1 bad ids
11-01 22:40:36 WARNING  Indexing Google Photos Files ...
11-01 22:40:36 INFO     searching for media start=2023-10-29 17:31:26, end=None, videos=True
11-01 22:40:36 DEBUG    mediaItems.search with body:
{'pageToken': None, 'pageSize': 100, 'filters': {'dateFilter': {'ranges': [{'startDate': {'year': 2023, 'month': 10, 'day': 29}, 'endDate': {'year': 3000, 'month': 1, 'day': 1}}]}, 'mediaTypeFilter': {'mediaTypes': ['ALL_MEDIA']}, 'featureFilter': {'includedFeatures': ['NONE']}, 'includeArchivedMedia': False}}
11-01 22:40:37 DEBUG    Skipped Index (already indexed) 1 photos/2023/10/IMG-20231029-WA0029.jpeg
11-01 22:40:37 DEBUG    Skipped Index (already indexed) 2 photos/2023/10/IMG_20231029_122721.jpg
11-01 22:40:37 DEBUG    Skipped Index (already indexed) 3 photos/2023/10/IMG_20231029_020233.jpg
11-01 22:40:37 DEBUG    search_media parsed 3 media_items with 100 PAGE_SIZE
11-01 22:40:37 WARNING  indexed 0 items
11-01 22:40:37 WARNING  Downloading Photos ...
11-01 22:40:37 WARNING  Downloaded 0 Items, Failed 0, Already Downloaded 4499
11-01 22:40:37 WARNING  WARNING: skipped 1 files listed in /home/pico/ssd/g.userone/bkdir/gphotos.bad_ids.yaml
11-01 22:40:37 INFO     Saving Database ...
11-01 22:40:37 INFO     Database Saved.
11-01 22:40:37 WARNING  Done.
11-01 22:40:37 INFO     Elapsed time = 0:00:01.220704

[gilesknap]

Thanks. Did you also get a .trace file - that would show the actual calls to the Google API.

What we want to do is verify that this line:
11-01 22:40:36 INFO searching for media start=2023-10-29 17:31:26, end=None, videos=True
is doing the right thing. The media start time should be the last time you ran gphotos-sync and if there are photos that are newer than that date they should get indexed and downloaded.

So it would be good to be able to show that

1. you have a photo that is newer than media start date
2. the call to the google API used the same date (which we could see in the trace file)
3. that nothing was found by the Google API

Then we either pin it on Google - we would have a detailed report to give them. Or work out that there is perhaps a mismatch between timezones used by gphotos and the Google API (unfortunately I'm thinking this is unlikely now that you have tried using GMT)

[daboynb]

Here you go the file of the first account.

11-02 13:10:21 gphotos_sync.restclient TRACE
REQUEST: POST to https://photoslibrary.googleapis.com/v1/mediaItems:search params={}
{"pageToken": null, "pageSize": 100, "filters": {"dateFilter": {"ranges": [{"startDate": {"year": 2023, "month": 10, "day": 31}, "endDate": {"year": 3000, "month": 1, "day": 1}}]}, "mediaTypeFilter": {"mediaTypes": ["ALL_MEDIA"]}, "featureFilter": {"includedFeatures": ["NONE"]}, "includeArchivedMedia": false}}
11-02 13:10:22 gphotos_sync.restclient TRACE
RESPONSE: 200
b'{}\n'

BTW the second account gets updated but sadly it not saved a trace file.
To let the trace file populate the gphoto trace file must be deleted before running the command.

11-02 01:00:16 WARNING  gphotos-sync 3.1.3 2023-11-02 01:00:16.656022
11-02 01:00:17 WARNING  Indexing Google Photos Files ...
11-02 01:00:20 WARNING  indexed 119 items
11-02 01:00:20 WARNING  Downloading Photos ...
11-02 01:00:51 WARNING  Downloaded 119 Items, Failed 0, Already Downloaded 7363
11-02 01:00:51 WARNING  Indexing Shared (titled) Albums ...
11-02 01:00:51 WARNING  Indexed 0 Shared (titled) Albums
11-02 01:00:51 WARNING  Indexing Albums ...
11-02 01:00:52 WARNING  Indexed 0 Albums
11-02 01:00:52 WARNING  Downloading Photos ...
11-02 01:00:52 WARNING  Downloaded 119 Items, Failed 0, Already Downloaded 7482
11-02 01:00:52 WARNING  Creating album folder links to media ...
11-02 01:00:52 WARNING  Created 0 new album folder links
11-02 01:00:52 WARNING  Done.

So, it synced November too... but only on this account. It synced 119 photos after some days....

EDIT:
I removed all the folders and re-downloaded all the photos. When I downloaded them for the first time, all the photos were synced.

[gilesknap]

OK your trace file is helping. It looks like I only pass the date as a filter and not the time. Therefore I assume its using midnight of the day as the cutoff. Chances are that this is in the global timezone (same as GMT).

On what date did you last upload a photo. Without getting heavily back into the code I'm guessing that I chose to use date only but picked the date before the last run.

[daboynb]

On what date did you last upload a photo.

The last photo that the tool downloaded was from yesterday and was uploaded at 13:49. This was the last one of the day. (correct since I ran the tool today at 14.30)

As for the last photos uploaded on gphoto today that haven't been downloaded, they were uploaded today around 16:00 .

EDIT : I just ran the script at 17:59. It downloaded one photo out of the five taken today. However, I noticed something unusual: the synced photo is from the camera, while the other four are from WhatsApp.

This is downloaded

This is not downloaded

As you can see they were uploaded around 16:00 .

The old WhatsApp photos instead were downloaded successfully.

[gilesknap]

if you look at the trace file it says to search for files from start date 31st Oct 2023 to end date 3000. Therefore the Google API appears to be to blame.

Unfortunately all bugs against the Google API that I have posted have not received any response for years.

I'm not sure that reading through this issue that I'm clear on the detail. If you would like to write a concise description of the issue, including that trace output I will post it to Google. Or you could try it yourself at https://issuetracker.google.com/issues?q=status:open%20componentid:385336&s=created_time:desc

[daboynb]

Oh okay, thank you for your time. I will send a bug report through you link.

[SonnyAD]

Hello, I am experiencing the same problem. Does that mean the tool is not capable of making differential updates? Or is there a way to redo a sort of full scan?

Edit: seems that with a --rescan it just fixes the issue. So I think that's enough for me. Until Google fixes their API xD

[gilesknap]

Honestly, I think there is a lurking bug in the date conversion, probably in sqlite3. It's not serious as the incremental update just sometimes needs an extra half day or so to kick in.

Indeed I have just tried updating this project to work with python 3.12 and it turns out date conversion in sqlite3 is being deprecated. I'm not that keen to write my own (which is the recommendation) because a change in behaviour that is hard to test is only going to create more issues.

So for now I've skipped the deprecation warning and am deferring worrying about this until python 3.13 when a few more published workarounds will be available I hope.

Closing for now as won't fix. Workarounds are:

• wait a bit longer
• use --rescan