Behold! Dart edition!

[TheLastGimbus]

Hi @everyone!

Yes, I am alive. For a long time, to any bigger issue that showed up here, i responded "idk, code is 💩, will rewrite it some day"

So I did 🎉

Behold, Dart

GPTH, on new gpth_dart branch, is now written in Dart language - it allows way better organization than Python, and has beautiful tools for manipulating lists, async/await, etc - thanks to this, whole code is 160 lines less now (without test/), while doing things more stable, perdictable, and actually being documented

Missing stuff

Sadly, some functionalities are missing now:

• script does not write timestamps/geo data from jsons back to files - this may be sad in some cases (where people manually edited the location in app) - but i think it's better for the simplicity
• --skip-extras-harder was removed
• --divide-to-dates not implemented yet - but will be 👍

Stuff working better than ever

• finding duplicates/identifing albums is now stable (at least from my perspective because i finally know how it works)
  anyone interested can take a look: https://github.com/TheLastGimbus/GooglePhotosTakeoutHelper/blob/gpth_dart/lib/duplicate.dart
  • Inconsistency of "duplicate removal" + missing transperancy #110 i'm looking at you
  • sadly, finding albums is still purely theoretical, because we don't know what to do about it
• testing - lib/ is ~77% covered 💯
• performance - didn't test yet but expect a big difference 💯 - On my 1.2k takeout, gpth_dart did it in 17s, while old one took one minute
  • you also now have clear choice between coping files - which leaves input folder completely untouched (in contrast to python version 🙄), and moving them, which should be very good performance

Fixed issues

• json names being trimed to 51 chars prevented script from finding jsons - JSON naming too long? #8
  I marked this as "too hard to resolve" in my head, but now in Dart it's fixed, meaning finding correct json should be 99% success. And if not...
• nicely standardized date extraction - @tkuenzle in Make date extraction more modular #112 suggested that date extraction should be cleaner and modularized - python script was to shitty for that, but now take a look at

  GooglePhotosTakeoutHelper/bin/gpth.dart

  Lines 68 to 72 in 3695208

  	final dateExtractors = <DateTimeExtractor>[
  	jsonExtractor,
  	exifExtractor,
  	if (args['guess-from-name']) guessExtractor,
  	];

  - nice and clean!
  • added "guess from name" based on @hheimbuerger work <3
• HEIC Support for EXIF data #44 🎉 since https://pub.dev/packages/exif supports it
• overall consistency and sureness - if new issue raises, it should be actually possible to fix instead of crying about it

I need you ☝️

I would be very happy if you guys could help and test how new version works for you etc:

• @MrModest - does Don't reset original created time for un-guessed files. #128 still bother you with Dart?
• @jacobastern do you get more success in guessing dates, which would resolve Some MVIMG files are not organized correctly with the --divide-to-dates flag #105 ?
• @tkuenzle how do you like the modularity?
• @giantpune i think i could implement Strategies for timestamping more photos and videos. #102 now - what do you think?

Binaries are in releases for you ❤️

[hheimbuerger]

I might have been one of the people silently doubting you whenever you promised a future Dart rewrite! Obviously, an apology is due — I'm sorry! 😊

Jokes aside, good job and congratulations for completing this! 🎉

Allow me one question. But first, let me remind you that I have zero skin in the game, as I (thanks to this project!) have effectively migrated off of Google Photos as my primary storage and probably won't need it again:
For me, the exclusive reason to use this tool was to get the meta data and geotags written into the photos from the JSON files. My goal was to embed the metadata into the files, that's what I used the project for. If that feature's gone, what is the new purpose?

I hope you don't mind my critical question. Again, the tool does not need to suit me anymore. I'm just watching from the sidelines and wondering.

Best of luck and success!

[TheLastGimbus]

After some some reaserch:

1. Exif is hard
2. Writing exif is even harder

Even doing this (writing) in Rust/Go isn't so obvious, and i didn't found any "rock solid stable complete" library for this in them... if anyone finds such a library, please let me know....

So for now, I'll just wait if bigflood/dartexif#24 ever closes

Also, i may try to:

1. Check how many photos contain valid exif
2. If more than 15% does not, gpth will print a warning and link to guide how to use exiftool to write this stuff back 👍 🌈

[hheimbuerger]

Photos had stripped off all EXIF data from my photos

Oh that's terryfing. Did you use them on iPhone or something? How old where they?

No, they were mostly taken on various Android phones between 2010 and 2020. I might have uploaded a couple from older earlier years, too.

did I spot check just the wrong photos

mayyyyy be the case 🤨

no idea how much has changed since

Well my takeouts are not stripped at all, but if there are any people/takeouts as yours, this must be implemented 💯

I'm seriously questioning my memory, and/or my approach at the time. My photo archive cleanup was a massive undertaking with hundreds of steps, and obviously lots of mistakes. I'm really starting to believe that I must have misread some signal in some app somewhere, to lead me to believe that they're all stripped into the JSON.
Also, I was probably going by the geolocation, because that's the EXIF data point I care about the most. However, for the longest time this was off by default in photo apps, and I sometimes didn't remember to turn it on in time. So when looking at a massive Google Takeout, it's a bit hard to tell which photos are supposed to have geolocation info in them, but it's stripped, and which ones are just lacking the information in general.

Or maybe I just read so somewhere? I did some quick googling now, and found a lot of articles and posts from the time all (quite likely erroneously) indicating that "Google Takeout strips all EXIF tags", e.g.

• "My photos are losing their EXIF data courtesy of Google!": https://photo.stackexchange.com/questions/130092/google-is-wiping-my-exif-data-gives-it-back-in-a-json-file-any-already-built-t
• "My concern is that i’ve see posts of people using the Google Takeout service remove all the metadata upon export": https://www.reddit.com/r/googlephotos/comments/qagdfo/google_takeout_didnt_remove_metadata_was_it/
• "This means all photos are stripped of their EXIF data": https://www.xyplorer.com/xyfc/viewtopic.php?t=21351
• "Google Photos strips your photos off all their Metadata and stores it separately": https://medium.com/@ty2/the-painful-experience-of-leaving-google-photos-914d9c63d4b2

[TheLastGimbus]

a lot of articles and posts

Truly terrifing ;_; Stripping exifs is a nightmare of this century... I will try to get this anyhow working in gpth...

By the way, I'm thinking of making a gallery app for android in Flutter... because for now, only real good one is Google Photos, i would like to have something with powers of photos, but without any cloud etc.

It would work locally, let you easily set their exif, and search them by location etc - it would leave the job of syncing out to you (Syncthing, Nextcloud/whatever)

Would you be interested?

[palijn]

I would personally not be interested in yet another photo browsing app on my phone. The integrated Oppo app works well for local files,and displays EXIF data just fine, I simply never search my pictures and rely on chronological view to browse them. And finally, my library size is vastly larger than the local storage of my phone, which is why I uploaded pictures on Google Photos in the first place, and now on my Synology NAS.

[hheimbuerger]

By the way, I'm thinking of making a gallery app for android in Flutter... because for now, only real good one is Google Photos, i would like to have something with powers of photos, but without any cloud etc.

Would you be interested?

I don't have the majority of photos on my phone either, so I pretty much never use a gallery app on my phone. Sometimes I need to review the last ten photos taken or so, then I'm using Google Photos, for that. But I never check the EXIF and would never edit anything on my phone.

I loved the cloud feature of Google Photos (having all my photos 'virtually available', even those not currently physically stored on the phone), but I had other misgivings about its photo management approach, that's why I unfortunately had to migrate away from it.

One day, I'll upgrade my Synology to DSM 7, and maybe then the Synology Photos app will be a good replacement. Until then, I simply don't have access to my photos on the road.

[palijn]

Chiming in (you already answered on Reddit) : a lot of my photos in the Takeout archive have bogus EXIF date, however a cursory check let me believe that the dates in the JSON file are actually valid. So, my use case is exactly this one.
It's been blocking my migration so far, so if you ever find a way to implement that, I'd be VERY happy !

[TheLastGimbus]

If you want this hard, just use https://exiftool.org/ :

exitool.exe "-EXIF:DateTimeOriginal<FileModifyDate" folder/with/photos/

This will write in exif time dates based on their file lastModified, which script does set

[palijn]

Just to be sure : did the python version write the EXIF data back into the files ??

[TheLastGimbus]

Yes it did. It probably had few unknown bugs, and was much slower, but yes it did

But, looks like brendan-duncan/image#457 may get resolved soon, so we will have this too in dart edition 🎉

[palijn]

Oh wow ! I'll keep waiting then !

[palijn]

Resurrecting this post since I saw you mentioning your're targeting writing EXIF data !
I'm not confident enough yet to use exiftool properly, soooo I haven't tried your suggestion yet. You're definitely more expert on that one.

I'll definitely pop Champagne when you can modify EXIF data !
That's about the only thing that prevents me from using your tool right now.

Synology Photos organizes its timeline with "photoTakenTime" EXIF data (that's the json key and I'm absolutely not sure of the EXIF equivalent). And Synology Photos blissfully ignores file modification time, which I understand your tool modifies, WHEN there is EXIF data available, however bogus it was made by Takeout. (It WOULD use the file's modification time IF there were no EXIF data available, though.)

[NILOOFAR654]

no no no

…

On Mon, Jan 2, 2023 at 10:28 PM palijn ***@***.***> wrote: I would personally not be interested in yet another photo browsing app on my phone. The integrated Oppo app works well for local files,and displays EXIF data just fine, I simply never search my pictures and rely on chronological view to browse them. And finally, my library size is vastly larger than the local storage of my phone, which is why I uploaded pictures on Google Photos in the first place, and now on my Synology NAS. — Reply to this email directly, view it on GitHub <#133 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQUFUKATLHN43QT7Q4YMVODWQMQOPANCNFSM6AAAAAATDK4ZK4> . You are receiving this because you were mentioned.Message ID: <TheLastGimbus/GooglePhotosTakeoutHelper/repo-discussions/133/comments/4574830 @github.com>

[TheLastGimbus]

You mean you wouldn't use it?

[Sparo099]

???can u help you guys

[TheLastGimbus]

Sure we can!

[NILOOFAR654]

No, I don't use it😶😶

…

On Thu, Jan 5, 2023 at 12:48 AM Mateusz Soszyński ***@***.***> wrote: You mean you wouldn't use it? — Reply to this email directly, view it on GitHub <#133 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQUFUKC33I7Y4B4ZWQUJBXTWQXSI5ANCNFSM6AAAAAATDK4ZK4> . You are receiving this because you were mentioned.Message ID: <TheLastGimbus/GooglePhotosTakeoutHelper/repo-discussions/133/comments/4595733 @github.com>

[TheLastGimbus]

✨ Interactive mode ✨

Hey guys! Maybe it's not as seen on Issues here, but i was getting TON of emails like "how to use cmd" "how should i unzip the takeout", etc

So now, gpth can be used just by double-clicking the .exe - it will pop up file pickers, unzip the takeouts for you, and tell you whats happening

I'm super proud of this, and tested as much as i could, but nothing is better than edge-cases 🎉 - could some of you guys test if new version works for you and maybe give some feedback about it?

@hheimbuerger @palijn @bitsondatadev (btw whats up man?) @roirraWedorehT @hansdg1 @isabelpothos