Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No folders named like the README instructions? #30

Closed
hnlkaitan opened this issue Dec 10, 2020 · 27 comments
Closed

No folders named like the README instructions? #30

hnlkaitan opened this issue Dec 10, 2020 · 27 comments

Comments

@hnlkaitan
Copy link

Hi, thought I'd give this a shot, however I noticed that after using takeout, I don't have any folders in the YYYY-MM-DD format or anything like that. Each takeout folder is mostly comprised of albums, with the closest being Photos from YYYY. I'm not sure if this is because it originally synced with Google Drive, or what.

And also, might be unrelated, but if using macOS, can I just copy-merge the folders so I have ./Takeout/Google\ Photos/<everything copy-merged here> or should it each Takeout folder just be put into a folder (no merge) like ./folder/<Takeout [1-N]>

@TheLastGimbus
Copy link
Owner

TheLastGimbus commented Dec 10, 2020

or should it each Takeout folder just be put into a folder

Hm, you can probably do any of this, it should work anyway

Can you describe how you used your phots? Did you just use the app, or did you upload them by hand from some other camera, or what?

Also, ls of the folders would help, just put it in the code block please

@JoLander
Copy link

Hi,
I just stumbled over this tool and want to try as well.
After running a takeout of all photos from Google I found the same strange folder structure.
It seems that all photos are ordered in folders called 'Photos from YYYY' for every calender year.
In addition, I have an extra folder for each album I created. These folders seems to have copies of the pictures already present in the 'Photos from YYYY' folders.
There are no folders at all named by a specific date.

@TheLastGimbus
Copy link
Owner

Jezus fucking Christ, Google...

@hnlkaitan
Copy link
Author

hnlkaitan commented Dec 10, 2020

Actually, now that you mention it, I believe that some time in the past, I did upload my entire photo library to Google Photos library (some time between 2016-2018, since I thought it'd be nice to have my entire library in the cloud). So everything before that time was uploaded probably through the website or Google Drive (I don't remember at this point, honestly), and then almost everything after that was through the Google Photos app.

Directory tree that I have:

/Volumes
  |--San Disk Extreme
      |--Takeout
          |--Takeout 1
          |--Takeout 2
          |--...
          |--Takeout N

Here's the ls:

✓ /Volumes/SanDisk Extreme/Takeout % ls Takeout\ 1/Google\ Photos 
<OMITTED>/
<...>/
Archive/
<OMITTED>/
<...>/
Photos from 2015/
Photos from 2017/
<OMITTED>/
<...>/
print-subscriptions.json
shared_album_comments.json

Removed album names, for privacy. But all the photos have that general structure, with custom albums, and then the Photos from YYYY (inside the ./Photos from 2015/ folder from Takeout 1 is all .json files, although I believe their corresponding image files might be in another Takeout <N>/Photos from 2015/folder, as I have several of those Photos from 2015 with fragments of the (I'm assuming) complete folder, and the same goes for other years

EDIT: Just realized the folder structure might be of importance so this is Takeout 1/Google Photos/Photos from 2015/

042.AVI.json                             DSCN0389.AVI.json                        IMG_1007.mov.json
056.AVI.json                             DSCN0390.AVI.json                        IMG_1077.MOV.json
0en9A6V.jpg.json                         DSCN0391.AVI.json                        IMG_1292.MOV.json
1449730175869.jpg.json                   DSCN0392.AVI.json                        IMG_1300.MOV.json
1450308274709.jpg.json                   DSCN0393.AVI.json                        IMG_1301.mov.json
2.jpg.json                               DSCN0397.AVI.json                        IMG_1302.mov.json
20151201_174045.jpg.json                 DSCN0398.AVI.json                        IMG_1303.mov.json
20151201_174050.jpg.json                 DSCN0399.AVI.json                        IMG_1304.MOV.json
20151201_174100.jpg.json                 DSCN0400.AVI.json                        IMG_1305.mov.json
20151202_070438.jpg.json                 DSCN0402.AVI.json                        IMG_1306.mov.json
20151202_070513.jpg.json                 DSCN0403.AVI.json                        IMG_1307.mov.json
20151207_063059.jpg.json                 DSCN0404.AVI.json                        IMG_1308.mov.json
20151207_183916.jpg.json                 DSCN0405.AVI.json                        IMG_1310.mov.json
20151207_192634.jpg.json                 DSCN0406.AVI.json                        IMG_1311.mov.json
20151207_192654.jpg.json                 DSCN0407.AVI.json                        IMG_1360.mov.json
20151209_122553.jpg.json                 DSCN0408.AVI.json                        IMG_1932.MOV.json
20151209_134859.jpg.json                 DSCN0409.AVI.json                        IMG_1998.MOV.json
20151209_141612_001.jpg.json             DSCN0411.AVI.json                        IMG_2006.MOV.json
20151210_145127.jpg.json                 DSCN0412.AVI.json                        IMG_2007.MOV.json
20151215_151522.jpg.json                 DSCN0414.AVI.json                        IMG_2009.MOV.json
20151215_155804.jpg.json                 DSCN0415.AVI.json                        IMG_2010.MOV.json
20151215_155806.jpg.json                 DSCN0416.AVI.json                        IMG_20151201_231932.jpg.json
20151215_155950.jpg.json                 DSCN0418.AVI.json                        IMG_2346.MOV.json
20151215_155954.jpg.json                 DSCN0419.AVI.json                        IMG_2353.MOV.json
20151215_160047.jpg.json                 DSCN0420.AVI.json                        IMG_2354.MOV.json
20151215_170328_001.jpg.json             DSCN0421.AVI.json                        IMG_2371.MOV.json
20151215_170330.jpg.json                 DSCN0422.AVI.json                        IMG_2553.MOV.json
20151216_130842.jpg.json                 DSCN0423.AVI.json                        IMG_3079.mov.json
20151216_130843.jpg.json                 DSCN0432.AVI.json                        IMG_3208.MOV.json
20151216_130845.jpg.json                 DSCN0434.AVI.json                        IMG_3883.MOV.json
20151216_130846.jpg.json                 DSCN0439.AVI.json                        IMG_3935.MOV.json
20151216_173948.jpg.json                 DSCN0911.AVI.json                        IMG_3936.MOV.json
20151217_142042.jpg.json                 DSCN0913.AVI.json                        IMG_3937.MOV.json
20151218_091016.jpg.json                 DSCN0914.AVI.json                        IMG_4170.MOV.json
20151218_193834.jpg.json                 DSCN0916.AVI.json                        IMG_4262.MOV.json
20151218_204707.jpg.json                 DSCN0917.AVI.json                        IMG_4264.MOV.json
20151219_180518.jpg.json                 DSCN0918.AVI.json                        IMG_4730.MOV.json
20151220_184842.jpg.json                 DSCN0919.AVI.json                        IMG_4902.MOV.json
20151220_191040.jpg.json                 DSCN0922.AVI.json                        IMG_4915.MOV.json
20151220_191603.jpg.json                 Dark-City-640x400.png.json               IMG_4924.MOV.json
20151220_192151.jpg.json                 Hl5jvZ8.jpg.json                         IMG_4925.MOV.json
20151220_200754.jpg.json                 Hv8AaGj.jpg.json                         IMG_4926.MOV.json
20151220_201054.jpg.json                 IMG_0001.JPG.json                        IMG_4927.MOV.json
20151220_201103.jpg.json                 IMG_0001.PNG.json                        IMG_4928.MOV.json
20151221_203554.jpg.json                 IMG_0002.PNG.json                        IMG_4929.MOV.json
20151221_203612_001.jpg.json             IMG_0003.PNG(1).json                     IMG_4930.MOV.json
20151221_203613.jpg.json                 IMG_0004.PNG.json                        IMG_4939.MOV.json
20151221_203614.jpg.json                 IMG_0005.PNG.json                        IMG_4942.MOV.json
20151221_203624.jpg.json                 IMG_0007.MOV.json                        IMG_5055.MOV.json
20151221_203625.jpg.json                 IMG_0079.MOV.json                        IMG_5180.MOV.json
20151221_203743.jpg.json                 IMG_0090.MOV.json                        IMG_5181.MOV.json
20151221_233313.jpg.json                 IMG_0113.MOV.json                        IMG_5436.MOV.json
20151222_120047.jpg.json                 IMG_0119.MOV.json                        IMG_5457.MOV.json
20151222_120056.jpg.json                 IMG_0120.MOV.json                        IMG_5469.MOV.json
20151225_111700.jpg.json                 IMG_0121.MOV.json                        LoTxdZo.jpg.json
20151225_130712.jpg.json                 IMG_0122.MOV.json                        OBpkBLR.jpg.json
20151228_163402.jpg.json                 IMG_0123.MOV.json                        Screenshot_2015-11-30-22-39-23.png.json
20151228_163405.jpg.json                 IMG_0124.MOV.json                        Screenshot_2015-12-07-12-40-52.png.json
20151230_114058.mp4.json                 IMG_0125.MOV.json                        Screenshot_2015-12-07-12-42-09.png.json
20151231_112608.jpg.json                 IMG_0126.MOV.json                        Screenshot_2015-12-07-19-42-19.png.json
3.JPG.json                               IMG_0127.MOV.json                        Screenshot_2015-12-07-20-23-23.png.json
3.jpg                                    IMG_0147.MOV.json                        Screenshot_2015-12-15-18-06-56.png.json
4IWLvtWl.jpg.json                        IMG_0165.MOV.json                        Screenshot_2015-12-15-23-29-11.png.json
4RBG0gY.jpg.json                         IMG_0166.MOV.json                        Screenshot_2015-12-15-23-30-57.png.json
4nH22Gn.jpg.json                         IMG_0177.MOV.json                        Screenshot_2015-12-16-09-37-25.png.json
A2iqzVL.jpg.json                         IMG_0186.MOV.json                        Screenshot_2015-12-18-09-32-47.png.json
DSCN0375.AVI.json                        IMG_0393.MOV.json                        Screenshot_2015-12-28-10-53-01.png.json
DSCN0378.AVI.json                        IMG_0481.MOV.json                        Screenshot_2015-12-31-13-17-10.png.json
DSCN0379.AVI.json                        IMG_0482.MOV.json                        b4oCRDq.jpg.json
DSCN0380.AVI.json                        IMG_0483.MOV.json                        dqPRALY.jpg.json
DSCN0381.AVI.json                        IMG_0536.MOV.json                        esfhuVv.jpg.json
DSCN0384.AVI.json                        IMG_0669.mov.json                        iSmdK46.jpg.json
DSCN0385.AVI.json                        IMG_0683.mp4.json                        itdEffT.jpg.json

@JoLander
Copy link

I just extractes all my takeout chunks into one folder.
As I said before, the specific album folders seems to contains copies of the files already present in the 'Photos from YYYY' folder.
It seems that they can safely get deleted to avoid duplicates (on the other hand it could be nice if the tool would also recreate the albums as folders. But this seems to be already in progress).
There are some special folders which do not contain duplicates within the 'Photos from YYYY' folder.
These are the 'Archive' and the 'Trash' folders. It seems that archived or deleted pictures are really 'moved' into the specific folders instead of being copied as in the albums.

@varnav
Copy link

varnav commented Dec 13, 2020

Yup Google now names folders "Photos from YYYY". Tool says:

Wrong folder name: Photos:fro 12:00:00
You probably forgot to remove 'album folders' from your takeout folder

@fdterr
Copy link

fdterr commented Dec 14, 2020

Yup Google now names folders "Photos from YYYY". Tool says:

Wrong folder name: Photos:fro 12:00:00
You probably forgot to remove 'album folders' from your takeout folder

+1 also having this issue with that exact error

@cristian-milea
Copy link

Hi, does it work if I manually rename the folders from "Photos from 2016" to "2016-01-01" ?

@TheLastGimbus
Copy link
Owner

Huh... that's a very interesting question

Generally, this script uses folder name as a last chance when picture has no exif data nor .json was found. So if that happens, you will have photos with that date - you will have them listed in final stats file 👍

Tell me how it went if you do that

@mtthidoteu
Copy link

Hey guys! What is the status on this issue then?
Cheers!

@TheLastGimbus
Copy link
Owner

Idk honestly - but it will probably quickly be solved - after Christmas 🎄

@TheLastGimbus
Copy link
Owner

Hey @ everyone! Looks like @bitsondatadev almosted finished #36 - now, I need to test this before merging and making new version

Could you pull from his fork: https://github.com/bitsondatadev/GooglePhotosTakeoutHelper/tree/handle-albums-with-hash - and test if your "new takeouts" work with it? I would be really grateful!

@popy2k14
Copy link

popy2k14 commented Jan 5, 2021

Also looking forward for support of the new structure.
thx a lot.

@TheLastGimbus
Copy link
Owner

@popy2k14 if you are looking forward, help us in testing it 😉

Branch above should fully work now - just checkout and test if it works for you

@varnav
Copy link

varnav commented Jan 5, 2021

I tested it under windows.
Worked for me. A lot of exif and json not found errors but I guess it's ok.

Correct "Created" date in windows is not set, please look at my tool how I managed that.

@popy2k14
Copy link

popy2k14 commented Jan 8, 2021

Test under Windows 10 Pro 20H2 build with newest python 3.9.1 from MS store.
Sadly it does'nt work for me and quits with an exception:

D:\Temp\#google_fotos\GooglePhotosTakeoutHelper-handle-albums-with-hash>python3 -m google_photos_takeout_helper -i D:\Temp\#google_fotos\Stefi -o D:\Temp\#google_fotos\Stefi_out --divide-to-dates
Heeeere we go!
=====================
Fixing files metadata and creation dates...
=====================
D:\Temp\#google_fotos\Stefi\Takeout\Google Fotos\Photos from 1990\11406828_10155658794900057_2219490452348186131_.jpg
Error setting creation date from string:
setctime() missing 2 required positional arguments: 'filepath' and 'timestamp'
Wrong date format in exif!
1990:01:01 12:00:00
does not match '%Y:%m:%d %H:%M:%S'
Error setting creation date from string:
setctime() missing 2 required positional arguments: 'filepath' and 'timestamp'
Wrong date format in exif!
1990:01:01 12:00:00
does not match '%Y:%m:%d %H:%M:%S'
Error setting creation date from string:
setctime() missing 2 required positional arguments: 'filepath' and 'timestamp'
Wrong date format in exif!
1990:01:01 12:00:00
does not match '%Y:%m:%d %H:%M:%S'
No creation date found in exif!
Couldn't find json for file
Last chance, coping folder meta as date...
Couldn't pull datetime from album meta
ERROR! There was literally no option to set date!!!
TODO: We should do something about this - move it to some separate folder, or write it down in another .txt file...
D:\Temp\#google_fotos\Stefi\Takeout\Google Fotos\Photos from 2000\1.jpg
Error setting creation date from string:
setctime() missing 2 required positional arguments: 'filepath' and 'timestamp'
Wrong date format in exif!
2000:01:01 12:00:00
does not match '%Y:%m:%d %H:%M:%S'
Error setting creation date from string:
setctime() missing 2 required positional arguments: 'filepath' and 'timestamp'
Wrong date format in exif!
2000:01:01 12:00:00
does not match '%Y:%m:%d %H:%M:%S'
Error setting creation date from string:
setctime() missing 2 required positional arguments: 'filepath' and 'timestamp'
Wrong date format in exif!
2000:01:01 12:00:00
does not match '%Y:%m:%d %H:%M:%S'
No creation date found in exif!
Error setting creation date from string:
setctime() missing 2 required positional arguments: 'filepath' and 'timestamp'
Traceback (most recent call last):
  File "D:\Temp\#google_fotos\GooglePhotosTakeoutHelper-handle-albums-with-hash\google_photos_takeout_helper\__main__.py", line 352, in set_creation_date_from_str
    _windoza_setctime.setctime()
TypeError: setctime() missing 2 required positional arguments: 'filepath' and 'timestamp'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "D:\Temp\#google_fotos\GooglePhotosTakeoutHelper-handle-albums-with-hash\google_photos_takeout_helper\__main__.py", line 666, in <module>
    main()
  File "D:\Temp\#google_fotos\GooglePhotosTakeoutHelper-handle-albums-with-hash\google_photos_takeout_helper\__main__.py", line 587, in main
    for_all_files_recursive(
  File "D:\Temp\#google_fotos\GooglePhotosTakeoutHelper-handle-albums-with-hash\google_photos_takeout_helper\__main__.py", line 130, in for_all_files_recursive
    file_function(file)
  File "D:\Temp\#google_fotos\GooglePhotosTakeoutHelper-handle-albums-with-hash\google_photos_takeout_helper\__main__.py", line 523, in fix_metadata
    set_creation_date_from_str(file, date)
  File "D:\Temp\#google_fotos\GooglePhotosTakeoutHelper-handle-albums-with-hash\google_photos_takeout_helper\__main__.py", line 356, in set_creation_date_from_str
    raise ValueError(f"Error setting creation date from string: {str_datetime}")
ValueError: Error setting creation date from string: 2000:01:01 12:00:00

any hints?

thx

@TheLastGimbus
Copy link
Owner

python3 -m google_photos_takeout_helper

You just used official version of script that is on pypi

To use our WIP code, you need to:

git clone https://github.com/bitsondatadev/GooglePhotosTakeoutHelper/ takeout-test
cd takeout-test
git checkout --track origin/handle-albums-with-hash
python3 google_photos_takeout_helper/__main__.py -i [INPUT] -o [OUTPUT]

@popy2k14
Copy link

popy2k14 commented Jan 9, 2021

I was using the new branch from the above link. Sadly, also with your approach, it doesnt work:

Microsoft Windows [Version 10.0.19042.685]
(c) 2020 Microsoft Corporation. Alle Rechte vorbehalten.

D:\Temp#google_fotos>git clone https://github.com/bitsondatadev/GooglePhotosTakeoutHelper/ takeout-test
Cloning into 'takeout-test'...
remote: Enumerating objects: 46, done.
remote: Counting objects: 100% (46/46), done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 442 (delta 18), reused 30 (delta 10), pack-reused 396R
Receiving objects: 100% (442/442), 106.43 KiB | 864.00 KiB/s, done.
Resolving deltas: 100% (220/220), done.

D:\Temp#google_fotos>cd takeout-test

D:\Temp#google_fotos\takeout-test>git checkout --track origin/handle-albums-with-hash
Switched to a new branch 'handle-albums-with-hash'
Branch 'handle-albums-with-hash' set up to track remote branch 'handle-albums-with-hash' from 'origin'.

D:\Temp#google_fotos\takeout-test>python3 google_photos_takeout_helper/main.py -i D:\Temp#google_fotos\Stefi -o D:\Temp#google_fotos\Stefi_out
Heeeere we go!

Fixing files metadata and creation dates...

D:\Temp#google_fotos\Stefi\Takeout\Google Fotos\Photos from 1990\11406828_10155658794900057_2219490452348186131_.jpg
Error setting creation date from string:
setctime() missing 2 required positional arguments: 'filepath' and 'timestamp'
Wrong date format in exif!
1990:01:01 12:00:00
does not match '%Y:%m:%d %H:%M:%S'
Error setting creation date from string:
setctime() missing 2 required positional arguments: 'filepath' and 'timestamp'
Wrong date format in exif!
1990:01:01 12:00:00
does not match '%Y:%m:%d %H:%M:%S'
Error setting creation date from string:
setctime() missing 2 required positional arguments: 'filepath' and 'timestamp'
Wrong date format in exif!
1990:01:01 12:00:00
does not match '%Y:%m:%d %H:%M:%S'
No creation date found in exif!
Couldn't find json for file
Last chance, coping folder meta as date...
Couldn't pull datetime from album meta
ERROR! There was literally no option to set date!!!
TODO: We should do something about this - move it to some separate folder, or write it down in another .txt file...
D:\Temp#google_fotos\Stefi\Takeout\Google Fotos\Photos from 2000\1.jpg
Error setting creation date from string:
setctime() missing 2 required positional arguments: 'filepath' and 'timestamp'
Wrong date format in exif!
2000:01:01 12:00:00
does not match '%Y:%m:%d %H:%M:%S'
Error setting creation date from string:
setctime() missing 2 required positional arguments: 'filepath' and 'timestamp'
Wrong date format in exif!
2000:01:01 12:00:00
does not match '%Y:%m:%d %H:%M:%S'
Error setting creation date from string:
setctime() missing 2 required positional arguments: 'filepath' and 'timestamp'
Wrong date format in exif!
2000:01:01 12:00:00
does not match '%Y:%m:%d %H:%M:%S'
No creation date found in exif!
Error setting creation date from string:
setctime() missing 2 required positional arguments: 'filepath' and 'timestamp'
Traceback (most recent call last):
File "D:\Temp#google_fotos\takeout-test\google_photos_takeout_helper_main_.py", line 352, in set_creation_date_from_str
_windoza_setctime.setctime()
TypeError: setctime() missing 2 required positional arguments: 'filepath' and 'timestamp'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\Temp#google_fotos\takeout-test\google_photos_takeout_helper_main_.py", line 666, in
main()
File "D:\Temp#google_fotos\takeout-test\google_photos_takeout_helper_main_.py", line 587, in main
for_all_files_recursive(
File "D:\Temp#google_fotos\takeout-test\google_photos_takeout_helper_main_.py", line 130, in for_all_files_recursive
file_function(file)
File "D:\Temp#google_fotos\takeout-test\google_photos_takeout_helper_main_.py", line 523, in fix_metadata
set_creation_date_from_str(file, date)
File "D:\Temp#google_fotos\takeout-test\google_photos_takeout_helper_main_.py", line 356, in set_creation_date_from_str
raise ValueError(f"Error setting creation date from string: {str_datetime}")
ValueError: Error setting creation date from string: 2000:01:01 12:00:00

D:\Temp#google_fotos\takeout-test>

@bitsondatadev Any hints?

Seems time converting fails.

thx

@TheLastGimbus
Copy link
Owner

Wrong date format in exif!
1990:01:01 12:00:00
does not match '%Y:%m:%d %H:%M:%S'

How the hell does 1990:01:01 12:00:00 not match Y-M-D H-M-S??

🤷

@popy2k14
Copy link

popy2k14 commented Jan 9, 2021

Thats the question :-)

@popy2k14
Copy link

popy2k14 commented Jan 10, 2021

I have found the issue.
Change line 352 from:

_windoza_setctime.setctime()

to

_windoza_setctime.setctime(file, timestamp)

And it works.

Seems there was the two arguments file & timestamp missing for the "setctime" function.
This was throwing the exception JUST on windows machines.

@TheLastGimbus
Copy link
Owner

TheLastGimbus commented Jan 10, 2021

Oh, I had to be really asleep when I was writing this xD

fixed, now try it

Ps. you need to

git pull origin handle-albums-with-hash

@popy2k14
Copy link

Its running now, but i got very often the message that no json was found.

Output of script:

D:\Temp\#google_fotos\Stefi\Takeout\Google Fotos\Photos from 2018\DSC05418(1).jpg Couldn't find json for file

After a look at the files if found this:

image

There is an (1) at another position in the filename.
It seems the json is the correct one.

Will this not be taken into account?

@popy2k14
Copy link

To overcome this i have done the following workaround on my folder with WSL:

find . -depth -exec rename 's/\(1\)//g' {} +

This removes the (1) and running the script afterwards sets the exif data from json correctly.

Would be nice if this would be confirmed and integrated into the script.

@TheLastGimbus
Copy link
Owner

It's okay, we have plans for better "json finding" - most important - does it generally work?

@popy2k14
Copy link

Yeah I think it's necessary to a better way to find the json file.

Generally: it works now.

@TheLastGimbus
Copy link
Owner

TheLastGimbus commented Jan 10, 2021

🎉

Will push new version to PyPi very soon

pip install google-photos-takeout-helper==2.0.0rc1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants