Flow to convert old MPGs in Photos to new MOV

[oPromessa]

Hello osxphotos users!

I wanted to convert all my previously loaded MPG old movies on Photos to MOV format. So that I can play with metadata which is not possible on MPG files.

So I was planning on:

1. List the files: osxphotos query
2. Convert the files (my files are all referenced) on the file system: ffmpeg
3. Set date/time and gps location related metadata to converted file: exiftool
4. As for keywords. I can assign keywords with exiftool to the file but once imported Photos does not recognize them. And osxphotos import doesn't assign keyword also. I'm trying activescript. Don't know if Python import osxphotos code can help out.
5. Also have to assign the converted file to the previously assigned album: possibly via python import osxphotos code
6. And finally delete the .MPG from Photos

So far I've managed steps 1, 2 and 3. For Steps 4, 5 and 6 any recommendations/comments?

[RhetTbull]

Edit: Apparently mpeg doesn't support metadata so writing with --exiftool on export won't preserve the keywords. Updated to use a sidecar file to then apply the metadata to the converted .mov file.

Edit 2: fixed sidecar filename

You should be able to do this all with osxphotos. I'd use an actual export to a temp folder instead of a query. The following commands are from memory so might need some slight tweaking. I suggest reading the docs to learn what each option does.

1., 2., 3. osxphotos export /path/to/export --uti public.mpeg --exiftool --directory "{folder_album}" --sidecar exiftool --post-command exported "ffmpeg --options -i {filepath|shell_quote} {shell_quote,{filepath.parent}/{filepath.stem}.mov}" --post-command "exiftool -overwrite_original -json={shell_quote,{filepath}.json} {shell_quote,{filepath.parent}/{filepath.stem}.mov}" --add-exported-to-album Exported_MPG_movies
4. 5. osxphotos import /path/to/export --glob "*.mov" --exiftool --walk --album "{filepath.parent}" --relative-to "/path/to/export" --split-folder "/"
6. Go to the Album "Exported_MPG_movies" which will have been created by the first command then delete all the files. (Select then Command + Delete)

I don't use ffmpeg so you'll need to adjust the options but using --post-command allows you to run a command on the newly exported photo.

[RhetTbull]

Not sure what to call it but it might be good to have a new osxphotos command that exports a photo, processes it with user provided command, then reimports it, re-applies all previous metadata and albums, and deletes the original. This would be useful for this and other use cases like fixing file extensions (#382).

[RhetTbull]

This will only work if exiftool can write metadata to MPEG. If it can't (and maybe that's the reason import wasn't importing keywords) then this will need some custom python code.

[RhetTbull]

Edit: The following will work but it just occurred to me you said your library used referenced files which means this won't work for your specific use case and the converted files would be in the export directory. You'd have to move them to the reference library path before the import. This could be done with another --post-command exported "mv ..." command. Left as an exercise for the reader.

Did some testing. Previous answer is close but unfortunately the exiftool JSON sidecars encode the SourceFile field which causes exiftool to reject the metadata when applied to a new source file (.mov vs .mpeg). However, looks like we can get around this with XMP sidecar. The two commands appear to accomplish all your steps with exception of #6 which you can accomplish by selecting all photos in the Exported_MPG_movies album.

Create a TOML file called export_mpg.toml with these contents (adjust the ffmpeg command as needed)

[export]
add_exported_to_album = "Exported_MPG_movies"
directory = "{folder_album}"
post_command = [ [ "exported", "ffmpeg -i {filepath|shell_quote} {shell_quote,{filepath.parent}/{filepath.stem}.mov}",], [ "exported", "exiftool -overwrite_original -tagsfromfile {shell_quote,{filepath}.xmp} -all:all {shell_quote,{filepath.parent}/{filepath.stem}.mov}",],]
report = "exported_mpg_files_{today}.csv"
sidecar = [ "xmp",]
uti = "public.mpeg"
verbose = true
timestamp = true

Run the following command (substituting your export folder)

osxphotos export ~/Desktop/export --load-config export_mpg.toml

Then run the following command to import:

osxphotos import ~/Desktop/export --glob "*.mov" --exiftool --walk --album "{filepath.parent}" --relative-to ~/Desktop/export --split-folder "/" --verbose --timestamp

This worked in my limited test. Recommend you try with a test library until you're sure it's working to your liking.

[RhetTbull]

note this will also export .mp4 files which are also mpegs so you may also need to further limit the export criteria if you really only want .mpg files.

[oPromessa]

Thanks so much. Yes the mpg vs mp4 tag I can filter out.

Super. Will give it a try later tonight.

[oPromessa]

Almost perfect!

Only glitch is that it did not recognise the GPS info on import even though it has been included on the file with exiftool.
Will do a bit more of testing with more files but wanted you to know the rest is working great!

$ osxphotos import export --no-progress --verbose --glob "*.mov" --exiftool --walk --album "{filepath.parent}" --relative-to "export" --split-folder "/"
Importing /Users/Shared/Pictures/Export/convert_mpg/export/iPhoto Events/(...)/MOV01046.mov
Imported MOV01046.mov with UUID F48B826F-54AB-4687-8CF2-7D95902CAD16
Setting metadata and location from EXIF for MOV01046.mov
Set metadata for MOV01046.mov:
title='', description='', keywords=['XXXXXXX']
No location to set for MOV01046.mov
Adding photo MOV01046.mov to 1 album
Adding photo MOV01046.mov to album iPhoto Events/(...)
Added MOV01046.mov (F48B826F-54AB-4687-8CF2-7D95902CAD16) to album iPhoto Events/(...)
Done: imported 1 file, 0 errors, 

$ exiftool -G -u -a "/Users/Shared/Pictures/Export/convert_mpg/export/iPhoto Events/(...)/MOV01046.mov" | grep GPS
[XMP]           GPS Latitude                    : XX.633301 N
[XMP]           GPS Longitude                   : YY.900000 W
[Composite]     GPS Latitude Ref                : North
[Composite]     GPS Longitude Ref               : West
[Composite]     GPS Position                    : XX.633301 N, YY.900000 W

[oPromessa]

Found that on the import osxphotos is looking at EXIF:GPSLatitude which is not available on MOV files.

osxphotos/osxphotos/cli/import_cli.py

Line 315 in 8caee5a

if latitude := metadata.get("EXIF:GPSLatitude"):

What is available is under XMP

[XMP]           GPS Latitude                    : XX.633301 N
[XMP]           GPS Longitude                   : YY.900000 W

So I'm putting up a code change for: #912

        if latitude is None:
            latitude = metadata.get("XMP:GPSLatitude"):
        if longitude := metadata.get("EXIF:GPSLongitude"):
            longitude = float(longitude)
            longitude_ref = metadata.get("EXIF:GPSLongitudeRef")
            if longitude_ref == "W":
                longitude = -longitude
            elif longitude_ref != "E":
                longitude = None
        if longitude is None:
            longitude = metadata.get("XMP:GPSLongitude"):

[oPromessa]

Convert MPG to MOV | Applicable to other conversions on Photos

Flow: Find .MPG files (hidden or not hidden) in Photos | export & convert | import | delete

• Export with Keywords and location to XMP sidecar
• Hardlink export option seems to work even with referenced files. Saves some space and disk processing on export by referencing the original files.
  • Maybe on edited files the actual file is exported: skip_original_if_edited = true
• Use ffmpeg to convert MPG to MOV files
• Use exiftool to copy attributes and set all dates including QuickTime:ContentCreateDate
• Set alldates to Original

Export

• Note: This will include hidden pics, use the --not-hidden flag to bypass them or the --hidden flag to slect just the hidden pics.

mkdir export;
osxphotos export export --name MOV02979 --name MOV01046 --name MOV02838 --load-config convert.toml 

Auxiliary TOML file

• Please note I use some exiftool related functions (see my gist on exiftool functions) like:
  • exifcreatedate to set all dates to the EXIF create date
  • exifsetMVI (originally for Canon MVI videos) to make sure all dates (incluing QuickTime are properly set. Applicable to all videos. Does not use Timezone.

[export]
add_exported_to_album = "Exported_MPG_movies"
directory = "{folder_album}"
query_eval = [ "photo.filename[-3:] == 'MPG'",]
post_command = [ 
# Select one of the following two options for ffmpeg
# ffmpeg standard quality/compression
    [ "exported", "ffmpeg -i {filepath|shell_quote} {shell_quote,{filepath.parent}/{filepath.stem}.MOV}",], 
#  ffmpeg better quality/slower conversion 
#   [ "exported", "ffmpeg -i {filepath|shell_quote} -c:v libx264 -preset veryslow -crf 17 {shell_quote,{filepath.parent}/{filepath.stem}.MOV}",],
    [ "exported", "exiftool -overwrite_original -tagsfromfile {shell_quote,{filepath}.xmp} -all:all {shell_quote,{filepath.parent}/{filepath.stem}.MOV}",],
    [ "exported", "exifcreatedate {shell_quote,{filepath.parent}/{filepath.stem}.MOV}",],
    [ "exported", "exifsetMVI {shell_quote,{filepath.parent}/{filepath.stem}.MOV}",],
]
report = "exported_mpg_files_{today}.csv"
sidecar = [ "xmp",]
uti = "public.mpeg"
skip_original_if_edited = true
verbose = true
timestamp = true
no_progress = true
export_as_hardlink = true

Import

• Note: The hidden pics exported will be imported as Not Hidden. You must hide them manually.

osxphotos import export --no-progress --verbose --timestamp --glob "*.MOV" --exiftool --walk --album "{filepath.parent}" --relative-to "export" --split-folder "/" --report "imported_mpg_files_{today}.csv"

Go to album "Exported_MPG_movies"(named above) and delete

• Once you ensure everything's fine...
• Select Command + Right-click to delete

[oPromessa]

An update on this topic. I am using this approach very successfully. It's a life/saver. Thanks!

Only twist is that, following the import -- as we know, it takes the Album names from the folders on the filesystem -- it ends-up creating a second Album on Photos with the same name. Please note many of my albums have international characters.

• The created album is on the correct Folder.

I've changed the name of the algum to "ÁLBUM"in the log file.

2023-05-21 17:47:40.850781 -- Importing /Users/Shared/Pictures/Export/rotate_uD800/export/iPhoto Events/2000-2005/ÁLBUM/PC100030_rotated.JPG
2023-05-21 17:47:41.297356 -- Imported PC100030_rotated.JPG with UUID 59C5D6D0-CF8D-4B9A-90AE-0582A94E12F3
2023-05-21 17:47:41.298081 -- Setting metadata and location from EXIF for PC100030_rotated.JPG
2023-05-21 17:47:41.696440 -- Set metadata for PC100030_rotated.JPG:
2023-05-21 17:47:41.697298 -- title='', description='OLYMPUS DIGITAL CAMERA         ', keywords=['XXXX']
2023-05-21 17:47:41.697613 -- No location to set for PC100030_rotated.JPG
2023-05-21 17:47:41.698971 -- Adding photo PC100030_rotated.JPG to 1 album
2023-05-21 17:47:41.699287 -- Adding photo PC100030_rotated.JPG to album iPhoto Events/2000-2005/ÁLBUM
2023-05-21 17:47:45.819553 -- Creating album 'ÁLBUM'
2023-05-21 17:47:46.009084 -- Added PC100030_rotated.JPG (59C5D6D0-CF8D-4B9A-90AE-0582A94E12F3) to album iPhoto Events/2000-2005/ÁLBUM
2023-05-21 17:47:46.028648 -- Wrote import report to imported_rotate_2023-05-21.csv
2023-05-21 17:47:46.029109 -- Done: imported 1 file, 0 errors, 

Wild guess!!! Would this be a Unicode UTF-8 NFD topic?

I'm running some tests to understand:

• Photos considers it to be two different Albums
• What does osxphotos info consider?
  • Seems to merge both entries and report them in one single album
• What does osxphotos query report?
• And osxphotos export to a local SSD disk on Mac, will it create two folders or end up with only one?
• And osxphotos export to a NAS disk on Mac, will it create two folders or end up with only one?

Side Note: Will add one more example of this transform/convert process (#909) which I'm using to fix directly on the JPG files the appropriate orientation for some old Pics (of one particular camera which did not record the pic orientation correctly!) which even though are edited/rotated in Photos, Photos does not create an edited file representation and osxphotos is stuck/forced to exporting the original #871 .

[oPromessa]

I'd say so. Yes.

Just don't want to break anyone else's flow should it be working for them.

[RhetTbull]

I'd forgotten about filter functions. You could use your existing code as a filter function:

function: Run custom python function to filter value; use in format ‘function:/path/to/file.py::function_name’. See example at https://github.com/RhetTbull/osxphotos/blob/master/examples/template_filter.py

So something like --album "{filepath.parent|function:fixunicode.py::fixunicode}"

[oPromessa]

Awsome! I believe it did the trick. Possibly didn't reach the full manual by I ended up using unicodedata.normalize('NFD', val) to convert into NFD when I expected I had to convert into NFC!

osxphotos import export --no-progress --verbose --timestamp --glob "PC10003[0]_rotated.JPG" --exiftool --walk --album "{filepath.parent|function:fixunicode.py::fixunicode}" --relative-to "export" --split-folder "/" --report "imported_rotate_{today}.csv"
2023-05-21 20:39:31.989522 -- Importing /Users/Shared/Pictures/Export/rotate_uD800/export/iPhoto Events/2000-2005/ÁLBUM/PC100030_rotated.JPG
2023-05-21 20:39:32.265087 -- Imported PC100030_rotated.JPG with UUID D225A529-D33A-48C5-AE8A-17D64D1837C1
2023-05-21 20:39:32.265862 -- Setting metadata and location from EXIF for PC100030_rotated.JPG
2023-05-21 20:39:32.691938 -- Set metadata for PC100030_rotated.JPG:
2023-05-21 20:39:32.692740 -- title='', description='OLYMPUS DIGITAL CAMERA         ', keywords=['XXXX']
2023-05-21 20:39:32.693059 -- No location to set for PC100030_rotated.JPG
2023-05-21 20:39:32.699007 -- Adding photo PC100030_rotated.JPG to 1 album
2023-05-21 20:39:32.699450 -- Adding photo PC100030_rotated.JPG to album iPhoto Events/2000-2005/ÁLBUM
2023-05-21 20:39:36.815703 -- Added PC100030_rotated.JPG (D225A529-D33A-48C5-AE8A-17D64D1837C1) to album iPhoto Events/2000-2005/ÁLBUM
2023-05-21 20:39:36.843163 -- Wrote import report to imported_rotate_2023-05-21.csv
2023-05-21 20:39:36.843659 -- Done: imported 1 file, 0 errors, 

and the file fixcode.py:

import sys
import unicodedata
from typing import List

def fixunicode(values: List[str]) -> List[str]:
    """ Custom filter to append convet folders (from Mac in UTF-8 NFD) to Photos compatible UTF-8 NFC to template value """
    values = [unicodedata.normalize('NFD', val) for val in values]
    return values

Will run some more tests!

[oPromessa]

Hi there. It worked very well.

But same issue found now with Keywords.

While exporting, I've added keywords to the files (via XMP) -- eg "Cão" -- which, on osxphotos import, via exiftool caused the same issues now with Keywords: their duplication: visually the same name but one in NFD and the other in NFC.

• On Photos under the Keywords windows you can see two entries for the "same" keyword: "Cão" and "Cão".
• On a Photos SmartAlbum if you select Keywords, oddly enough, it only shows one option to filter under Keyword, but shows the total pics, say 100.
• Also querying with `osxphotos query --keyword "Cão"

Workaround:

1. Tried using --keyword "{keyword|function:fixunicode.py::fixunicode}" with and without --merge-keywords on the osxphotos import but itdos not seem to change keywords!
2. I can query on Photos on Keyword "Cão" and imported in the last 30 days - basically the pics from the EXPORT-CONVERT-IMPORT and remove the bad keyword and add the "old" one.
   • Testing export... to see if merge-keywords will still pickup the "new" keyword tagged on the file's EXIF.
   • Hmmm.. there isn't really a way to tell on the exported files ;)

[RhetTbull]

There are two things in programming I really hate: dates and Unicode! 😀 I will take a look to see where the problem is. This can be fixed if all Unicode is normalized before comparing strings -- it's just a lot of work to find all the places in the code where this needs to happen. The export command already does this for filename a when exporting.

[RhetTbull]

@oPromessa I'm trying to replicate all these issues so I can write tests for them then fix them. Have done some initial refactoring of the unicode code in #1085

Only twist is that, following the import -- as we know, it takes the Album names from the folders on the filesystem -- it ends-up creating a second Album on Photos with the same name. Please note many of my albums have international characters.

In my tests, Photos does not create a duplicate album if the album exists at the top level of the library but it does create a duplicate album if the album is in a folder. Have you seen an occurrence where you got a duplicate top-level album?

Album âlbüm created in both NFC and NFD variants.

Top level: no duplicate

In folder: duplicates (also duplicate folder when folder name is NFC vs NFD)

[RhetTbull]

Success! only 1 album getting created now regardless of unicode variant of album name:

[oPromessa]

Similar issue with Keywords.

If you have a pic with keyword "Sümmer" being imported it will create a second keyword -- when one already exist in Photos -- with same name "Sümmer" on keyword manager window.

But then on the photos Query smart albums you only "see" one entry... not being able to identify which pics have which keyword.

If I recall, exporting via exiftool the keywords its then up to the 3rd party tool using such files to manage it - in my case Synology Photos, AFAIK, seems to only have one tag/keyword "Sümmer".

[RhetTbull]

Keywords are next in my queue!

[RhetTbull]

Give v0.60.5 a try. I think this fixes the unicode problems on import.

[oPromessa]

I will. Thanks so much for the hardwork!