Working on macOS 12 (Monterey)

[RhetTbull]

If any osxphotos users have access to the macOS 12 / Moneterey developer preview, I'd like to get a copy of a test Photos database to ensure osxphotos remains compatible.

[RhetTbull]

Thanks to a reader in the r/MacOSBeta subreddit, I was able to get a couple of test libraries (s/he asked I credit analytic.eth so thank-you!). The database diffs from Big Sur are on the wiki. I was able to hack together some code to be able to read the database and show that osxphotos will still work but it'll take some significant engineering to get osxphotos fully compatible with Monterey.

The biggest change seems minor but will be actually a pain to work around is ZUNIFORMTYPEIDENTIFIER is dropped from ZINTERNALRESOURCE. osxphotos uses this to get the UTI for the original image because Photos switches the UTI of edited images to either public.jpeg or public.heic once image is edited. osxphotos needs the UTI of the original image for use with --convert-to-jpeg and a few other things (e.g. assigning the right extension to photos downloaded with the --download-missing option). It's possible to get UTI from the file extension of the original filename but the functions osxphotos uses to do this (objective C calls to UTTypeCopyPreferredTagWithClass and it's corollary) but these functions are deprecated and won't work in Monterey. They've been replaced by UTType but this is Swift-only and I know no way of calling Swift easily from python. (osxphotos uses PyObjC a fair amount to call Objective-C) I'll need to come up with a reliable way to go round-trip between UTI and file extension.

ZUNIFORMTYPEIDENTIFIER appears to be replaced by ZCOMPACTUTI but this is just an int and doesn't appear to be an index to another table so not sure how Photos turns this back into a UTI.

Another interesting thing is it looks like from this table that osxphotos might be able to get access to the new "live text" feature to get the character recognition data that's built into Monterey's Photos:

+CREATE TABLE ZCHARACTERRECOGNITIONATTRIBUTES (
+    Z_PK INTEGER PRIMARY KEY,
+    Z_ENT INTEGER,
+    Z_OPT INTEGER,
+    ZALGORITHMVERSION INTEGER,
+    ZMEDIAANALYSISASSETATTRIBUTES INTEGER,
+    ZADJUSTMENTVERSION TIMESTAMP,
+    ZDATA BLOB
+);
+

[RhetTbull]

It won't be pretty but one possible solution is to feed every possible extension into UTTypeCreatePreferredIdentifierForTag to get the UTI then feed every UTI into UTTypeCopyPreferredTagWithClass to get the preferred extension. (Ignoring the dynamically generated UTIs which are created for unknown extensions). This would be give me two dicts that could be used to lookup every known UTI (as of this moment) which I could package into an easy to use python package.

def get_uti_for_extension(extension):
    with objc.autorelease_pool():
        uti = CoreServices.UTTypeCreatePreferredIdentifierForTag(
            CoreServices.kUTTagClassFilenameExtension, extension, None
        )
        return uti

[RhetTbull]

Progress! osxphotos can now process a Monterey Photos database. The only thing that isn't working yet is ability to process the RAW pair of a RAW+JPEG pair. Photos 7 (? -- assume it's Photos 7) changes how UTI information is stored and I've not yet figured out how to get the data needed to find the paths of the associated RAW files.

[I] ➜ python -m pytest tests/test_monterey_dev_beta_12_0_0.py
==================================================== test session starts ====================================================
platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/rhet/Dropbox/Code/osxphotos
plugins: mock-3.6.1
collected 97 items

tests/test_monterey_dev_beta_12_0_0.py c.............................................................................. [ 80%]
.F..sssss....s.....                                                                                                   [100%]

========================================================= FAILURES ==========================================================
_________________________________________________________ test_raw __________________________________________________________

photosdb = osxphotos.PhotosDB(dbfile='/Users/rhet/Dropbox/Code/osxphotos/tests/Test-12.0.0.dev-beta.photoslibrary/database/photos.db')

    def test_raw(photosdb):
        """Test various raw properties"""

        for uuid, rawinfo in RAW_DICT.items():
            photo = photosdb.get_photo(uuid)
            assert photo.original_filename == rawinfo.original_filename
            assert photo.has_raw == rawinfo.has_raw
            assert photo.israw == rawinfo.israw
            assert photo.uti == rawinfo.uti
            assert photo.uti_original == rawinfo.uti_original
>           assert photo.uti_raw == rawinfo.uti_raw
E           AssertionError: assert None == 'com.canon.cr2-raw-image'
E            +  where None = osxphotos.PhotoInfo(db=osxphotos.PhotosDB(dbfile='/Users/rhet/Dropbox/Code/osxphotos/tests/Test-12.0.0.dev-beta.photos...0, 'keywords': [], 'hasPersons': 0, 'persons': [], 'albums': ['68001ACE-DE4E-46B7-A3F8-3B2D03D39D59'], 'hasAlbums': 1}).uti_raw
E            +  and   'com.canon.cr2-raw-image' = RawInfo(comment='raw+jpeg, jpeg original', original_filename='IMG_1994.JPG', has_raw=True, israw=False, raw_original=False, uti='public.jpeg', uti_original='public.jpeg', uti_raw='com.canon.cr2-raw-image').uti_raw

tests/test_monterey_dev_beta_12_0_0.py:1201: AssertionError
================================================== short test summary info ==================================================
FAILED tests/test_monterey_dev_beta_12_0_0.py::test_raw - AssertionError: assert None == 'com.canon.cr2-raw-image'
========================================== 1 failed, 90 passed, 6 skipped in 3.43s ==========================================

osxphotos on  master [$!?] via osxphotos took 5s
[I] ➜ cp tests/test_catalina_10_15_7.py te

[RhetTbull]

I've not found any way to get the raw UTI from Monterey database. The only value stored is an int in ZINTERNALRESOURCE.ZCOMPACTUTI but I've not found a way to turn this back into a UTI. This value is probably something generated by the new UTType methods but these are Swift-only so even if I could figure it out, can't call these from Python.

Support for RAW+JPEG in Monterey will be very limited if I can't figure out how to get the RAW UTI reliably.

If the file is present on disk, it appears to be in originals with _4 suffix. But this isn't true if the files were imported as reference.

[I] ➜ find ~/Pictures/Photos\ Library.photoslibrary | grep E3DD04AF-CB65-4D9B-BB79-FF4C955533DB
/Users/rhet/Pictures/Photos Library.photoslibrary/resources/derivatives/masters/E/E3DD04AF-CB65-4D9B-BB79-FF4C955533DB_4_5005_c.jpeg
/Users/rhet/Pictures/Photos Library.photoslibrary/resources/derivatives/E/E3DD04AF-CB65-4D9B-BB79-FF4C955533DB_1_105_c.jpeg
/Users/rhet/Pictures/Photos Library.photoslibrary/originals/E/E3DD04AF-CB65-4D9B-BB79-FF4C955533DB_4.cr2
/Users/rhet/Pictures/Photos Library.photoslibrary/originals/E/E3DD04AF-CB65-4D9B-BB79-FF4C955533DB.jpeg

osxphotos on  master [$?] via osxphotos took 2s
[I] ➜ find ~/Pictures/Photos\ Library.photoslibrary | grep 4D4EE0F4-27AF-4BDD-84ED-7EDC65F832C6
/Users/rhet/Pictures/Photos Library.photoslibrary/resources/derivatives/masters/4/4D4EE0F4-27AF-4BDD-84ED-7EDC65F832C6_4_5005_c.jpeg
/Users/rhet/Pictures/Photos Library.photoslibrary/resources/derivatives/4/4D4EE0F4-27AF-4BDD-84ED-7EDC65F832C6_1_105_c.jpeg
/Users/rhet/Pictures/Photos Library.photoslibrary/originals/4/4D4EE0F4-27AF-4BDD-84ED-7EDC65F832C6_4.dng
/Users/rhet/Pictures/Photos Library.photoslibrary/originals/4/4D4EE0F4-27AF-4BDD-84ED-7EDC65F832C6.jpeg

[RhetTbull]

And here's a clue. The value for the "alternate photo" (e.g. the RAW image) constant is 4

https://developer.apple.com/documentation/photokit/phassetresourcetype/phassetresourcetypealternatephoto?language=objc

PHAssetResourceTypeAlternatePhoto = 4

[Rott-Apple]

Does it help that when RAW + JPG pairs are imported, ZINTERNALRESOURCE.ZRESOURCETYPE = 4 in Ventura, Monterey, Big Sur at least? Or is this issue already sorted? I haven't needed this feature so far so I don't know.

[RhetTbull]

@Rott-Apple thanks. The export of RAW+JPEG has been fixed for Monterey+. The fix is a bit of a hack (see here) but it works and successfully allows osxphotos to export the raw images on Monterey and above.

[jcommisso07]

So sorry for the delay in responding. Here are the files you requested on Reddit. Hope they help!

db_schema.txt
sqlite_schema.txt

[RhetTbull]

Thanks @jcommisso07

[RhetTbull]

@all-contributors please add @jcommisso07 for data

[Rott-Apple]

Just started looking at the Photos for Monterey database, and Apple has again changed some table names. Z_26ALBUMS is now Z_27ALBUMS, Z_26ASSETS is Z_27ASSETS, and Z_36KEYWORDS is Z_38KEYWORDS. Haven't looked inside if there are any changes in column names etc.

[RhetTbull]

Thanks! I think the current osxphotos code will account for these so let me know if you run into any issues on Monterey.

[endlessedits]

Newbie here. Just upgraded to Monterey from Big Sur (12.0.1). Have used command "osxphotos export --exiftool /my-output-folder" successfully hundreds of times on my 25k photo set with no errors ever. Suddenly since upgrade to Monterey I am getting "OperationalError: no such column: ZDETECTEDFACE.ZBALDTYPE".

Amazing tool -- I can't thank you enough for it (allowing me to avoid frustrating Apple bugs in Photos). If you need more info to evaluate please let me know. Thanks in advance for any help!

[RhetTbull]

Thanks. I don't have Monterey to test (I've got an older unsupported Mac) so made the changes for Monterey based on data I got from someone who had the developer beta. That field was in the beta but must have been removed. It would be helpful if you could create a small Photos library with just one or two photos and send that to me so I could make the necessary updates.

(If you hold down the Option key while starting Photos you'll be presented with an option to create a new library)

You should be able to zip the library and attach it here.

[endlessedits]

Monterey test photoslibary and pdf of output.zip

Thanks for the quick response -- please let me know if you need anything else.

[rsr-at-mindtwin]

adding a comment to volunteer as well if there's anything else that is needed.

[RhetTbull]

@endlessedits @rsr-at-mindtwin @Rott-Apple I've updated v0.43.0 based on the sample library @endlessedits sent and everything seems to be working as far as I can tell. Let me know if you run into any other issues.

[endlessedits]

New v0.43.0 is working perfectly for me under Monterey 12.0.1 now. Export of a large photoslibrary appears flawless. Osxphotos is indispensable -- awesome turnaround time on this new version too. Thank you very much, Sir!

[rsr-at-mindtwin]

Hmmm.

% osxphotos --version
osxphotos, version 0.43.0
% osxphotos export --from-date "2021-10-01" --update --finder-tag-keywords --person-keyword --xattr-template findercomment "{title}{title?{descr?{newline},},}{descr}" --exiftool-merge-keywords --exiftool-merge-persons --exiftool --keyword-template "{label,nolabel}" --directory '{created.year}-{created.mm}/{place.name.city,unknown}' --filename '{created.year}-{created.mm}-{created.dd} {original_name}' --strip /[redacted]/Exported/
Using last opened Photos library: /Users/myusername/Pictures/Photos Library.photoslibrary
Exporting 158 photos to /Volumes/mumble/Exported...
[------------------------------------] 0%2021-10-29 14:18:15,679 - WARNING - findertags.py - 96 - Could not open plist file: /Users/myusername/Library/SyncedPreferences/com.apple.finder.plist, exception: [Errno 2] No such file or directory: '/Users/myusername/Library/SyncedPreferences/com.apple.finder.plist'

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/myusername/.local/bin/osxphotos:8 in │
│ │
│ 5 from osxphotos.main import cli │
│ 6 if name == 'main': │
│ 7 │ sys.argv[0] = re.sub(r'(-script.pyw|.exe)?$', '', sys.argv[0]) │
│ ❱ 8 │ sys.exit(cli()) │
│ 9 │
│ │
│ /Users/myusername/.local/pipx/venvs/osxphotos/lib/python3.9/site-packages/click/core.py:1128 in │
│ call │
│ │
│ 1125 │ │
│ 1126 │ def call(self, *args: t.Any, **kwargs: t.Any) -> t.Any: │
│ 1127 │ │ """Alias for :meth:main.""" │
│ ❱ 1128 │ │ return self.main(*args, **kwargs) │
│ 1129 │
│ 1130 │
│ 1131 class Command(BaseCommand): │
│ │
│ /Users/myusername/.local/pipx/venvs/osxphotos/lib/python3.9/site-packages/click/core.py:1053 in main │
│ │
│ 1050 │ │ try: │
│ 1051 │ │ │ try: │
│ 1052 │ │ │ │ with self.make_context(prog_name, args, **extra) as ctx: │
│ ❱ 1053 │ │ │ │ │ rv = self.invoke(ctx) │
│ 1054 │ │ │ │ │ if not standalone_mode: │
│ 1055 │ │ │ │ │ │ return rv │
│ 1056 │ │ │ │ │ # it's not safe to ctx.exit(rv) here! │
│ │
│ /Users/myusername/.local/pipx/venvs/osxphotos/lib/python3.9/site-packages/click/core.py:1659 in invoke │
│ │
│ 1656 │ │ │ │ super().invoke(ctx) │
│ 1657 │ │ │ │ sub_ctx = cmd.make_context(cmd_name, args, parent=ctx) │
│ 1658 │ │ │ │ with sub_ctx: │
│ ❱ 1659 │ │ │ │ │ return _process_result(sub_ctx.command.invoke(sub_ctx)) │
│ 1660 │ │ │
│ 1661 │ │ # In chain mode we create the contexts step by step, but after the │
│ 1662 │ │ # base command has been invoked. Because at that point we do not │
│ │
│ /Users/myusername/.local/pipx/venvs/osxphotos/lib/python3.9/site-packages/click/core.py:1395 in invoke │
│ │
│ 1392 │ │ │ echo(style(message, fg="red"), err=True) │
│ 1393 │ │ │
│ 1394 │ │ if self.callback is not None: │
│ ❱ 1395 │ │ │ return ctx.invoke(self.callback, **ctx.params) │
│ 1396 │ │
│ 1397 │ def shell_complete(self, ctx: Context, incomplete: str) -> t.List["CompletionItem"]: │
│ 1398 │ │ """Return a list of completions for the incomplete value. Looks │
│ │
│ /Users/myusername/.local/pipx/venvs/osxphotos/lib/python3.9/site-packages/click/core.py:754 in invoke │
│ │
│ 751 │ │ │
│ 752 │ │ with augment_usage_errors(__self): │
│ 753 │ │ │ with ctx: │
│ ❱ 754 │ │ │ │ return __callback(*args, **kwargs) │
│ 755 │ │
│ 756 │ def forward( │
│ 757 │ │ __self, __cmd: "Command", *args: t.Any, **kwargs: t.Any # noqa: B902 │
│ │
│ /Users/myusername/.local/pipx/venvs/osxphotos/lib/python3.9/site-packages/click/decorators.py:38 in │
│ new_func │
│ │
│ 35 │ """ │
│ 36 │ │
│ 37 │ def new_func(*args, **kwargs): # type: ignore │
│ ❱ 38 │ │ return f(get_current_context().obj, *args, **kwargs) │
│ 39 │ │
│ 40 │ return update_wrapper(t.cast(F, new_func), f) │
│ 41 │
│ │
│ /Users/myusername/.local/pipx/venvs/osxphotos/lib/python3.9/site-packages/click/decorators.py:26 in │
│ new_func │
│ │
│ 23 │ """ │
│ 24 │ │
│ 25 │ def new_func(*args, **kwargs): # type: ignore │
│ ❱ 26 │ │ return f(get_current_context(), *args, **kwargs) │
│ 27 │ │
│ 28 │ return update_wrapper(t.cast(F, new_func), f) │
│ 29 │
│ │
│ /Users/myusername/.local/pipx/venvs/osxphotos/lib/python3.9/site-packages/osxphotos/cli.py:1859 in │
│ export │
│ │
│ 1856 │ │ │ │ ) │
│ 1857 │ │ │ │ │
│ 1858 │ │ │ │ if finder_tag_keywords or finder_tag_template: │
│ ❱ 1859 │ │ │ │ │ tags_written, tags_skipped = write_finder_tags( │
│ 1860 │ │ │ │ │ │ p, │
│ 1861 │ │ │ │ │ │ photo_files, │
│ 1862 │ │ │ │ │ │ keywords=finder_tag_keywords, │
│ │
│ /Users/myusername/.local/pipx/venvs/osxphotos/lib/python3.9/site-packages/osxphotos/cli.py:3468 in │
│ write_finder_tags │
│ │
│ 3465 │ │ ] │
│ 3466 │ │ tags.extend(rendered_tags) │
│ 3467 │ │
│ ❱ 3468 │ tags = [osxmetadata.Tag(tag) for tag in set(tags)] │
│ 3469 │ for f in files: │
│ 3470 │ │ md = osxmetadata.OSXMetaData(f) │
│ 3471 │ │ if sorted(md.tags) != sorted(tags): │
│ │
│ /Users/myusername/.local/pipx/venvs/osxphotos/lib/python3.9/site-packages/osxphotos/cli.py:3468 in │
│ │
│ │
│ 3465 │ │ ] │
│ 3466 │ │ tags.extend(rendered_tags) │
│ 3467 │ │
│ ❱ 3468 │ tags = [osxmetadata.Tag(tag) for tag in set(tags)] │
│ 3469 │ for f in files: │
│ 3470 │ │ md = osxmetadata.OSXMetaData(f) │
│ 3471 │ │ if sorted(md.tags) != sorted(tags): │
│ │
│ /Users/myusername/.local/pipx/venvs/osxphotos/lib/python3.9/site-packages/osxmetadata/findertags.py:131 │
│ in init │
│ │
│ 128 │ │ │ │ │ f"color must be in range 0 to {_MAX_FINDER_COLOR} inclusive: {color[ │
│ 129 │ │ │ │ ) │
│ 130 │ │ │
│ ❱ 131 │ │ self._finder_tags = get_finder_tags() │
│ 132 │ │ self._name = name │
│ 133 │ │ │
│ 134 │ │ # if no color provided use color from Finder if it exists │
│ │
│ /Users/myusername/.local/pipx/venvs/osxphotos/lib/python3.9/site-packages/osxmetadata/findertags.py:97 │
│ in get_finder_tags │
│ │
│ 94 │ │
│ 95 │ except Exception as e: │
│ 96 │ │ logging.warning(f"Could not open plist file: {plist_file}, exception: {e}") │
│ ❱ 97 │ │ raise e │
│ 98 │ │
│ 99 │ return tags │
│ 100 │
│ │
│ /Users/myusername/.local/pipx/venvs/osxphotos/lib/python3.9/site-packages/osxmetadata/findertags.py:78 │
│ in get_finder_tags │
│ │
│ 75 │ ) │
│ 76 │ tags = {} │
│ 77 │ try: │
│ ❱ 78 │ │ with open(plist_file, "rb") as fp: │
│ 79 │ │ │ pl = plistlib.load(fp) │
│ 80 │ │ │ try: │
│ 81 │ │ │ │ finder_tags = pl["values"]["FinderTagDict"]["value"]["FinderTags"] │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: [Errno 2] No such file or directory: '/Users/myusername/Library/SyncedPreferences/com.apple.finder.plist'

[RhetTbull]

@rsr-at-mindtwin Could you check and see if you can see this file (e.g. it exists): /Users/myusername/Library/SyncedPreferences/com.apple.finder.plist

I wonder if it exists in Monterey and can't be viewed by osxphotos due to system integrity protection (SIP) or if it no longer exists in Monterey.

[Rott-Apple]

Hi, don't mean to preempt @rsr-at-mindtwin but on Mac running Monterey I don't see this file in SyncedPreferences (which stores settings that are synchronized with other systems), although it's there in Preferences (which provides for individual settings for each user). Could this be due to my Sync preferences? Also, /Users/myusername/Library/SyncedPreferences/com.apple.kvs/ChangeTokens/NoEncryption/Finder has another com.apple.finder.

[RhetTbull]

@Rott-Apple thanks. Could you try the following command:

plutil -p ~/Library/Preferences/com.apple.finder.plist

and let me know if the output looks something like this:

{
  "changecount" => 2
  "initialsync" => 1
  "values" => {
    "FinderTagDict" => {
      "remotevalue" => {length = 549, bytes = 0x01f0c57b e0b32527 00000000 62706c69 ... 00000000 0000016f }
      "timestamp" => 656782304
      "value" => {
        "FinderTagDictVersion" => 2
        "FinderTags" => [
          0 => {
            "n" => "Untitled"
            "p" => 1
          }
          1 => {
            "l" => 3
            "n" => "Purple"
            "p" => 1
...

What I'm interested in is the "FinderTags" property which maps Finder tags to Finder colors. If this exists in your version of the file in Preferences, then I can have the code look there in addition to synched preferences.

[Rott-Apple]

Ouch sorry, was on vacation. Glad to see v0.43.1 works for @rsr-at-mindtwin. Checked the above; no mention of FinderTags in the output. And although no com.apple.finder.plist in SyncedPreferences on my Monterey Mac, the file com.apple.syncedpreferences.plist mentions its path in SyncedPreferences folder, below.

[RhetTbull]

Thanks @Rott-Apple. A lot of this stuff is undocumented and I've discovered from reverse engineering and poking around the system. In this case, the plist in SyncedPreferences contains a mapping of known Finder tags and tag colors (users can apply custom colors to tags). Without this, osxmetadata, which writes the Finder tags, can't guarantee the tags written will have the same color the user previously defined if they match a pre-defined tag. The current work-around will still write the tags but at the risk that they'll have the wrong custom color. For most use cases, this isn't an issue but if you use a lot of custom tag colors, it might be. I'll have to eventually figure out where Monterey stores this information. The system must have a central place for this info as Finder needs it when you apply a new tag to a file.

[RhetTbull]

Give v0.43.1 a try. I think I've worked around the issue @rsr-at-mindtwin was having and you should be able to use finder tags now but I'm going to guess that finder comments are still broken.

[rsr-at-mindtwin]

It appears confirmed fixed from my POV.

[sgarratt77]

I am using mac osx monterey 12.1 on an m1 macbook pro and using osxphotos, version 0.44.9. I'm not sure if this is the latest version of osxphotos but I literally installed it yesterday using pipx so I would imagine it would have installed the latest version (?). I query this because upon using it it said "WARNING - photosdb.py - 107 - WARNING: This module has only been tested with macOS versions [10.12, 10.13, 10.14, 10.15, 10.16, 11.0, 11.1, 11.2, 11.3, 11.4, 11.5, 11.6]: you have Darwin, OS version: 12.1"
That sounds like it isn't tested on my system, even though I've seen on the main git page that it supports mac osx 12.0.

I started it running through a tiny 4.45gb photos library yesterday at about 7pm and it has been going through the night! it is 10am as I type this and it is only 81% through! with an estimated 02:29:48 left. granted I am exporting the media to my nas on my internal 100mbps network but surely this really shouldn't take that long!? what is causing this? should I export to a local hard drive instead? i can't believe the network can be the problem

Ste

[sgarratt77]

wow! thanks Rhet, you really are sh*t hot with support dude! thank you so much. is there a way to tell it to export the first 200gb worth of media? and then go on to export the second 200gb worth or not? is that something you could integrate in the future if not?
ill give that a try though

[RhetTbull]

is there a way to tell it to export the first 200gb worth of media? and then go on to export the second 200gb worth or not? is that something you could integrate in the future if not?

Unfortunately not as there's not really a way to specify "first 200gb" relative to what? Only real choice would be "date of earliest photo in the library" but what if you then add a photo with an earlier date between updates? Also, it's not known until export time how big everything will be -- e.g. are there edits, live photos, previews, etc. being exported and if so the size of each of those needs to be computed. If you're using --download-missing, then osxphotos wouldn't know the size of the missing photos until they are downloaded, etc.

It would be possible to specify max number of items to export at a time then incrementally export then next X items as each item has a unique database ID when it's added to the library and these are sequential. For example, something like this:

osxphotos export /path/to/export --max-items 1000 --update

and if run again, would export next 1000 items and so on.

This would take some rework but it's something to think about.

I can also see a --year would be helpful to export only one year at a time, e.g. -year 2017 then --year 2018, and so on.

[oPromessa]

Hello @sgarrat77

Take also a look at #582

In case it might help. What I'm doing is

• use the --export-db option to place the working database file .osxphotos_export.db on a local folder on the Mac to avoid added load on the network. This for me resulted in quite a reduction. Thanks @RhetTbull for adding this feature.
• use the retry option just in case
• use the update and cleanup options and exiftool of course
• also use the touch-file option as the modified date on the files copied to the NAS was being set as the time of the copy. Then my NAS Synology Photos app was wrongly classifying file dates, especially for MPG files which exiftool can't add OriginalDate to it. See what happens on your particular NAS system.
• and then I'm using the from-date to add year after year to the mix. So first run would be from 2021-01-01T00:00:00, second run from 2020-01-01T00:00:00, going back one further year and so on.
• I share my osxphotos command line and toml in case it might be helpful.

osxphotos export /Volumes/photo-1/Family.Photos --exportdb ./.osxphotos_export.db --load-config ./colibri.toml

[export]
cleanup = true
db = "/Users/Shared/Pictures/iPhoto Shared Library.photoslibrary"
directory = "{album[/,.|:,.]}"
#dry_run = true
exiftool = true
exiftool_merge_keywords = true
exiftool_merge_persons = true
touch_file = true
from_date = 2010-01-01T00:00:00
keyword_template = [ "{keyword}",]
not_hidden = true
report = "colibri.csv"
retry = 2
skip_original_if_edited = true
strip = true
update = true
verbose = true

[sgarratt77]

THANK YOU oPromessa. that is helpful. sadly I have no time today to take a look but I will. i think I am going to look all through all the commands so I don't miss anything, I will use your example and rhets advice to guide my decisions :)

[RhetTbull]

@sgarratt77 @oPromessa I've added a new --limit option to osxphotos export in v0.48.6. When used, --limit will cause osxphotos to export at most X items. For example, osxphotos export /path/to/export --limit 200 will export only 200 photos then stop. This can be used with --update to incrementally complete an export. (that is, each subsequent update will export the next X items). Closes #592

[sgarratt77]

ok I'm trying the export x amount of files thing. i chose this root because it is harder to estimate how big an export a date range will be.
i tried the command you mentioned 'osxphotos tutorial' and got stuff I didn't understand and then this at the bottom:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/Ste/.local/pipx/
venvs/osxphotos/lib/python3.10/site-packages/osxphotos/tutorial.md'
something is probably not set up right(?) I installed everything (homebrew, pipx and then osxphotos) as directed so I don't know what all that means

[RhetTbull]

That's a bug as the file should be present. I'll take a look.

[sgarratt77]

thanks. I'm getting paranoid. just before i begin exporting all my stuff so that I can import it into lightroom classic. can I ask if the command that I'm going to use will export absolutely EVERYTHING so that I don't leave anything behind (I mean, originals, edits, all keywords, faces/people, albums, smart albums, all types of shots - burst, live, panoramas, and anything else I didn't think about, etc?)

my intention is to use the other components mentioned in Robs guide from the previous link I mentioned to import everything into LR classic. commands I was going to use:

osxphotos export /Volumes/external hd/OSXPhotos/Main/001 --max-items 1000 --update --sidecar xmp
--keyword-template "PhotosExport>{folder_album(>)}"
--keyword-template "{edited?edited-before-lightroom,}"
--exiftool

id hate to realise later that I missed something and have deleted my photo library or have to re-export it all again lol

[RhetTbull]

can I ask if the command that I'm going to use will export absolutely EVERYTHING so that I don't leave anything behind (I mean, originals, edits, all keywords, faces/people, albums, smart albums, all types of shots - burst, live, panoramas, and anything else I didn't think about, etc?)

I recommend you do some testing and read the docs for osxphotos. I don't use lightroom so can't tell you how it will handle exported metadata. If you use --exiftool, the following are exported:

            EXIF:ImageDescription (may include template)
            XMP:Description (may include template)
            XMP:Title
            IPTC:ObjectName
            XMP:TagsList (may include album name, person name, or template)
            IPTC:Keywords (may include album name, person name, or template)
            IPTC:Caption-Abstract
            XMP:Subject (set to keywords + persons)
            XMP:PersonInImage
            EXIF:GPSLatitudeRef, EXIF:GPSLongitudeRef
            EXIF:GPSLatitude, EXIF:GPSLongitude
            EXIF:GPSPosition
            EXIF:DateTimeOriginal
            EXIF:OffsetTimeOriginal
            EXIF:ModifyDate
            IPTC:DateCreated
            IPTC:TimeCreated
            QuickTime:CreationDate
            QuickTime:CreateDate (UTC)
            QuickTime:ModifyDate (UTC)
            QuickTime:GPSCoordinates
            UserData:GPSCoordinates

The ones that reference a template can use the osxphotos template system to create custom keywords (that's what this does: --keyword-template "PhotosExport>{folder_album(>)}".

smart albums

Smart albums are not handled by osxphotos and won't get exported. You could create regular albums from each smart album and those would.

burst

Bursts are exported but the unselected bursts are not. This is consistent with Photos which treats unselected bursts as if they don't exist (until they are selected).

--max-items 1000

There is no --max-items option currently. I suggested that as a possible feature to add to do what you wanted but it's not implemented. See #592

[RhetTbull]

I fixed the tutorial bug so try pipx upgrade osxphotos then osxphotos tutorial

[RhetTbull]

There is no --max-items option currently. I suggested that as a possible feature to add to do what you wanted but it's not implemented. See #592

--limit is now implemented which provides the ability to limit export to a max number of items. And it works with --update so you can do incremental exports.