--export-as-hardlink doesn't work with --download-missing (--update doesn't seem to be working?)

[narensankar0529]

I am using the following command line in a script that runs every 6 hours or so to export all media -

/usr/local/bin/osxphotos export --verbose --update --download-missing --use-photokit --not-hidden --not-shared
--export-as-hardlink --skip-original-if-edited --skip-raw --sidecar xmp
--directory '{media_type}/{created.year}/{created.month}' --filename '{media_type}{original_name}{created.date}_{created.sec}'
${libraryExportDir} >> $logFileName 2>&1

And it looks like the update doesn't happen in place now and instead is creating multiple copied of each media -

-rw-rw-r--@ 6 nsankar staff 1074088 Nov 15 19:12 photo_IMG_3759_2019-06-30_58 (1).HEIC
-rw-rw-r-- 1 nsankar staff 1572 Nov 25 00:01 photo_IMG_3759_2019-06-30_58 (1).HEIC.xmp
-rw-rw-r--@ 6 nsankar staff 1074088 Nov 15 19:12 photo_IMG_3759_2019-06-30_58 (2).HEIC
-rw-rw-r-- 1 nsankar staff 1572 Nov 25 06:07 photo_IMG_3759_2019-06-30_58 (2).HEIC.xmp
-rw-rw-r--@ 6 nsankar staff 1074088 Nov 15 19:12 photo_IMG_3759_2019-06-30_58 (3).HEIC
-rw-rw-r-- 1 nsankar staff 1572 Nov 25 12:19 photo_IMG_3759_2019-06-30_58 (3).HEIC.xmp
-rw-rw-r--@ 6 nsankar staff 1074088 Nov 15 19:12 photo_IMG_3759_2019-06-30_58 (4).HEIC
-rw-rw-r-- 1 nsankar staff 1572 Nov 25 19:49 photo_IMG_3759_2019-06-30_58 (4).HEIC.xmp

[RhetTbull]

I have this partially figured out but not entirely.

1. --download-missing doesn't work with --export-as-hardlink and I cannot make it easily do so (I'll update the app to abort with error if you try to do this). osxphotos reads a snapshot of the Photos library when it loads so any changes to the library that occur after you started Photos will be unknown to osxphotos. --download-missing uses either the AppleScript interface or PhotoKit API (if used with --use-photokit) to download missing images. If used without --use-photokit, --download-missing will download the missing image into the Photos library but osxphotos won't have access to the path until after osxphotos is re-run and reloads the library. So, instead, it uses AppleScript to copy the eported image (thus it won't work with --export-as-hardlink). But, the next time you run osxphotos export, the downloaded file should be in the library and the hardlink would work. This would result in two copies because in the export directory: the first one (not a hardlink) and the new one (a hardlink). On the other hand, --use-photokit does not guarantee a file is actually created in the library so it could result in a new file every time.

2. In your example above each of the HEIC appear to be hardlinks to the same image (inode count is 6). I'm not sure why this is happening as that indicates the original file is present. I've not been able to replicate this but will look into it. In my testing it appears --update is working for cases where the original file was not missing from the library (and thus osxphotos did not use PhotoKit to do the download).

3. Your filename template could result in duplicates if the filename and time is the same (I've got several cases of there where I duplicated an image).

To get around limitations of #1 I could add an option to force download of all missing images before the export (there's an example script in the examples folder on github) that does this. This would work with the AppleScript interface but not --use-photokit.

Do you have "Download originals to this Mac" set in your Photos preferences? If so, there should be few if any files not actually downloaded and thus you wouldn't need to use --download-missing. If you don't have this set, Photos could theoretically delete the files to free space thus forcing a new download with repeated exports.

[narensankar0529]

Yes I do have all Originals downloaded to this Mac set in the iCloud settings. But since there appeared to be some missing in the past I was using --download-missing at all times.

Is it possible to do a dry-run with --download-missing --use-photokit to make sure everything is actually downloaded first and the run the actual export with hardlinks? This would guarantee that all photos were actually present before export.

For the filename template - my assumption was that using the original filename would prevent conflicts. Is it possible to add a sequence number to your templating system which will then guarantee non-collisions?

Basically my use case is as follows - I am exporting both my library and my wife's library to folders and then importing them to Synology NAS for display. Since we both take pictures at the same time there are a lot of files with the same metadata that conflicts. So my hope was to use the some way to differentiate and I settled on the original file name. I tried to use a our names in the template but that did not work.

'{media_type}$username{created.date}_{created.sec}'

Not sure if I used it incorrectly

[narensankar0529]

Ideally using the PhotoKit interface for downloading would be great because right now osxphotos is rock-solid and doesn't crash at all

[RhetTbull]

Adding a sequence number would be difficult but osxphotos will add (1), (2) etc and will keep the two files straight (it knows which file goes with which photo).

To get the user name in the template you could do this:

--filename "${USER}_{original_name}"

You need to use double quotes to cause the shell to expand ${USER} to the username before it is passed to osxphotos.

I'll think about the possibility of adding a two phase export that first downloads missing files then runs again to do the hardlinks. Unfortunately this won't work with --use-photokit. While the PhotoKit interface is much better for performance and stability, it does not always download the file into the Photos library so there's nothing to hardlink against. For regular photos, it will download them but for special types like slo-mo video, live photos, etc it does not guarantee the photos end up in the user's library and I can't do anything about that (it's Apple's design).

The alternative to use AppleScript does work but as you've likely noticed, Photos crashes a lot because the AppleScript interface is buggy (Apple's problem but they don't seem interested in fixing it...I suspect they may drop AppleScript support in future MacOS release as they've basically stopped supporting it).

Why not skip the hardlinks in folders and just export a copy directly to the NAS?

[RhetTbull]

Also, are you exporting your wife's library and yours into the same folder? If so, I'm not sure how osxphotos will handle this as two differently libraries would be using the same export database. I think it should work but I've not tested this use case.

[narensankar0529]

The downloads are into separate folders. I am running multi-user on my Mac and downloading in her account and mine.

The reason for the hard links is because our libraries are pretty big - together around 800GB so not enough space on the Mac for now. and I found that exporting to a SMB mounted folder is excruciatingly slow. However export as hard links on the other hand takes only around 5 to 6 minutes for both of us. And then sync to the NAS in the background using Resilio sync is around 10 times faster.

It would be great if you could add some information to the log to indicate there are missing downloads so then I can script around it using your python example. So don't need to do a two step download. And if you are able to implement the --cleanup option then it can be tied together externally. I can script it to avoid duplicates.

[RhetTbull]

It would be great if you could add some information to the log to indicate there are missing downloads

I am working on adding this information to the new --report option (#275)

If you use --verbose without --download-missing, you'll get a message like this in the verbose output:

Skipping missing photo IMG_4948.JPG

[RhetTbull]

Try the following which I think will do close to what you want. You may need to delete your export folder and start over -- I'm still not sure why you got the duplicate exports but suspect some interaction between --use-photokit and --export-as-hardlink

First, get list of missing files by unique ID (UUID):

/usr/local/bin/osxphotos query --missing | tail -n +2 | cut -d"," -f1 > missing.txt

This will write a list of missing UUIDs to missing.txt

Then download just the missing files but don't use hardlinks:

/usr/local/bin/osxphotos export --verbose --update (rest of your options here) --uuid-from-file missing.txt --download-missing (--use-photokit if desired)

This will download and export only the images in the missing.txt file

Finally, run your usual export command but without --download-missing:

/usr/local/bin/osxphotos export --verbose --update --not-hidden --not-shared
--export-as-hardlink --skip-original-if-edited --skip-raw --sidecar xmp
--directory '{media_type}/{created.year}/{created.month}' --filename '{media_type}{original_name}{created.date}_{created.sec}'
${libraryExportDir} >> $logFileName 2>&1

This will export copies of only the missing images then hardlink everything else

[RhetTbull]

I added a "--missing" flag to v0.37.2. You can thus do

osxphotos export /path/to/export --missing --download-missing

To download just the missing photos thus eliminating step 1 above and simplifying step 2.

It must be used with --download-missing (since it only tries to download the missing images)

[RhetTbull]

Note that if your Mac is running the APFS filesystem (default for Mojave+) then you don't get much advantage from hardlinks as osxphotos now uses the APFS "copy on write" ability that provides quick copy with no additional space used on the disk.