Unstuck RAW iCloud sync

[PhilipTrauner]

I've consistently managed to break iCloud photo sync when importing (built-in Photos app import functionality) RAW photos.

The only reliable workaround to unstuck iCloud sync I've discovered so far, is to rotate all photos that are likely stuck, followed by undoing the rotation again.

osxphoto inspect reveals that the workaround results in the "Date modified" field being set for photos it was applied to, which presumably fixes something in the syncing machinery.

osxphotos has already helped me tremendously by listing which photos are actually stuck (osxphotos query --not-incloud --not-shared), as rotating unaffected photos, which have been edited externally, sometimes corrupts their sidecar file.

Now I was wondering if there's a way that osxphotos could be used instead of the workaround.

I already tried using your (pretty awesome) script to set location data based on a .gpx file, in hopes that a metadata change would unstuck syncing, but that doesn't appear to be the case.

Is there any operation which could potentially trigger a sync?
More than willing to share my photo database in case that's useful 😇

[RhetTbull]

I don't know how to force iCloud to upload the photo but if you change the time, it can force a metadata sync. (Though this might not work in your case as it sounds like you already tried changing metadata). If you want to give it a try, select an offending photo then run:

osxphotos timewarp --time-delta "+1s"

This adds a second to the time. You can then immediately run the reverse:

osxphotos timewarp --time-delta "-1s"

Which restores the time to what it was.

I'm glad the gpx script was helpful! I plan to incorporate that functionality into the osxphotos add-locations command.

[PhilipTrauner]

Here you go, thank you so much for doing this 😇

bad.txt
good.txt

[RhetTbull]

One clue:

Good:

ZASSET:
        'ZCLOUDLOCALSTATE': 1,
        'ZCLOUDPLACEHOLDERKIND': 4,
        'ZDEFERREDPROCESSINGNEEDED': 0,

Bad:

ZASSET:
        'ZCLOUDLOCALSTATE': 0,
        'ZCLOUDPLACEHOLDERKIND': 0,
        'ZDEFERREDPROCESSINGNEEDED': 4,

For the "bad" photo, ZCLOUDLOCALSTATE =0 indicates the photo is not synced to iCloud (which you know). What's interesting is the value ZDEFERREDPROCESSINGNEEDED=4 which is 0 for the "good" photo. I've not explored ZDEFERREDPROCESSINGNEEDED but in my library of 36k photos, there isn't a single one where this value is not 0. I'm wondering if that column means Photos thinks it needs to do something to the photo before uploading to iCloud?

Try the following commands. At the terminal, run:

sqlite3 ~/Pictures/Photos\ Library.photoslibrary/database/Photos.sqlite

Then type the following:

SELECT COUNT(*) FROM ZASSET WHERE ZDEFERREDPROCESSINGNEEDED <> 0;

Finally, you can press Ctrl+D to exit the sqlite prompt.

Let me know the count. I get 0:

sqlite> SELECT COUNT(*) FROM ZASSET WHERE ZDEFERREDPROCESSINGNEEDED <> 0;
0

If your count is close to the number of "stuck" photos, we may be on to something. Will just need to figure out what this means or maybe try to change one and see if it gets "unstuck".

[PhilipTrauner]

I currently have a single stuck photo in my library, and the query does indeed return 1 🙌

[RhetTbull]

Definitely a good clue then. Unfortunately I'm not sure what to do about it. How long as the photo been "stuck"?

A few more data points:

There's a "cloud metadata" record stored in ZCLOUDMASTERMEDIAMETADATA table. The "good" photo has this record completed as expected (below is a snippet of what's in there) but the "bad" photo has no record in this table. It may be this is computed on the cloud but I believe that nearly all processing of photos is done "on device" as Apple is very privacy sensitive. It could be some process that's supposed to update this metadata hasn't completed or crashed (and could be linked to the ZDEFERREDPROCESSINGNEEDED value).

'{DNG}': {
            'DNGBackwardVersion': [1, 2],
            'PreviewColorSpace': 2,
            'AnalogBalance': [1.0, 1.0, 1.0],
            'CalibrationIlluminant1': 17,
            'BaselineExposure': 0.3064727783203125,
            'BaselineSharpness': 1.0,
            'CalibrationIlluminant2': 21,
            'WhiteLevel': [14328],
            'BaselineNoise': 1.0,
            'BlackLevelRepeatDim': [2, 2],
            'ColorMatrix2': [

Other clues:

ZASSET.ZMASTER is null. This should be the foreign key to the ZCLOUDMASTER record for the photo. But it might be expected that's it null as there is no master photo in the cloud as the photo's not in the cloud.

ZASSET.ZMEDIAANALYSISATTRIBUTES is null. This is a foreign key to another table so whatever fills this in hasn't completed. (Maybe photoanalysisd though this isn't one of the values osxphotos currently uses to get the analysis attributes (category labels, etc.)). ZASSET.ZPHOTOANALYSISATTRIBUTES is not null so at least some analysis has been completed on the photo (this is also a foreign key).

I'll keep searching. It might be possible to manually change the value of ZDEFERREDPROCESSINGNEEDED to see if that unsticks it but there's a risk of corrupting something in the library as we don't fully understand what that means.

If this photo (R0000443.DNG) is one that you can share, would you mind sending to me at rturnbull+git@gmail.com and I will import it into a test library and monitor the database activity to see if it is perhaps something unique to these RAW photos themselves or something else.

[PhilipTrauner]

Just sent you a mail, some more findings from my end: Editing inside the Photos app and consequently undoing all edits also seems to cause the syncing machinery to kick into gear, which would suggest that adding and consequently removing a sidecar file also does something 😄