Library Version 2

[steilerDev]

Describe the ultimate goal you want to achieve

The current local file structure has certain design goals, which initially seems to have made sense, but based on the community feedback are resulting in some challenges, when it comes to use by a wider audience, requiring more flexibility and extensibility.

Additional some challenges (around filetype extensions) have surfaced that might be impossible to solve with the current structure.

Key goals:

• Extensibility - Support for multiple files per picture. Addresses support for Live Photos (Live Photo Support #121) and support for different resolution (iCloud China region support and 'resOriginalAltRes' syncing support. #279)
• Removing file system flakiness that e.g. leads to modification time not matching iCloud modification time (LIBRARY_ASSET_MODIFICATION_TIME: File's modification time does not match iCloud record #342)
• Streamline loading & validating local library (Feature Request: Consistency check #291)
• No longer dependent on a static list of filetype extensions ("Warning: Detected unknown filetype (descriptor with ext)" #143)
• More robust (and deduplicated) archiving support to address feature requests like Feature Idea: Soft delete #314 and Archive older than XX days, continue syncing most recent #284

How do you think the feature should be implemented

Current proposal:

• Each asset would be stored as a folder, with adjacent metadata, where a user can specify which versions of the asset they want to sync (excluding e.g. live photos videos, or edited pictures, or only sync lower res versions of the file)
• The format of the metadata should mirror the iCloud Photos backend as of the state of the sync and should be the foundation of making diffing decisions
• Given I currently don't understand the iCloud provided checksum, the metadata file could hold our own checksum

assetUUID/
├─ resOriginalVidComplRes
├─ resOriginalRes
├─ metadata.json

• Those assets would be part of a raw data directory, that will also provide context on their lifecycle (either they are synced from the primary or shared library, or they were archived or even soft deleted - see Feature Idea: Soft delete #314)

library/
├─ .library-data/
│  ├─ primary/
│  │  ├─ <assetUUID1>/
│  ├─ shared/
│  │  ├─ ...
│  ├─ backup/
│  │  ├─ ...
│  ├─ deleted/
│  │  ├─ ...

• The final piece will be the user folders and meta albums (like 'All Photos', 'Shared Photos', 'Archived Photos')
• Each folder will contain its metadata (primarily its UUID and folder content)
• Since all asset metadata will be stored separately, we could arbitrarily name those files and give users the option to pass their preferred formatting string as well as link a specific version of the file (and/or multiple versions of a file, depending on the user's preference)

library/
├─ .library-data/
│  ├─ ...
├─ _All Photos/
│  ├─ ...
├─ _Shared Photos/
│  ├─ ...
├─ _Archived Photos/
│  ├─ ...
├─ <userFolder1>/
│  ├─ metadata.json
│  ├─ 2023-01-01 - someFile -> ../.library-data/primary/<assetUUID1>/resOriginalRes

Timeline & Expectations

This would require a massive rework, and I would need feedback from the community if a migration path would be necessary (which would increase the amount of work that needs to be done)

I will address some sync/library related issues first before tackling this initiative as part of a v2 release. So it would be great if we could have more inputs on this topic to come up with the best possible (long term) solution

[steilerDev]

Additional documentation

[steilerDev]

Ongoing issue in pyicloud about using the provided hash value - does not seem to be going anywhere though