Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate: Read metadata from Apple iPhoto / Photos app XMP #636

Closed
3 of 4 tasks
ylluminate opened this issue Nov 22, 2020 · 28 comments
Closed
3 of 4 tasks

Migrate: Read metadata from Apple iPhoto / Photos app XMP #636

ylluminate opened this issue Nov 22, 2020 · 28 comments
Assignees
Labels
idea Feedback wanted / feature request released Available in the stable release

Comments

@ylluminate
Copy link

ylluminate commented Nov 22, 2020

I have a rather large photo and video library that spans a few decades and is over 2TB in size. Up to now I've been managing it with Apple's ecosystem, but the time has come to move away from them for various reasons.

Is there a sane migration path that exists to facilitate taking all of the contents of an iPhoto / Photos.app library over into a PhotoPrism setup?


Acceptance Criteria:

  • Read title from Apples XMP
  • Read keywords from Apples XMP
  • Read takenAt from Apples XMP
  • Read GPS information from Apples XMP
@lastzero
Copy link
Member

Excellent question, didn't test this yet - only with Google Photos, which provides JSON files in its export. Wondering how Apple stores metadata? Maybe inside image files. In that case, we might be able to just index it.

@graciousgrey graciousgrey added the technical-support Support request label Nov 23, 2020
@swingstate
Copy link

I am working through the same hassle with a 1TB library, 99% IOS based photos and videos.
I am testing imports through the web-based import and even with a chunk of 4000 files at once this worked well. Import of HEIC photos works very well.
However, photoprism does not seem to support the video codecs which Apple uses to shot. Hence, I am not sure if I'll end with this solution. Going through another step to convert videos first and then upload doesn't make sense to me resp. I want to automate this.
Be aware, the WebUI doesn't allow to delete photos (I don't get why as this is a basic file management task since Windows 3.x) However, deleting files seem to work through WebDAV which I still need to test.

@lastzero
Copy link
Member

Any reason you import / upload the files? Indexing the iPhoto library should be easier / faster?

Deleting was not implemented as many users didn't want us to change anything in their library at all, that's also why we added the read-only mode. Guess we'll implement this now together with a config option to disabled it. Note that there already is a delete button for sidecar files.

Transcoding is also on our todo. Financial support enables us to spend more time with this project and develop features faster.

@swingstate
Copy link

To your last point, I am considering to donate, however want to be sure where and for what. For me it's basically a long-term decision to move away from the vendor lock-in to a self-hosted library. I like the archive feature to move unwanted files to one place, However, there might be shots I don't need anymore (often a lot) and I prefer to keep my lib as clean as possible.

Regarding the video transcoding - I am testing handbrake atm, which eventually could help for the interim to convert from 265 to 264 and to put the output in the photoprism import folder.

Regarding first point - My workflow in this self-hosted setup will probably very simple.

  1. I shoot with the Iphone and use an app to automatically upload via SSH connection to an folder on my UnRaid host.
  2. I browse the folder with a Webbased filebrowser to scan through the photos and remove the stuff I don't want in my library
  3. I move the photos which I want to keep to the photoprism import folder (didn't test that feature yet, hope it exists)
  4. I sort the imported photos into albums

Event though this flow is simple it forces me to decide on the photos to keep before I move them into Photoprism. Going there later via WebDav to delete photos is only a workaround and so far at leat my Win10 host struggles to connect. Didnt test it on my Mac yet.

So my point is simplicity. Let me dump all my files into one import folder in the file system and then let me decide later in the photoprism app where I want to put photos and which I want to keep.
I think I get your point regarding read only and the UI is simple and clean. I don't aim to destroy that, hence if you could enable the delete function as an "to be activated" feature you would earn one more lasting customer. Video transcoding can be added later I think.

@lastzero
Copy link
Member

Thanks for your detailed feedback! Helps a lot. Funding details are documented on https://docs.photoprism.org/funding/.

@Engr-AllanG
Copy link

I agree an option to delete would be ideal. My workflow is to use photoprism as a one stop shop - after initially importing my library (mostly complete) I will be auto uploading from my phone (primary source of photos) with my seafile server using the android/ios auto upload feature (and then webdav hourly sync between photoprism and seafile). My point is that I will have a lot of junk photos I will need to delete. Storage is cheap so archiving isn't terrible, but would prefer to remove permanently

I too will be donating. I contribute a small monthly amount to another application I have found very useful.

@swingstate
Copy link

Before moving my entire library into the app - wanted to confirm whether the delete function is coming. Or is this still being reviewed?

@lastzero
Copy link
Member

@swingstate We're going to implement #167, just can't tell you when exactly as there are ton of feature requests we need to work through (like face recognition and batch edit).

@swingstate
Copy link

ok, cool! Didn't mean to spam as I am still learning how GitHub works. I'll post a few more ideas shouldn't there be any existing entries.

@swingstate
Copy link

I'll give it a try and meanwhile imported the first chunk of 1500 Photos. Any chance I can map the Archive through WebDAV to delete unwanted photos? Or can I somehow list the files (filenames) which are in the archive? Even in the terminal would be fine.

@lastzero
Copy link
Member

Our WebDAV implementation is based on the actual file system, not database contents. Guess we really need to implement #167 soon...

@shoeper
Copy link

shoeper commented Dec 2, 2020

@swingstate just a quick note regarding video conversion: When you have a graphics card it could be much faster to utilize it which is possible e.g. using ffmpeg. I recently tried several methods and using an Nvidia (GTX 1060) graphics card was by far the fastest solution (and also seemed to result in better quality compared with software encoding).

@swingstate
Copy link

swingstate commented Dec 2, 2020 via email

@lastzero lastzero changed the title Converting a large iPhoto / Photos.app library to PhotoPrism? Indexer: Support for Apple iPhoto / Photos app library metadata Dec 6, 2020
@lastzero lastzero added idea Feedback wanted / feature request and removed technical-support Support request labels Dec 6, 2020
@tbrtje
Copy link

tbrtje commented Jan 2, 2021

Excellent question, didn't test this yet - only with Google Photos, which provides JSON files in its export. Wondering how Apple stores metadata? Maybe inside image files. In that case, we might be able to just index it.

It actually stores metadata inside an sqlite database. The schema is undocumented though, but there are some interesting attributes, like different quality metrics and object detection stuff, which cannot really be exported. It would be nice if Photoprism would support Apple Photos.app databases, but I guess thats not realistically going to happen, as the undocumented schema of the database could change at any point, if apple wants it.
I would love to work on this though.

@toredash
Copy link

I assume this project would help if the goal are to import from Photos.app: https://github.com/RhetTbull/osxphotos

@simoncoul
Copy link

simoncoul commented Mar 20, 2021

TL;DR
Select all your albums, File>Export>Export Unmodified, file name "Album with Numbers" subfolder "Moments", choose the export folder as the originals location for Photoprism. Wait for export to complete and then index originals within Photoprism.

iPhoto/Photos has two options to export:

  1. File>Export>Export "XXX" Items
  2. File>Export>Export Unmodified Originals For "XXX" Items

Option 1 will result in process of files, ie if you have RAW files it will only export the jpeg or if you have 4k videos it will convert to 720p by default. This is likely not a good option as all of this processing can be handled by Photoprism already (as of 2021/03/20) and just results in quality loss and taking more time to export.

Option 2 will provide you the original files and provides several options during the export process as defined here (https://support.apple.com/en-ca/guide/photos/pht6e157c5f/mac). Then folders icon can be selected on the side bar and then select all albums in the main window, then you can execute this option. Selecting File Name as "Album Name with Number" and Subfolder Format as "Moments" will result in a single folder for each album and all photos being placed in dated subfolders with unique names.

Orphan files (ie not in an album) can be found by creating a Smart Album, File>New Smart Album, and add the Filter "Album" "is not" "Any"

The library I have been trying this with is ~65k items spread over 20 years and about 400GB in size. Items have been well organized within albums and coming from over few dozen phones and about a dozen still cameras (P&S, SLR, Mirrorless). Using the option 2 I outlined resulted in a fairly painless change between Apple Photos and Photoprism but took around a day to export from Photos and then another day to index by Photoprism

The only issue I've encountered is that the date/time is either not being read correctly by Photoprism and written correctly by Apple Photos on a few hundred photos. I believe the date issue could be fixed easily since the photos are stored in a dated folder, time is likely a lost cause but really only impacts events where you are trying to align multiple sources (ie a wedding with personal DSLR photos, phone photos, and professional will all be off by a few minutes due to their internal clocks being set differently)

There is an option to also export an XMP sidecar file, which is likely the most straight forward way to get the metadata without having to create an "Apple" specific method, however it seems like XMP is not supported yet with Photoprism.

@graciousgrey
Copy link
Member

@simoncoul thank you for sharing this :)
Regarding the files with date problems - if you send us an example file from your export to hello@photoprism.app we can have a closer look.
PhotoPrism already reads some common XMP fields. An example XMP file created from apple would help us as well!

@simoncoul
Copy link

simoncoul commented Mar 22, 2021

@graciousgrey here is an example of the issue.

  • Without XMP it will import the file with an unknown date, and then later change the date to the import day
  • With XMP it will import the file with a date of 2018/02/21, however within Photos it was organized to the date of 2018/01/30 and looking at the XMP it also shows the same date. The XMP does definitely help get closer, however as I mentioned in the wedding example before this type of behaviour can really mess up the order of an album.

This particular photo was a downloaded proof from the photographer and I believe the 2018/02/21 is the download date or date the photo was edit (watermarked), but again this is not the date I organized it to in Photos or what the XMP is showing.

EDIT I reviewed all of the photos that are showing this issue (~2k) and they all have a common element, they are all from a 3rd party source (ie not a photo that was taken directly by me). They are usually downloaded from a website (professional photographer, social media, etc) or scanned from a physical print. Again within Photos they were all organized correctly the the export subfolder shows the correct date (the XMP likely does but I don't want to review 2k XMP files...)

Example.zip

@graciousgrey
Copy link
Member

@simoncoul The jpeg file itself does not have a taken at or create date in its exif data. It only has the "FileModifyDate": "2018:02:22 02:30:48+01:00". That might be the same for the other photos from 3rd party sources.

The create date is saved in the xmp:
photoshop:DateCreated2018-01-30T20:30:48-05:00</photoshop:DateCreated>

PhotoPrism does not read this xmp field yet. What PhotoPrism does is using the import date as create date when using “import” or using the FileModifyDate as date when using “index”.

Regarding your previous workflow - did you

  1. Download the jpeg from a third party source
  2. Add the jpeg file to Apple Photos
  3. Within Apple Photos edit the date to the 30.01.2018
  4. Then export the photos from apple and apple created the xmp

Or did you

  1. Download the jpeg and xmp from a third party source
  2. Add both files to Apple photos
  3. Export all files from Apple

In case Apple created the xmp file, could you check some more xmp files to see whether all of them use this tag photoshop:DateCreated2018-01-30T20:30:48-05:00</photoshop:DateCreated> ?

Regarding using the folder name to extract the date. We already do this when we see paths like 2018/02.
We might support folder names like January 30, 2018 later as well but this is not as easy to implement because the name depends on the language.

@simoncoul
Copy link

@graciousgrey I use the first workflow you outlined (steps 1-4).

I checked several other XMP files and they all use the photoshop:DateCreated tag.

Regarding the subfolder I found a few other items:

  • If the photo is geotagged, it will include the location in the folder name (see attached example)
  • The system language will influence the folder structure as you stated, but also the order (see example, system was switched to french and the folder is 12 décembre 2010 but english was december 12, 2010)

In the attached example is a RAW file with all the metadata fields completed so you can see how they look coming out of Photos (include face ID, geotagging, etc).
Example_2.zip

@graciousgrey
Copy link
Member

@simoncoul thank your for providing these examples! I will create a separate ticket to support the DateCreated field Apple uses. Can I attach the two xmp files from you to this issue?

@simoncoul
Copy link

@graciousgrey Yes that is no issue. Please let me know if you want me to test your solution once it is ready.

@graciousgrey
Copy link
Member

@simoncoul our latest preview now supports reading: title, description, keywords and takenAt date from the apple photos xmp. We've updated our docs as well: https://docs.photoprism.org/user-guide/use-cases/apple/

Besides interpreting GPS information from the xmp created by apple is there something else to do before we can close this issue?

@simoncoul
Copy link

simoncoul commented Apr 27, 2021

@graciousgrey I've updated my unraid docker today and reindex a couple album and it appears to correct read the date/time on the XMP! I will reindex my entire library and let you know if there are any issues.

One item you may want to add to the docs is to select Subfolder Format: Moment Name. This will result all of your Apple Photo albums being placed into individual folders and then this folders can be converted back into albums in PhotoPrism. I've attached screenshots of the menus which way also help a new user.

The only other item within the XMP that may be useful is People identification. I know facial recognition is on your roadmap, but even knowing who is in the photo right now would be useful for sorting and searching (ie find all pictures of my child). Currently it takes this field and put's into the keywords but truncates it to the only the first name, so in the Example_2 I uploaded previously Halle Coulson becomes just a keyword halle.

Export_options
Export_menu

@graciousgrey graciousgrey changed the title Indexer: Support for Apple iPhoto / Photos app library metadata Migrate: Read Apple iPhoto / Photos app library metadata Nov 3, 2021
@graciousgrey graciousgrey changed the title Migrate: Read Apple iPhoto / Photos app library metadata Migrate: Read metadata from Apple iPhoto / Photos app XMP Nov 3, 2021
@joshka
Copy link

joshka commented Sep 9, 2022

Hi there, I'm experimenting in a similar way on this. There seems to be a timezone bug at play here still. I have the following exported apple live photo (HEIC, MOV, and XMP file). This photo was taken at 5:48:27pm on 2022-09-03 in Seattle (PDT timezone = UTC-7)

Import log:

INFO[2022-09-09T08:05:18Z] media: 'IMG_7701 (1).HEIC' was taken at 2022-09-04 00:48:27 +0000 UTC (meta)
INFO[2022-09-09T08:05:18Z] import: moving related mov file 'IMG_7701 (1).mov' to 2022/09/20220904_004827_4BD6151B.mov
INFO[2022-09-09T08:05:18Z] import: moving related xmp file 'IMG_7701 (1).xmp' to 2022/09/20220904_004827_4BD6151B.xmp
INFO[2022-09-09T08:05:18Z] import: moving main heif file 'IMG_7701 (1).HEIC' to 2022/09/20220904_004827_4BD6151B.heic

exiftool info:

$ exiftool IMG_7701\ * | grep -E '(Date|==)'
======== IMG_7701 (1).HEIC
File Modification Date/Time     : 2022:09:06 03:59:04-07:00
File Access Date/Time           : 2022:09:09 01:03:53-07:00
File Inode Change Date/Time     : 2022:09:06 03:59:04-07:00
Modify Date                     : 2022:09:03 17:48:27
Date/Time Original              : 2022:09:03 17:48:27
Create Date                     : 2022:09:03 17:48:27
GPS Date Stamp                  : 2022:09:04
Profile Date Time               : 2022:01:01 00:00:00
Create Date                     : 2022:09:03 17:48:27.083-07:00
Date/Time Original              : 2022:09:03 17:48:27.083-07:00
Modify Date                     : 2022:09:03 17:48:27-07:00
======== IMG_7701 (1).mov
File Modification Date/Time     : 2022:09:06 03:59:05-07:00
File Access Date/Time           : 2022:09:09 01:03:53-07:00
File Inode Change Date/Time     : 2022:09:06 03:59:05-07:00
Create Date                     : 2022:09:04 00:48:26
Modify Date                     : 2022:09:04 00:48:27
Track Create Date               : 2022:09:04 00:48:26
Track Modify Date               : 2022:09:04 00:48:27
Media Create Date               : 2022:09:04 00:48:26
Media Modify Date               : 2022:09:04 00:48:27
Creation Date                   : 2022:09:03 17:48:26-07:00
======== IMG_7701 (1).xmp
File Modification Date/Time     : 2022:09:06 03:59:05-07:00
File Access Date/Time           : 2022:09:09 01:03:53-07:00
File Inode Change Date/Time     : 2022:09:06 03:59:05-07:00
GPS Date/Time                   : 2022:09:04 00:48:27Z
Date Created                    : 2022:09:03 17:48:26-07:00

XMP contents

$ cat IMG_7701\ \(1\).xmp
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 6.0.0">
   <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
      <rdf:Description rdf:about=""
            xmlns:exif="http://ns.adobe.com/exif/1.0/"
            xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/">
         <exif:GPSHPositioningError>35</exif:GPSHPositioningError>
         <exif:GPSImgDirection>193.66278065630397</exif:GPSImgDirection>
         <exif:GPSAltitudeRef>0</exif:GPSAltitudeRef>
         <exif:GPSAltitude>63.631790123456788</exif:GPSAltitude>
         <exif:GPSLatitudeRef>N</exif:GPSLatitudeRef>
         <exif:GPSLatitude>47.675403333333335</exif:GPSLatitude>
         <exif:GPSLongitudeRef>W</exif:GPSLongitudeRef>
         <exif:GPSLongitude>122.31739166666667</exif:GPSLongitude>
         <exif:GPSImgDirectionRef>T</exif:GPSImgDirectionRef>
         <exif:GPSSpeed>0.0</exif:GPSSpeed>
         <exif:GPSSpeedRef>K</exif:GPSSpeedRef>
         <exif:GPSTimeStamp>2022-09-04T00:48:27Z</exif:GPSTimeStamp>
         <photoshop:DateCreated>2022-09-03T17:48:26-07:00</photoshop:DateCreated>
      </rdf:Description>
   </rdf:RDF>
</x:xmpmeta>

The metadata in the photoprism web-ui shows this as taken at 00:48:27 on 2022-09-04 in the America/Los Angeles timezone. So it looks like there's something that is using the UTC time rather than local, but showing as if it's local regardless.

Running from Docker, on a Synology NAS. File was imported by dropping it in the import folder and hitting the import button in the UI.

P.s. Just wanted to say photoprism really looks great. Noted on the support options. Let me know if there's a place I can help on this fix. (Especially if there's some background that would help me to understand how timezone logic has changed over time).

Happy to shoot the files mentioned offline, I picked this file as it wasn't really anything particularly private.

@graciousgrey
Copy link
Member

@joshka Thanks for reporting! We'll let you know once we have a fix for this.

lastzero added a commit that referenced this issue Oct 31, 2022
Signed-off-by: Michael Mayer <michael@photoprism.app>
@lastzero
Copy link
Member

How about we close this issue as done and create a new issue for reading GPS information?

@lastzero lastzero added the please-test Ready for acceptance test label Oct 31, 2022
@lastzero lastzero self-assigned this Oct 31, 2022
@lastzero
Copy link
Member

lastzero commented Nov 4, 2022

Nobody complained so far, so I'll move ahead and close this as done!

@lastzero lastzero closed this as completed Nov 4, 2022
@lastzero lastzero added released Available in the stable release and removed please-test Ready for acceptance test labels Nov 4, 2022
@lastzero lastzero moved this to Released 🌈 in Roadmap 🚀✨ Jun 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
idea Feedback wanted / feature request released Available in the stable release
Projects
Status: Release 🌈
Development

No branches or pull requests

10 participants