-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Migrate: Read metadata from Apple iPhoto / Photos app XMP #636
Comments
Excellent question, didn't test this yet - only with Google Photos, which provides JSON files in its export. Wondering how Apple stores metadata? Maybe inside image files. In that case, we might be able to just index it. |
I am working through the same hassle with a 1TB library, 99% IOS based photos and videos. |
Any reason you import / upload the files? Indexing the iPhoto library should be easier / faster? Deleting was not implemented as many users didn't want us to change anything in their library at all, that's also why we added the read-only mode. Guess we'll implement this now together with a config option to disabled it. Note that there already is a delete button for sidecar files. Transcoding is also on our todo. Financial support enables us to spend more time with this project and develop features faster. |
To your last point, I am considering to donate, however want to be sure where and for what. For me it's basically a long-term decision to move away from the vendor lock-in to a self-hosted library. I like the archive feature to move unwanted files to one place, However, there might be shots I don't need anymore (often a lot) and I prefer to keep my lib as clean as possible. Regarding the video transcoding - I am testing handbrake atm, which eventually could help for the interim to convert from 265 to 264 and to put the output in the photoprism import folder. Regarding first point - My workflow in this self-hosted setup will probably very simple.
Event though this flow is simple it forces me to decide on the photos to keep before I move them into Photoprism. Going there later via WebDav to delete photos is only a workaround and so far at leat my Win10 host struggles to connect. Didnt test it on my Mac yet. So my point is simplicity. Let me dump all my files into one import folder in the file system and then let me decide later in the photoprism app where I want to put photos and which I want to keep. |
Thanks for your detailed feedback! Helps a lot. Funding details are documented on https://docs.photoprism.org/funding/. |
I agree an option to delete would be ideal. My workflow is to use photoprism as a one stop shop - after initially importing my library (mostly complete) I will be auto uploading from my phone (primary source of photos) with my seafile server using the android/ios auto upload feature (and then webdav hourly sync between photoprism and seafile). My point is that I will have a lot of junk photos I will need to delete. Storage is cheap so archiving isn't terrible, but would prefer to remove permanently I too will be donating. I contribute a small monthly amount to another application I have found very useful. |
Before moving my entire library into the app - wanted to confirm whether the delete function is coming. Or is this still being reviewed? |
@swingstate We're going to implement #167, just can't tell you when exactly as there are ton of feature requests we need to work through (like face recognition and batch edit). |
ok, cool! Didn't mean to spam as I am still learning how GitHub works. I'll post a few more ideas shouldn't there be any existing entries. |
I'll give it a try and meanwhile imported the first chunk of 1500 Photos. Any chance I can map the Archive through WebDAV to delete unwanted photos? Or can I somehow list the files (filenames) which are in the archive? Even in the terminal would be fine. |
Our WebDAV implementation is based on the actual file system, not database contents. Guess we really need to implement #167 soon... |
@swingstate just a quick note regarding video conversion: When you have a graphics card it could be much faster to utilize it which is possible e.g. using ffmpeg. I recently tried several methods and using an Nvidia (GTX 1060) graphics card was by far the fastest solution (and also seemed to result in better quality compared with software encoding). |
I set up following .mov conversion flow: Upload via WebDav into Nextcloud -> Conversion with Handbrake Docker container -> import into photoprism.
… Am 02.12.2020 um 19:58 schrieb Sven ***@***.***>:
@swingstate <https://github.com/swingstate> just a quick note regarding video conversion: When you have a graphics card it could be much faster to utilize it which is possible e.g. using ffmpeg. I recently tried several methods and using an Nvidia (GTX 1060) graphics card was by far the fastest solution (and also seemed to result in better quality compared with software encoding).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#636 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARSCO2V22CMDUFOQ4R55NGDSS2E6VANCNFSM4T6YNSEQ>.
|
It actually stores metadata inside an sqlite database. The schema is undocumented though, but there are some interesting attributes, like different quality metrics and object detection stuff, which cannot really be exported. It would be nice if Photoprism would support Apple Photos.app databases, but I guess thats not realistically going to happen, as the undocumented schema of the database could change at any point, if apple wants it. |
I assume this project would help if the goal are to import from Photos.app: https://github.com/RhetTbull/osxphotos |
TL;DR iPhoto/Photos has two options to export:
Option 1 will result in process of files, ie if you have RAW files it will only export the jpeg or if you have 4k videos it will convert to 720p by default. This is likely not a good option as all of this processing can be handled by Photoprism already (as of 2021/03/20) and just results in quality loss and taking more time to export. Option 2 will provide you the original files and provides several options during the export process as defined here (https://support.apple.com/en-ca/guide/photos/pht6e157c5f/mac). Then folders icon can be selected on the side bar and then select all albums in the main window, then you can execute this option. Selecting File Name as "Album Name with Number" and Subfolder Format as "Moments" will result in a single folder for each album and all photos being placed in dated subfolders with unique names. Orphan files (ie not in an album) can be found by creating a Smart Album, File>New Smart Album, and add the Filter "Album" "is not" "Any" The library I have been trying this with is ~65k items spread over 20 years and about 400GB in size. Items have been well organized within albums and coming from over few dozen phones and about a dozen still cameras (P&S, SLR, Mirrorless). Using the option 2 I outlined resulted in a fairly painless change between Apple Photos and Photoprism but took around a day to export from Photos and then another day to index by Photoprism The only issue I've encountered is that the date/time is either not being read correctly by Photoprism and written correctly by Apple Photos on a few hundred photos. I believe the date issue could be fixed easily since the photos are stored in a dated folder, time is likely a lost cause but really only impacts events where you are trying to align multiple sources (ie a wedding with personal DSLR photos, phone photos, and professional will all be off by a few minutes due to their internal clocks being set differently) There is an option to also export an XMP sidecar file, which is likely the most straight forward way to get the metadata without having to create an "Apple" specific method, however it seems like XMP is not supported yet with Photoprism. |
@simoncoul thank you for sharing this :) |
@graciousgrey here is an example of the issue.
This particular photo was a downloaded proof from the photographer and I believe the 2018/02/21 is the download date or date the photo was edit (watermarked), but again this is not the date I organized it to in Photos or what the XMP is showing. EDIT I reviewed all of the photos that are showing this issue (~2k) and they all have a common element, they are all from a 3rd party source (ie not a photo that was taken directly by me). They are usually downloaded from a website (professional photographer, social media, etc) or scanned from a physical print. Again within Photos they were all organized correctly the the export subfolder shows the correct date (the XMP likely does but I don't want to review 2k XMP files...) |
@simoncoul The jpeg file itself does not have a taken at or create date in its exif data. It only has the "FileModifyDate": "2018:02:22 02:30:48+01:00". That might be the same for the other photos from 3rd party sources. The create date is saved in the xmp: PhotoPrism does not read this xmp field yet. What PhotoPrism does is using the import date as create date when using “import” or using the FileModifyDate as date when using “index”. Regarding your previous workflow - did you
Or did you
In case Apple created the xmp file, could you check some more xmp files to see whether all of them use this tag photoshop:DateCreated2018-01-30T20:30:48-05:00</photoshop:DateCreated> ? Regarding using the folder name to extract the date. We already do this when we see paths like |
@graciousgrey I use the first workflow you outlined (steps 1-4). I checked several other XMP files and they all use the photoshop:DateCreated tag. Regarding the subfolder I found a few other items:
In the attached example is a RAW file with all the metadata fields completed so you can see how they look coming out of Photos (include face ID, geotagging, etc). |
@simoncoul thank your for providing these examples! I will create a separate ticket to support the DateCreated field Apple uses. Can I attach the two xmp files from you to this issue? |
@graciousgrey Yes that is no issue. Please let me know if you want me to test your solution once it is ready. |
@simoncoul our latest preview now supports reading: title, description, keywords and takenAt date from the apple photos xmp. We've updated our docs as well: https://docs.photoprism.org/user-guide/use-cases/apple/ Besides interpreting GPS information from the xmp created by apple is there something else to do before we can close this issue? |
@graciousgrey I've updated my unraid docker today and reindex a couple album and it appears to correct read the date/time on the XMP! I will reindex my entire library and let you know if there are any issues. One item you may want to add to the docs is to select Subfolder Format: Moment Name. This will result all of your Apple Photo albums being placed into individual folders and then this folders can be converted back into albums in PhotoPrism. I've attached screenshots of the menus which way also help a new user. The only other item within the XMP that may be useful is People identification. I know facial recognition is on your roadmap, but even knowing who is in the photo right now would be useful for sorting and searching (ie find all pictures of my child). Currently it takes this field and put's into the keywords but truncates it to the only the first name, so in the Example_2 I uploaded previously Halle Coulson becomes just a keyword halle. |
Hi there, I'm experimenting in a similar way on this. There seems to be a timezone bug at play here still. I have the following exported apple live photo (HEIC, MOV, and XMP file). This photo was taken at 5:48:27pm on 2022-09-03 in Seattle (PDT timezone = UTC-7) Import log:
exiftool info:
XMP contents
The metadata in the photoprism web-ui shows this as taken at 00:48:27 on 2022-09-04 in the Running from Docker, on a Synology NAS. File was imported by dropping it in the import folder and hitting the import button in the UI. P.s. Just wanted to say photoprism really looks great. Noted on the support options. Let me know if there's a place I can help on this fix. (Especially if there's some background that would help me to understand how timezone logic has changed over time). Happy to shoot the files mentioned offline, I picked this file as it wasn't really anything particularly private. |
@joshka Thanks for reporting! We'll let you know once we have a fix for this. |
Signed-off-by: Michael Mayer <michael@photoprism.app>
How about we close this issue as done and create a new issue for reading GPS information? |
Nobody complained so far, so I'll move ahead and close this as done! |
I have a rather large photo and video library that spans a few decades and is over 2TB in size. Up to now I've been managing it with Apple's ecosystem, but the time has come to move away from them for various reasons.
Is there a sane migration path that exists to facilitate taking all of the contents of an iPhoto / Photos.app library over into a PhotoPrism setup?
Acceptance Criteria:
The text was updated successfully, but these errors were encountered: