Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

error opening osxphotos export db during export #1435

Closed
jasonhollis opened this issue Mar 6, 2024 · 24 comments
Closed

error opening osxphotos export db during export #1435

jasonhollis opened this issue Mar 6, 2024 · 24 comments

Comments

@jasonhollis
Copy link

I'm not sure what the bug is but I don't think it likes --keyword "álainn" My destination is also a folder called álainn but álainn is listed as a keyword when I run osxphotos info.

osxphotos export /Volumes/Jason/álainn --exiftool --update --download-missing --finder-tag-keywords --keyword “álainn”
Using last opened Photos library: /Users/jason/Pictures/Photos Library.photoslibrary
Something went wrong and osxphotos encountered an error:
disk I/O error
Crash log written to '/Volumes/Jason/Photos/osxphotos_crash.log'
Please file a bug report at https://github.com/RhetTbull/osxphotos/issues with the crash log attached.

I expected it to extract all of the photos with the keyword "álainn" into /Volumes/Jason/álainn

MacOS 14.3.1 (23D60)
You have the latest version of osxphotos: 0.67.6
@jasonhollis
Copy link
Author

osxphotos_crash.log

@RhetTbull
Copy link
Owner

This isn't due to the keyword or the unicode characters in the keyword. I created a folder named álainn and a keyword with the same and then ran your export command and it worked fine (see below). There's at least 2 problems;

  1. You used "smart quotes" with your keyword --keyword “álainn” passed the keyword as “álainn” not as álainn, in other words, it included the quotes. I'm guessing you typed the command first in TextEdit or Notes then pasted it to Terminal. Unless explicitly disabled, MacOS will convert the regular " mark to a "smart quote" “ also known as a curly quote. The shell and thus osxphotos see this as just another character, not a quote and include it as part of the keyword. OSXPhotos will warn you if you do this in a file path but for things like keywords, it assumes you meant what you typed.
  2. The error your getting has nothing to do with the keyword (even though the keyword argument was malformed) but looks like an error occurred during creation or migration of the export database as crash occurs in the code that is opening the export database which OSXPhotos uses for tracking which photos have been exported.

Were you exporting into a previously used export location? That is, you had run osxphotos export in the past to export photos here and now you want to export new photos using the --update? If not, the easiest thing to do is to delete the export database (There's a file .osxphotos_export.db in the folder /Volumes/Jason/álainn that you should delete). You won't see this in Finder as it's hidden so open the folder in Finder, then press shift+command+. (shift command dot) This will toggle the "view dot files" feature in Finder. Find .osxphotos_export.db and delete it. Alternatively, in the Terminal, type: rm /Volumes/Jason/álainn/.osxphotos_export.db*

If this was a previous export and you want to try preserving the export state, you could try these command:

osxphotos exportdb --repair Volumes/Jason/álainn
osxphotos exportdb --upgrade Volumes/Jason/álainn

And run the command again (this time without the smart quotes).

Alternatively, you could delete the entire export directory and re-run the original export.

I'm not sure what happened to get the database into this state though. Did osxphotos crash on you at some point prior to this? Did you cancel an export? Was this the first time running the export in this folder? Any other data you can provide would help me track down the source of the actual issue.

osxphotos export ~/Desktop/export/álainn --exiftool --update --download-missing --finder-tag-keywords --keyword álainn --verbose
osxphotos version: 0.67.6
Python version: 3.11.5 (main, Aug 25 2023, 01:57:11) [Clang 14.0.3 (clang-1403.0.22.14.1)]
Platform: macOS-13.5.1-x86_64-i386-64bit, 13.5.1
Verbose level: 1
exiftool path: /usr/local/bin/exiftool
Using last opened Photos library: /Users/rhet/Pictures/Test-13.0.0 copy.photoslibrary
Using export database /Users/rhet/Desktop/export/álainn/.osxphotos_export.db
Processing database /Users/rhet/Pictures/Test-13.0.0 copy.photoslibrary/database/photos.db
Processing database /Users/rhet/Pictures/Test-13.0.0 copy.photoslibrary/database/Photos.sqlite
Processing database.
Database version: 6000, 8.
Processing persons in photos.
Processing detected faces in photos.
Processing albums.
Processing keywords.
Processing photo details.
Processing import sessions.
Processing additional photo details.
Processing face details.
Processing photo labels.
Processing EXIF details.
Processing computed aesthetic scores.
Processing comments and likes for shared photos.
Processing moments.
Processing syndication info.
Processing shared iCloud library info
Done processing details from Photos library.
Exporting 2 photos to /Users/rhet/Desktop/export/álainn...
Exporting wedding.jpg (E9BC5C36-7CD1-40A1-A72B-8B8FAC227D51.jpeg) (1/2)
Writing metadata with exiftool for wedding.jpg
Exported wedding.jpg to /Users/rhet/Desktop/export/álainn/wedding.jpg
Exported new file /Users/rhet/Desktop/export/álainn/wedding.jpg
Exporting edited version of wedding.jpg (E9BC5C36-7CD1-40A1-A72B-8B8FAC227D51.jpeg)
Writing metadata with exiftool for wedding_edited.jpeg
Exported wedding.jpg to /Users/rhet/Desktop/export/álainn/wedding_edited.jpeg
Exported new file /Users/rhet/Desktop/export/álainn/wedding_edited.jpeg
Writing Finder tags to /Users/rhet/Desktop/export/álainn/wedding.jpg
Writing Finder tags to /Users/rhet/Desktop/export/álainn/wedding_edited.jpeg
Skipping Finder tags for /Users/rhet/Desktop/export/álainn/wedding_edited.jpeg: nothing to do
Skipping Finder tags for /Users/rhet/Desktop/export/álainn/wedding.jpg: nothing to do
Exporting exif1.jpg (407A44AE-30AF-4E54-8549-9DB0B1B48861.jpeg) (2/2)
Writing metadata with exiftool for exif1.jpg
Exported exif1.jpg to /Users/rhet/Desktop/export/álainn/exif1.jpg
Exported new file /Users/rhet/Desktop/export/álainn/exif1.jpg
Writing Finder tags to /Users/rhet/Desktop/export/álainn/exif1.jpg
Skipping Finder tags for /Users/rhet/Desktop/export/álainn/exif1.jpg: nothing to do
Exporting 2 photos ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Processed: 2 photos, exported: 3, updated: 0, skipped: 0, updated EXIF data: 3, missing: 0, error: 0
Elapsed time: 0:00:01
Cleaning up lock files

@RhetTbull RhetTbull changed the title Don't think it likes Irish error opening osxphotos export db during export Mar 6, 2024
This was referenced Mar 6, 2024
Open
Open
RhetTbull added a commit that referenced this issue Mar 6, 2024
@RhetTbull
Fix for improperly initialized export db, see #1435
7f41aed
@jasonhollis
Copy link
Author

jasonhollis commented Mar 6, 2024 via email

@jasonhollis
Copy link
Author

jasonhollis commented Mar 6, 2024 via email

@oPromessa
Copy link
Contributor

Hey there. On scheduling osxphotos to run see discussion #1080

@oPromessa
Copy link
Contributor

oPromessa commented Mar 6, 2024 •
edited

Me again.

On your topic

The second thought I had you might find interesting. I am leveraging a NAS for the export. I noticed in your docs the notion of using a ramdisk for the .osxphotos_export.db. I was considering a launch script that copies \Volumes\Jason\Photos.osxphotos_export.db to the ramdisk, then runs the OSXPhotos operation and then copies the db back to the NAS on completion.

I'll let Rhet comment further but I believe that's what osxphotos already does.

I also export to NAS. Besides using --ramdb I also use --exportdb option and place the exportdb in the local Mac disk and not on the NAS. This was more relevant prior to --ramdb but still avoids some NAS traffic and the beginning and end of the run.

You must however keep track of the location of your exportdb.

@RhetTbull
Copy link
Owner

RhetTbull commented Mar 6, 2024 •
edited

I was actually running a second instance in parallel to a much bigger job on the same machine, which is probably why the database issue occurred.

The database should be safe for access in parallel but this isn't an explicitly supported use case. I've added code to catch this error and attempt to repair the database but that's not been released yet.

Yesterday I was trying to work out the simplest way to schedule OSXPhotos to run on a routine basis using the --update switch.

See this discussion where @rajscode outlines the steps he uses to do this

I am building a companion app that has a GUI and will allow you to schedule periodic exports. I'll announce it on the Discussions page when ready for testing. My plan is for this to be a paid app to support osxphotos development.

I am leveraging a NAS for the export. I noticed in your docs the notion of using a ramdisk for the .osxphotos_export.db.

If using a NAS, you should use the --ramdb which copies the export database to RAM during the export then back to the NAS when done. You can also use --exportdb with or without --ramdb to specify an alternate path to the database (Eg on an internal disk). If you do this you should move the .osxphotos_export.db to the new location first.

but I'm guessing you don't build out the DB in an internal array because of memory use?

Not exactly. OSXPhotos is actually a memory hog because it loads the whole Photos database into memory. It's just that SQLite is so much easier to work with as a persistent format. There's also a whole set of tools for working with the export database under osxphotos exportdb and it can be opened in any SQLite editor. But it has very poor performance on a NAS.

Is it possible to change the creation date or the modified date in the exported files to match the creation date from the meta data in Photos?

You can set modification time to match the time in Photos with --touch-file. It is not (easily) possible to change creation date on APFS which is the default for MacOS. Your NAS might be able to but that's not the primary export target.

@oPromessa
Copy link
Contributor

oPromessa commented Mar 6, 2024 •
edited

On your topic

I have exiftool setup. Is it possible to change the creation date or the modified date in the exported files to match the creation date from the meta data in Photos?

That's the purpose of the --exiftool and --touch options. Give --touch option a go.

@jasonhollis
Copy link
Author

jasonhollis commented Mar 6, 2024 via email

@oPromessa
Copy link
Contributor

On your topic

OSXPhotos to run on a routine basis using the --update switch

See also --cleanup option and comments on #1180 in case you want to delete exported files you've deleted from the Photos app.

@jasonhollis
Copy link
Author

jasonhollis commented Mar 6, 2024 via email

@jasonhollis
Copy link
Author

jasonhollis commented Mar 6, 2024 via email

@jasonhollis
Copy link
Author

jasonhollis commented Mar 6, 2024 via email

@jasonhollis
Copy link
Author

jasonhollis commented Mar 7, 2024 via email

@jasonhollis
Copy link
Author

jasonhollis commented Mar 7, 2024 via email

@jasonhollis
Copy link
Author

jasonhollis commented Mar 7, 2024 via email

@jasonhollis
Copy link
Author

jasonhollis commented Mar 7, 2024 via email

@RhetTbull
Copy link
Owner

So that comes out to about 7 photos per second. For comparison, my M1 mini exporting to fast SSD via Thunderbolt 4 gets about 5 photos per second on update.

I don't spend a lot of time chasing performance but will fix obvious things. (For example added an index to export database recently that helped a lot). I view export/update as a task that can run in background or at night and I don't care how long it takes.

@jasonhollis
Copy link
Author

jasonhollis commented Mar 7, 2024 via email

@RhetTbull
Copy link
Owner

0:43:29❌️ exiftool error for file /Volumes/jasonfast/Photos/_/RenderedImage (27).HEIC:
Error: Not a valid HEIC (looks more like a JPEG) -
/var/folders/sn/rsf5ll9d023f2lw0rx9mbh800000gn/T/osxphotos_export__f90__ne/F1CBE
2D9-9F4A-437E-BCE8-46A3880A71BD_exiftool.heic

These errors are from exiftool and I've never found an instance where exiftool was wrong. If exiftool says it's a JPEG then it likely is. This usually happens when a photo is imported from or edited in an external app that didn't follow the PhotoKit API spec as to how images were saved. They may be listed as one thing in the database (and by extension) but be something else. Photos app doesn't seem to care -- it'll happily display any photo it can read regardless of what the database or file extension says. There isn't currently much osxphotos can do about these. They'll still get exported but won't have the metadata embedded by exiftool.

I'm working on the ability to add a command that can export the photo, modify it (for example, fix the file extension) then replace the original in Photos with the new photo while preserving metadata. See #909 This has required reverse engineering some undocumented private APIs. I've made good progress but I only have about 10 hours/week to work on OSXPhotos so it's slow going. Most of time has been spent on the GUI job scheduler for OSXPhotos.

You can see a list of all photos that had errors in the export set via osxphotos exportdb /path/to/export/ --errors

@jasonhollis
Copy link
Author

jasonhollis commented Mar 7, 2024 via email

RhetTbull added a commit that referenced this issue Mar 9, 2024
@RhetTbull
Fix for improperly initialized export db, see #1435 (#1439)
e2343d7
@RhetTbull
Copy link
Owner

The original error due to malformed database should be fixed in 0.67.7

@jasonhollis
Copy link
Author

jasonhollis commented Mar 19, 2024 via email

@RhetTbull
Copy link
Owner

@jasonhollis this fixes the original error you encountered (opening an export database that is malformed generates error). It does not fix the Error: Not a valid HEIC (looks more like a JPEG) - errors as those come from exiftool, not OSXPhotos.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jasonhollis @oPromessa @RhetTbull