export with --dry-run parameter results in Error - RetryError[<Future at 0x10ae539d0 state=finished raised DataError>]

[RhetTbull]

Discussed in #1070

^{Originally posted by rajscode May 13, 2023}
Have a working export command. As soon as I add the --dry-run option it causes the command to not work anymore and reports an error:

Error Message and Crash Log

Loaded options from file /Volumes/WDSSD2TB/photo_library_export/osxphotos_backup.toml
osxphotos version: 0.60.0
Python version: 3.11.3 (main, Apr 7 2023, 20:13:31) [Clang 14.0.0 (clang-1400.0.29.202)]
Platform: macOS-13.3.1-arm64-arm-64bit, 13.3.1
Verbose level: True
Something went wrong and osxphotos encountered an error:
RetryError[<Future at 0x10ae539d0 state=finished raised DataError>]
Crash log written to '/Volumes/WDSSD2TB/photo_library_export/osxphotos_crash.log'
Please file a bug report at https://github.com/RhetTbull/osxphotos/issues with the crash log attached.

Crash Log contents:
osxphotos crash log
Created: 2023-05-13 15:16:54.611664
osxphotos version: 0.60.0
Platform: macOS-13.3.1-arm64-arm-64bit
Python version: 3.11.3 (main, Apr 7 2023, 20:13:31) [Clang 14.0.0 (clang-1400.0.29.202)]
sys.argv: ['/Users/userC/.local/bin/osxphotos', 'export', '/Volumes/WDSSD2TB/photo_library_export/photo_library_backup/', '--load-config', '/Volumes/WDSSD2TB/photo_library_export/osxphotos_backup.toml']
CRASH_DATA:
locals: {'ctx': <click.core.Context object at 0x1086f1c10>, 'cli_obj': <osxphotos.cli.cli.CLI_Obj object at 0x10100b3d0>, 'db': None, 'photos_library': (), 'add_exported_to_album': None, 'add_missing_to_album': None, 'add_skipped_to_album': None, 'added_after': None, 'added_before': None, 'added_in_last': None, 'album': (), 'album_keyword': False, 'alt_copy': False, 'append': False, 'beta': False, 'burst': False, 'cleanup': False, 'cloudasset': False, 'config_only': False, 'convert_to_jpeg': False, 'current_name': False, 'deleted': False, 'deleted_only': False, 'description': (), 'description_template': None, 'dest': '/Volumes/WDSSD2TB/photo_library_export/photo_library_backup/', 'directory': None, 'download_missing': False, 'dry_run': False, 'duplicate': False, 'edited': False, 'edited_suffix': None, 'exif': (), 'exiftool': False, 'exiftool_merge_keywords': False, 'exiftool_merge_persons': False, 'exiftool_option': (), 'exiftool_path': None, 'export_as_hardlink': False, 'export_by_date': False, 'exportdb': None, 'external_edit': False, 'favorite': False, 'favorite_rating': False, 'filename_template': None, 'finder_tag_keywords': False, 'finder_tag_template': (), 'folder': (), 'force_update': False, 'from_date': None, 'from_time': None, 'has_comment': False, 'has_likes': False, 'has_raw': False, 'hdr': False, 'hidden': False, 'ignore_case': False, 'ignore_date_modified': False, 'ignore_signature': False, 'in_album': False, 'incloud': False, 'is_reference': False, 'jpeg_ext': None, 'jpeg_quality': None, 'keep': (), 'keyword': (), 'keyword_template': (), 'label': (), 'limit': None, 'live': False, 'load_config': '/Volumes/WDSSD2TB/photo_library_export/osxphotos_backup.toml', 'location': False, 'max_size': None, 'min_size': None, 'missing': False, 'name': (), 'no_comment': False, 'no_description': False, 'no_keyword': False, 'no_likes': False, 'no_location': False, 'no_place': False, 'no_progress': False, 'no_title': False, 'not_burst': False, 'not_cloudasset': False, 'not_edited': False, 'not_favorite': False, 'not_hdr': False, 'not_hidden': False, 'not_in_album': False, 'not_incloud': False, 'not_live': False, 'not_missing': False, 'not_panorama': False, 'not_portrait': False, 'not_reference': False, 'not_screenshot': False, 'not_selfie': False, 'not_shared': False, 'not_slow_mo': False, 'not_time_lapse': False, 'only_movies': False, 'only_new': False, 'only_photos': False, 'original_suffix': None, 'overwrite': False, 'panorama': False, 'person': (), 'person_keyword': False, 'place': (), 'portrait': False, 'post_command': (), 'post_function': (), 'preview': False, 'preview_if_missing': False, 'preview_suffix': None, 'print_template': (), 'query_eval': (), 'query_function': (), 'ramdb': False, 'regex': (), 'replace_keywords': False, 'report': None, 'retry': None, 'save_config': None, 'screenshot': False, 'selfie': False, 'shared': False, 'sidecar': (), 'sidecar_drop_ext': False, 'skip_bursts': False, 'skip_edited': False, 'skip_live': False, 'skip_original_if_edited': False, 'skip_raw': False, 'skip_uuid': (), 'skip_uuid_from_file': None, 'slow_mo': False, 'strip': False, 'theme': None, 'time_lapse': False, 'timestamp': False, 'title': (), 'tmpdir': None, 'to_date': None, 'to_time': None, 'touch_file': False, 'update': False, 'update_errors': False, 'use_photokit': False, 'use_photos_export': False, 'uti': None, 'uuid': (), 'uuid_from_file': None, 'verbose_flag': 0, 'xattr_template': (), 'year': (), 'selected': False}
cfg: {'add_exported_to_album': None, 'add_missing_to_album': None, 'add_skipped_to_album': None, 'added_after': None, 'added_before': None, 'added_in_last': None, 'album': (), 'album_keyword': False, 'alt_copy': False, 'append': True, 'beta': False, 'burst': False, 'cleanup': True, 'cloudasset': False, 'convert_to_jpeg': False, 'current_name': False, 'db': '/Volumes/WDSSD2TB/Macbook_Pro_14_PhotosLibrary_Full_Originals_Backup/Photos Library.photoslibrary', 'deleted': False, 'deleted_only': False, 'description': (), 'description_template': None, 'directory': '{created.year}/{created.year}-{created.mm}', 'download_missing': False, 'dry_run': True, 'duplicate': False, 'edited': False, 'edited_suffix': None, 'exif': (), 'exiftool': False, 'exiftool_merge_keywords': False, 'exiftool_merge_persons': False, 'exiftool_option': (), 'exiftool_path': None, 'export_as_hardlink': False, 'export_by_date': False, 'exportdb': None, 'external_edit': False, 'favorite': False, 'favorite_rating': True, 'filename_template': None, 'finder_tag_keywords': False, 'finder_tag_template': (), 'folder': (), 'force_update': False, 'from_date': None, 'from_time': None, 'has_comment': False, 'has_likes': False, 'has_raw': False, 'hdr': False, 'hidden': False, 'ignore_case': False, 'ignore_date_modified': False, 'ignore_signature': False, 'in_album': False, 'incloud': False, 'is_reference': False, 'jpeg_ext': None, 'jpeg_quality': None, 'keep': (), 'keyword': (), 'keyword_template': ('Album: {folder_album}', '{label}', '{favorite?is_favorite}', 'is_{media_type}'), 'label': (), 'limit': None, 'live': False, 'location': False, 'max_size': None, 'min_size': None, 'missing': False, 'name': (), 'no_comment': False, 'no_description': False, 'no_keyword': False, 'no_likes': False, 'no_location': False, 'no_place': False, 'no_progress': False, 'no_title': False, 'not_burst': False, 'not_cloudasset': False, 'not_edited': False, 'not_favorite': False, 'not_hdr': False, 'not_hidden': False, 'not_in_album': False, 'not_incloud': False, 'not_live': False, 'not_missing': False, 'not_panorama': False, 'not_portrait': False, 'not_reference': False, 'not_screenshot': False, 'not_selfie': False, 'not_shared': False, 'not_slow_mo': False, 'not_time_lapse': False, 'only_movies': False, 'only_new': False, 'only_photos': False, 'original_suffix': None, 'overwrite': False, 'panorama': False, 'person': (), 'person_keyword': True, 'photos_library': (), 'place': (), 'portrait': False, 'post_command': (), 'post_function': (), 'preview': False, 'preview_if_missing': False, 'preview_suffix': None, 'print_template': (), 'query_eval': (), 'query_function': (), 'ramdb': False, 'regex': (), 'replace_keywords': False, 'report': '/Volumes/WDSSD2TB/photo_library_export/osxphotos_report_backup.sqlite', 'retry': None, 'screenshot': False, 'selected': False, 'selfie': False, 'shared': False, 'sidecar': ('xmp',), 'sidecar_drop_ext': True, 'skip_bursts': False, 'skip_edited': False, 'skip_live': False, 'skip_original_if_edited': False, 'skip_raw': False, 'skip_uuid': (), 'skip_uuid_from_file': None, 'slow_mo': False, 'strip': False, 'theme': 'dark', 'time_lapse': False, 'timestamp': False, 'title': (), 'tmpdir': '/Volumes/WDSSD2TB/photo_library_export/osxphotos_tmp', 'to_date': None, 'to_time': None, 'touch_file': False, 'update': True, 'update_errors': False, 'use_photokit': False, 'use_photos_export': False, 'uti': None, 'uuid': (), 'uuid_from_file': None, 'verbose': True, 'xattr_template': (), 'year': ()}
osxphotos version: 0.60.0
Error: RetryError[<Future at 0x1050b3050 state=finished raised DataError>]
Traceback (most recent call last):
File "/Users/userC/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/tenacity/init.py", line 382, in call
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/Users/userC/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/osxphotos/export_db.py", line 939, in _open_export_db
conn.cursor().executescript(dbdump.read())
sqlite3.DataError: query string is too large

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/userC/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/osxphotos/crash_reporter.py", line 36, in wrapped
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/userC/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/osxphotos/cli/export.py", line 1327, in export
export_db = ExportDBInMemory(dbfile=export_db_path, export_dir=dest)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/userC/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/osxphotos/export_db.py", line 876, in init
self._conn = self._open_export_db(self._dbfile)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/userC/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/tenacity/init.py", line 289, in wrapped_f
return self(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/Users/userC/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/tenacity/init.py", line 379, in call
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/userC/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/tenacity/init.py", line 326, in iter
raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x1050b3050 state=finished raised DataError>]

Tried it two different ways:
osxphotos export /Volumes/WDSSD2TB/photo_library_export/photo_library_backup/ --dry-run --verbose --load-config /Volumes/WDSSD2TB/photo_library_export/osxphotos_backup.toml

.toml config contents for above command

[export]
update = true
cleanup = true

report = "/Volumes/WDSSD2TB/photo_library_export/osxphotos_report_backup.sqlite"
append = true

db = "/Volumes/WDSSD2TB/Macbook_Pro_14_PhotosLibrary_Full_Originals_Backup/Photos Library.photoslibrary"

tmpdir = "/Volumes/WDSSD2TB/photo_library_export/osxphotos_tmp"

directory = "{created.year}/{created.year}-{created.mm}"

theme = "dark"

sidecar = [ "xmp",]
sidecar_drop_ext = true

person_keyword = true
favorite_rating = true

keyword_template = [
"Album: {folder_album}",
"{label}",
"{favorite?is_favorite}",
"is_{media_type}",
]

Removed the verbose and dry-run from the command line and instead put it in the TOML file
osxphotos export /Volumes/WDSSD2TB/photo_library_export/photo_library_backup/ --load-config /Volumes/WDSSD2TB/photo_library_export/osxphotos_backup.toml

.toml config contents for above command

[export]
verbose = true
dry_run = true

update = true
cleanup = true

report = "/Volumes/WDSSD2TB/photo_library_export/osxphotos_report_backup.sqlite"
append = true

db = "/Volumes/WDSSD2TB/Macbook_Pro_14_PhotosLibrary_Full_Originals_Backup/Photos Library.photoslibrary"

tmpdir = "/Volumes/WDSSD2TB/photo_library_export/osxphotos_tmp"

directory = "{created.year}/{created.year}-{created.mm}"

theme = "dark"

sidecar = [ "xmp",]
sidecar_drop_ext = true

person_keyword = true
favorite_rating = true

keyword_template = [
"Album: {folder_album}",
"{label}",
"{favorite?is_favorite}",
"is_{media_type}",
]

NOTE: If I leave verbose option ON and remove the dry run the commands work fine. If I remove the verbose option and leave only dry run (either way) it still gives same error.

[RhetTbull]

Thanks for the detailed report. I see the problem, will work on a fix but probably will be next weekend before I can get to it.

When used in --dry-run mode, osxphotos creates an in-memory copy of the backup database so as not to change the database (which is needed to calculate what to export). However, it currently does this by dumping the SQL for the database then loading that into a new database and it's exceeding the sqlite query length:

File "/Users/userC/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/osxphotos/export_db.py", line 939, in _open_export_db
conn.cursor().executescript(dbdump.read())
sqlite3.DataError: query string is too large

I'll open a new issue to fix this. The correct way is to use sqlite backup()

[RhetTbull]

sqlite3 backup() was introduced in Python 3.7, after I had developed the --dry-run code and in-memory database which is why I didn't originally use it. Using backup should fix this problem.

[RhetTbull]

Fixed in v0.60.1