Import: Disable auto import by default, as large or incomplete uploads can lead to problems

[phemmer]

1. What is not working as documented?

While using PhotoSync on android, 2 large video files were uploaded (via WebDav). However the automatic import kicked in while the second video was still being transferred, causing it to be corrupted.

2. How can we reproduce it?

Steps to reproduce the behavior:

1. Use photosync to import 2 large files.
2. Have the automatic import kick in while the second file is still transferring (manual import trigger would probably do it too).

3. What behavior do you expect?

There are several behaviors I would expect that would have mitigated this issue:

1. The automatic import should ignore files with a recent modification timestamp. "recent" may be hard to define, but I would think at least the value of PHOTOPRISM_AUTO_IMPORT. Though for people with the value of -1, maybe at least a few seconds?
2. Since the video was being uploaded via WebDav, I would expect PhotoPrism to know that the transfer is in progress, and to skip the file.
3. In-progress transfers should have a "hidden" filename (e.g. having a . prefix, and a .tmp suffix), and the import should skip hidden files. This could be an implementation to address point 2 above, but if point 2 is addressed through some other means, I think this behavior should exist anyway, as it would support transfers through means other than WebDav.
4. This is an assumption, but it would appear that the file is copy/deleted into "originals" rather than a filesystem move (rename) operation. If it were done via filesystem rename, the file handle that the WebDav process had open would have still pointed to a valid file, and the upload would have been able to complete.
5. The video was unplayable due to corruption (missing trailer), and I don't think corrupt files should be imported (not automatically anyway).
6. The video was over the default size limit (1g), and thus wasn't visible in the UI. I would not expect a file to be put in "originals" that is not present in the UI. Otherwise you end up with invisible files.

4. What could be the cause of your problem?

None of the expected behaviors were performed.

5. Can you provide us with example files for testing, error logs, or screenshots?

2024-05-30T18:33:09.691317462-04:00 time="2024-05-30T22:33:09Z" level=info msg="webdav: PUT /import/Samsung%20SM-G991U1/Camera/20240530_164057.mp4"
2024-05-30T18:34:03.602395452-04:00 time="2024-05-30T22:34:03Z" level=info msg="copying files from import"
2024-05-30T18:34:03.602543301-04:00 time="2024-05-30T22:34:03Z" level=info msg="classify: loading nasnet"
2024-05-30T18:34:07.049230394-04:00 time="2024-05-30T22:34:07Z" level=info msg="classify: loading labels from labels.txt"
2024-05-30T18:34:15.013186953-04:00 time="2024-05-30T22:34:15Z" level=info msg="media: 20240530_164057.mp4 was taken at 2024-05-30 20:53:25 +0000 UTC (meta)"
2024-05-30T18:34:15.847294796-04:00 time="2024-05-30T22:34:15Z" level=info msg="media: 20240530_165347.mp4 was taken at 2024-05-30 22:34:15.804374245 +0000 UTC (file mod time)"
2024-05-30T18:34:16.043293935-04:00 time="2024-05-30T22:34:16Z" level=info msg="import: moving main mp4 file 'Samsung SM-G991U1/Camera/20240530_164057.mp4' to 2024/05/20240530_205325_1774BF42.mp4"
2024-05-30T18:34:16.641238178-04:00 time="2024-05-30T22:34:16Z" level=info msg="import: moving main mp4 file 'Samsung SM-G991U1/Camera/20240530_165347.mp4' to 2024/05/20240530_223415_C819ED5B.mp4"
2024-05-30T18:35:36.397614343-04:00 time="2024-05-30T22:35:36Z" level=info msg="convert: converting 20240530_205325_1774BF42.mp4 to 20240530_205325_1774BF42.mp4.jpg (ffmpeg)"
2024-05-30T18:35:36.804967163-04:00 time="2024-05-30T22:35:36Z" level=info msg="convert: 20240530_205325_1774BF42.mp4.jpg created in 452.695457ms (ffmpeg)"
2024-05-30T18:35:37.222856490-04:00 time="2024-05-30T22:35:37Z" level=info msg="media: generated 10 thumbnails for 2024/05/20240530_205325_1774BF42.mp4.jpg [416.008858ms]"
2024-05-30T18:35:37.225538874-04:00 time="2024-05-30T22:35:37Z" level=warning msg="import: 2024/05/20240530_205325_1774BF42.mp4 exceeds file size limit (2.6 GB / 1.0 GB)"
2024-05-30T18:36:10.247433883-04:00 time="2024-05-30T22:36:10Z" level=info msg="convert: converting 20240530_223415_C819ED5B.mp4 to 20240530_223415_C819ED5B.mp4.jpg (ffmpeg)"
2024-05-30T18:36:10.351759556-04:00 time="2024-05-30T22:36:10Z" level=error msg="import: failed to create preview image for 2024/05/20240530_223415_C819ED5B.mp4 (ffmpeg version 6.1.1-3ubuntu5 Copyright (c) 2000-2023 the FFmpeg developers  built with gcc 13 (Ubuntu 13.2.0-23ubuntu3)  configuration: --prefix=/usr --extra-version=3ubuntu5 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --disable-omx --enable-gnutls --enable-libaom --enable-libass --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libharfbuzz --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-openal --enable-opencl --enable-opengl --disable-sndio --enable-libvpl --disable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-ladspa --enable-libbluray --enable-libjack --enable-libpulse --enable-librabbitmq --enable-librist --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libx264 --enable-libzmq --enable-libzvbi --enable-lv2 --enable-sdl2 --enable-libplacebo --enable-librav1e --enable-pocketsphinx --enable-librsvg --enable-libjxl --enable-shared  WARNING: library configuration mismatch  avcodec     configuration: --prefix=/usr --extra-version=3ubuntu5 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --disable-omx --enable-gnutls --enable-libaom --enable-libass --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libharfbuzz --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-openal --enable-opencl --enable-opengl --disable-sndio --enable-libvpl --disable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-ladspa --enable-libbluray --enable-libjack --enable-libpulse --enable-librabbitmq --enable-librist --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libx264 --enable-libzmq --enable-libzvbi --enable-lv2 --enable-sdl2 --enable-libplacebo --enable-librav1e --enable-pocketsphinx --enable-librsvg --enable-libjxl --enable-shared --enable-version3 --disable-doc --disable-programs --disable-static --enable-libaribb24 --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libtesseract --enable-libvo_amrwbenc --enable-libsmbclient  avformat    configuration: --prefix=/usr --extra-version=3ubuntu5 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --disable-omx --enable-gnutls --enable-libaom --enable-libass --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libharfbuzz --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-)"
2024-05-30T18:36:10.352311183-04:00 time="2024-05-30T22:36:10Z" level=info msg="import: deleted empty folder 'Samsung SM-G991U1/Camera'"
2024-05-30T18:36:10.352647628-04:00 time="2024-05-30T22:36:10Z" level=info msg="import: deleted empty folder 'Samsung SM-G991U1'"
2024-05-30T18:36:12.832164746-04:00 time="2024-05-30T22:36:12Z" level=info msg="import completed in 129 s"
2024-05-30T18:37:04.619055799-04:00 time="2024-05-30T22:37:04Z" level=info msg="webdav: PUT /import/Samsung%20SM-G991U1/Camera/20240530_165347.mp4"
2024-05-30T18:38:03.550108631-04:00 time="2024-05-30T22:38:03Z" level=info msg="copying files from import"
2024-05-30T18:38:03.626502841-04:00 time="2024-05-30T22:38:03Z" level=info msg="import completed in 0 s"

6. Which software versions do you use?

(a) PhotoPrism Architecture & Build Number: AMD64, ARM64, ARMv7,...
x86_64
Build 240523-923ee0cf7

(b) Database Type & Version: MariaDB, MySQL, SQLite,...
SQLite

(c) Operating System Types & Versions: Linux, Windows, Android,...
TrueNAS Scale 23.10.2

(d) Browser Types & Versions: Firefox, Chrome, Safari on iPhone,...
Upload was by PhotoSync on android. Version 4.0

(e) Ad Blockers, Browser Plugins, and/or Firewall Software?
N/A

7. On what kind of device is PhotoPrism installed?

Intel N100 mini pc

(a) Device / Processor Type: Raspberry Pi 4, Intel Core i7-3770, AMD Ryzen 7 3800X,...
Intel N100

(b) Physical Memory & Swap Space in GB
16gb/2gb

(c) Storage Type: HDD, SSD, RAID, USB, Network Storage,...
HDD

(d) Anything else that might be helpful to know?

8. Do you use a Reverse Proxy, Firewall, VPN, or CDN?

N/A

[lastzero]

Since we never received any such report with the old default delay for the auto import, we'll switch back to that. The reason I've reduced the wait time in the last release was that users in the past asked us "why does it wait so long? How can it be more aggressive?". So now we know why that is... I would also advise against auto importing very large files in general, completely independent of what security measures are in place, because it can always be that the mobile or wifi connection goes away and then you have half uploaded files. We don't have a team here working full time on heuristics to figure out what was intended and what not, and then to provide detailed info, warning and error messages with advice for each case... In fact, it might be best not to offer an auto import at all if there's a danger of corruption.

[lastzero]

@phemmer Based on your report and because the auto import feature is inherently unsafe¹, these changes disable it by setting the default value to -1.

An updated development preview that includes the changed defaults is available on Docker Hub, so you can test it:

• https://docs.photoprism.app/getting-started/updates/#development-preview
• https://hub.docker.com/r/photoprism/photoprism/tags?page=1&name=preview

Footnotes

1. In addition to the potential (and as seen here, practical) issues with safely handling large and incomplete uploads (without risking locking the file system forever and allowing denial of service attacks), users might also perform manual changes to the files and folders in the import directory, including cases where multiple users are simultaneously accessing the server's file system, performing web uploads, and using WebDAV in unpredictable ways. ↩

[lastzero]

We've also updated the Release Notes to mention these changes:

• https://docs.photoprism.app/release-notes/#development-preview

[phemmer]

I see the title was changed to "Disable auto import by default", which is the resolution of the issue, and not the issue itself. Does that mean that this issue will be closed once "Disable auto import by default" is done? If so, I still think the 6 items mentioned would be good functionality to implement. Disabling it doesn't even completely remediate the issue, as it's still possible a manual import could be triggered, and a file to be added while that's happening.

Any of the 6 items (5 really, since the 6th would have mitigated this exact scenario, and can prevent other problems, but doesn't address the general incomplete-upload problem) would help make it safe.

You mentioned file locking, and that might be a solution for point 2, but I would probably go with a more general solution at first. Though file locking shouldn't be a DoS issue if you're just locking an individual file. I don't see why the whole directory would need to be locked.

In any case, items 1, 2, 3, and (possibly) 4 seem like they should be fairly easy to implement. While disabling auto import does reduce the risk of corruption, the risk is still there, and I would like to eliminate that risk, and would like more to have auto-import. If any of those is acceptable, given that they should be easy, I might be willing to work on them. However I have been pretty busy lately, so I can't say for sure that I would.

[lastzero]

Since this was a bug report related to file corruption, we have just released a new stable version that disables this feature by default (so we can safely go on vacation without any other accidents happening during this time):

• https://docs.photoprism.app/release-notes/#may-31-2024

Details about the implementation of this feature and its history can be found in the original issue:

• UX: Auto Import / Index #281

Bottom line is that we currently don't have the resources to make this more sophisticated and the risks were clearly communicated, at least at the time of implementation / release. Please let us know if we should include additional warnings in our documentation until we might eventually have a version that can work around all the potential issues I mentioned in the footnote above (any contributions towards this goal are of course most welcome):

• Import: Disable auto import by default, as large or incomplete uploads can lead to problems #4310 (comment)