Synology Moments can NOT show photos after completed rebuild Index #147
Comments
|
Same issue here on a Synology DS918+ and Synology Photos. I am also using my own UID/GUID to upload these images. |
|
I have been looking into this for the past 2 days now and I believe I have narrowed down the issue. It seems that Synology does not kickstart the indexing services (synofoto-bin-index-tool) and that specifically relates to docker volumes mapped to Synology volumes.. There are plenty of scripts/tools to actually tackle this problem, but I would ideally like to avoid running a separate indexing service on my NAS. I did manage to find a clue on how the Synology indexing service can be initiated and that is via "touch" on the uploaded file itself, which worked for me and the image immediately appeared on the Synology Photos app - see Reddit post here @tokyoqiang - can you give the following command a try? Do note that you need to run "touch" on the file itself and not on the folder:
@boredazfcuk - I took a quick look at the code and it does not seem that a touch of the file is taking place for successful uploads. There are a few references to "touch --reference="${heic_file}" "${heic_file%.HEIC}.JPG", but these are only taking place for conversions it seems. Is this something you could introduce? Thanks for your help! |
|
@tokyoqiang - I have setup a temporary solution via the task scheduler. I enforce a file "touch" for files not older than a day in my specified volume. This triggers the indexing for newly added files. Task scheduler -> Run command -> 'user defined scripts': Replace USERNAME with your own. Make sure you to schedule the script to run daily or change the mtime value to something different, ie. -2. which in theory would search for files that have been added within the past two days. @boredazfcuk - would still appreciate if you could consider my message above, because that would help our images appear instantly in our photos app. Cheers |
|
Hi, When the files are downloaded, their date/time stamp is set to be the same date/time as the photo was taken. By touching the files after download, the date/time stamp is set to the current date/time. This will then cause a mismatch between the local file and the file stored in iCloud. I've implemented a new variable I've not tested it, as I don't have a Synology device. Can you give it a try and let me know? |
|
Thanks, mate. I wasn't aware that touch would change the date. Looks like I ruined my entire library :) I am starting from scratch and will keep you up to date! Cheers |
|
Hi @boredazfcuk, quick update. I downloaded my entire library again (appx 200gb) with the
I am using the following compose: |
|
FYI- I did double check on the UID and GUID and these seem in order. The creator/owner of the file is reflected correctly. |
|
Hi, I've just tested it and confirmed it is does work as intended. It creates a reference file for each HEIC, changes the time on the HEIC file to current, changes the time back by using the reference file, and finally removes the reference file. How are you viewing the files? From within the container itself? I'm just wondering if there is a mismatch between what you are seeing and the date/time stamps of the actual files... Is this the photos app which has indexed the files on the first touch but not the second? |
|
I've set the synology_photos_app_fix=true, and in the log I can see it's playing with the newly downloaded files, like creating tmp files and such, but the Synology Photos app is not indexing anything still. |
|
Can you force a manual index? As described here: There's a request here, for a modification which would allow synchronisations to occur at specific times of day. If I implemented this, it would allow you to schedule one of the multimedia re-indexing runs, shortly afterwards. Keeping your photos app fairly up to date. |
|
yeah I can do manual re-index via synology photos app. I guess I'll just do that for now. |
Hi @boredazfcuk, sorry for my delayed response. The screenshot I have shared is from DSM (DiskStation manager). That's Synology's linux gui. I have tried running
What I am now a bit concerned about is the timeline of images that will be shown in the Photos App given the fact that the images are only reflecting the "upload" date/time. I am running a re-index as we speak and it may take a bit more time for me to be able to share the results. Will keep you up to date. Also, what also confuses me is the fact that when running the container, it does not attempt to download the images again although there is a mismatch in time/date and file name. The reason I am wondering is because from what I understood, the container would attempt to match existing (on disk) images by comparing date/time and name vs the same attributes on the cloud and attempt to download if no match is made. Finally, I also verified if the fix is running and that does not seem to be the issue: Cheers, |
Same on me. I see people start the nfs service and mount the directory into the docker container to solve this issue, if anyone wants to try. I have too many photos and have already download and manually re-indexed so just use it... :-) |
How do you mount it? also, do you have a script set to re-index? Or you just click yourself |
|
Is this fix actually needed then guys? It sounds like it does what was requested, but that doesn't actually fix the issue. If it doesn't actually fix the problem, then I'd like to remove it. I've now created a |
|
It does solve my problem and pix are being indexed with the touch, but I would appreciate it if you could come back to me on my previous comment? |
|
Your issue suggests that the "fix" is not working. The underlying application should set the file's date/time stamp to be the same as the EXIF data's DateTimeOriginal field (if it exists). The fix changes the date/time stamp to be the current date/time, then changes it back to to what it was. It seems it has not changed it back for your files. Can you check the EXIF data on the file and post the details? Connect to your container's command line like this: then run the identify on the HEIC file (you'll need to make sure the user/path is correct for your system: |
|
I think I'm onto something. I just SSH'ed into the Synology and ran this command so is there any way to incorporate this command into the docker? so that every time after it downloads some photo, it just run this command? |
|
HI @boredazfcuk, thanks for your response. I did not manage to find my images under path I have the following environment and volumes mapped: Given the above, I have my pictures located here: The exif data seems to be correctly reflecting the date/time the image was taken. I have tested a few more images and they all seem fine. Could it be that my folder structure within docker is the reason why the modification of the files is not taking place? I did validate my UID/GID as well and the files are correctly written with my username, so I dont think this is causing the issue here: Thanks mate, onestix |
|
@flyingwasabi - if you are looking at scheduling this, then you can do so via Syno's task scheduler, ie. run command every day at 2am. You can of course run this more often but it is an "intensive" task that may slow down your Syno while it is being executed. This would obviously leave a gap as to what you see in Syno Photos vs actually downloaded/available images on your Cloud. I am trying to avoid this option because I would need my images to show immediately in Syno photos. Also, you could try running the "basic" index, which is less intensive/quicker. It should do the job as well.
|
Right I’m currently running it in the synology task scheduler every 12 hours. However i was asking because I thought if we could build this script into the docker script, that way we wont be having any ‘empty’ scans trigger by scheduler, which would be more efficient. But it’s not a big deal anyway. |
|
@flyingwasabi - the issue with this command is that it reindexes the entire folder and not a single image- hence resource intensive. The "touch" proposal triggers the indexing of a single image- hence my proposal above. Can you double check if the "basic_reindex" (as opposed to "reindex") also solves your issue? This one is less resource intensive and should do the job as well. If we are not getting anywhere with the "touch" fix, then we might as well consider a "basic_reindex" once the job is completed. I agree that I would prefer having this executed by the container as opposed to a scheduled job in task scheduler which creates a gap between uploaded images vs running of scheduled job. Cheers, onestix |
Not easily. The Docker container is its own little Linux OS which has no access to the rest of your Synology device, only to the volumes you mount into it. You could mount the /var/packages/SynologyPhotos directory to the same location in your container and then kicking off the command, but at a guess the database lives somewhere else, so that would need mapping in too. There may also be permissions issues as you're using sudo to invoke it (so using the hosts root account), and your Docker containers may be running as a restricted user. |
It shouldn't be the folder structure, but I'm wondering if it could be a permissions issue creating the tmp file. It seems that the first touch command is working, as it's setting the creation time to be the time the file was downloaded. It just seems that it's not changing it back. That must be something to do with the time of the temp file being incorrect, or the tmp files not being created at all so the script has no reference time for the file. I've pushed a new version of the container which has a |
|
Thanks @boredazfcuk, appreciate your help. Will give it a try right away and keep you posted. |
@boredazfcuk It doesn't fix but I think it's really a useful function for synology users. Usage:
synoindex [OPTIONS]
Index Options:
-h, --help
this help text
-A dirpath
add a dir
-a filepath
add a file
-D dirpath
delete a dir
-d filepath
delete a file
-N new_dirpath old_dirpath
rename a dir
-n new_filepath old_filepath
rename a file
-R [all|media|photo|music|video|thumb|dirpath]
all: reindex all dirpath that registered in each package
media: reindex dirpath that registered in MediaIndex package
photo: reindex photo dirpath
music: reindex music dirpath
video: reindex video dirpath
thumb: check converted video of each video file
dirpath: reindex this specific dirpath
-R user:{user_name}
reindex personal photo dirpath
-R share:{share_name}
reindex share dirpath
-R [type_music|type_video|type_photo]
reindex dirpath that registered with specific type in MediaIndex
Package Index Options:
-P [MediaIndex|{package_name}] {index_option}
index operation only apply on this package
-p [MediaIndex|{package_name}] {index_option}
index operation apply all packages except for this package
File Index Options:
-f {index_option}
index operation apply on file index
-U photo
update photo imageswe can mount the |
|
Hi @boredazfcuk , quick update here. The TMP files are being created:
Interesting observation here is that they are being created with owner "root", whereas my actual image files are created under my own username that I have defined as part of the compose:
Given your hint that this may relate to permissions, I thought I'd change both "user" and "user_id" to that of my "docker" user- and voila, the modify date now seems to be reflecting the correct date the image was taken:
Can I just confirm with you that this is now as expected? So the 'modify' date reflects the date that the image was taken and 'create' will always reflect the date that the image was uploaded? Also, Synology Photos is picking up these new images with the Modify date (actual date image taken), which in theory is exactly what we wanted to achieve:
So, this is looking really good and I'll attempt to download my entire library with this fix (again)! :) Thanks mate! |
You mention that it is not "fixing" your problem. Do you mind providing some more details here? You can try running below command and check if the image shows up in the Synology Photos app. Do note that you need to run "touch" on the file itself and not on the folder: Also, regarding your proposal to run synoindex - please see comments above. Synoindex forces an index of the full folder and not a single file. It is resource intensive and not in the interest of everyone to be performing a full folder index after a single image upload. Take my example, I am currently performing my uploads on a very regular basis and have approximately 200,000 pictures in the folder. A reindex here would take hours. If you do wish to do so, then run a scheduled task? |
|
well if synoindex force the full index, it's not a good idea for me too, I didn't realize that. |
Interesting. Option two is irrelevant and not expected to work, because you are running the command from within the container and this is the exact issue we trying to resolve in first place (file changes on mounted docker drives are not indexed by Syno). Can you try running the container with your docker user? To find out the UID and GUID of your docker user, ssh into your Syno and type in |
|
I don't have docker user... I tried running as root, it always stuck in 'Generating list of files in iCloud...' |
|
@0x5e, that's really strange. Do you mind looking up your users? There must be one dedicated to docker:
|
|
There's only 4 user: admin, 2 users greate by me, and guest. admin and guest are disabled. I'm DSM 7.1 and docker installed from package center. really strange... |
|
Hm, you mentioned that root doesn't work for you because it gets stuck on the file check. Can you try running as root while skipping the file check? Env |
|
Currently using icloudpd installed by
|
|
Hi @Alano-i, I am not familiar with that setup, but I believe it should work. Give it a try and let us know. The challenge that we have is that Synology does not index media files on mounted/external folders, hence the touch, to trigger the indexing service. |
|
I've pushed a new version to Docker hub which now performs the touch commands as the ${user} (instead of root). Can someone see if this triggers the re-index? I'm wondering if the Synology doesn't pick up on it because it's the root account which is manipulating the files. |
|
@boredazfcuk - works for me! Running the container as my "docker" user. Thanks! |
|
Does it work with the container being run as root though? There may be a bunch of other problems if the main script is not launched with user id 0 as it handles user switching within the script via |
|
Hi, I've read the whole thread as I have the same issue. But can someone explain why the touch inside the container would work and makes the file to be indexed, while just placing the file doesn't? Above with the 'option 1 / 2' scenario, manually touching did not kick off the indexing. How does the script itself would manage this? For me it is still not working, running docker container as root. |
|
I would have expected the download to trigger a rebuild too, but I don't have one of these devices to actually test it though, so can only go by the feedback I receive:
I took it to mean that the I was thinking that it may just be ignoring operations performed on the filesystem performed by root, so that system/background operations don't trigger the re-index all the time. |
Synology does not index media files added by certain protocols. See here
The same also applies for Docker mapped folders. Have you tried sshing into your NAS and running touch on a single image? Also, I assume you that have added the folder in question to your indexed folders (Control Panel > Indexing Service > Media Indexing > Indexed Folder)? |
|
Yes I get that media is not scanned. When I touch it or copy a file into the folder it gets indexed and shows up. So back to my question, how does a touch from within the container should fix this issues? Of docker volumes are not indexed anyway.
… On 12 May 2022, at 22:45, boredazfcuk ***@***.***> wrote:
I would have expected the download to trigger a rebuild too, but I don't have one of these devices to actually test it though, so can only go by the feedback I receive:
It does solve my problem and pix are being indexed with the touch
I took it to mean that the touch solution in the container was giving the wanted behaviour... Triggering an index for each file that is touched, and not having to perform a full/partial re-index of all files, as that would be processor intensive, especially on large libraries.
I was thinking that it may just be ignoring operations performed on the filesystem performed by root, so that system/background operations don't trigger the re-index all the time.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.
|
|
The index is triggered by ‘Inotify’ and is ignored for docker containers (or doesn’t work for whatever reason for docker) this is also the case for any docker solution for file syncing like nextcloud and similar solutions. So you can’t use tools like nextcloud to sync Synology folders.
… On 12 May 2022, at 22:52, Ron Bruins ***@***.***> wrote:
Yes I get that media is not scanned. When I touch it or copy a file into the folder it gets indexed and shows up. So back to my question, how does a touch from within the container should fix this issues? Of docker volumes are not indexed anyway.
>> On 12 May 2022, at 22:45, boredazfcuk ***@***.***> wrote:
>>
>
> I would have expected the download to trigger a rebuild too, but I don't have one of these devices to actually test it though, so can only go by the feedback I receive:
>
> It does solve my problem and pix are being indexed with the touch
>
> I took it to mean that the touch solution in the container was giving the wanted behaviour... Triggering an index for each file that is touched, and not having to perform a full/partial re-index of all files, as that would be processor intensive, especially on large libraries.
>
> I was thinking that it may just be ignoring operations performed on the filesystem performed by root, so that system/background operations don't trigger the re-index all the time.
>
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you commented.
|
|
I am using my 'docker' user to run the container and the touch does in fact trigger the indexing. Have you tried a different UID/GUID besides root? |
|
That kind of surprised me too. I installed the docker package and I have no docker user… just runs as root (by default?)
… On 12 May 2022, at 23:11, onestix ***@***.***> wrote:
I am using my 'docker' user to run the container and the touch does in fact trigger the indexing. Have you tried a different UID/GUID besides root?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.
|
|
Hm strange, try your regular user that you use on your Syno? |
|
I mapped the user for icloudpd to the user for photos. However I’m on dsm 7, how about you?
When I touch the file from ssh it gets indexed immediately… not from within docker.
Maybe I should create a script that pulls exif and does the same at syno level… run it in task scheduler
… On 12 May 2022, at 23:24, onestix ***@***.***> wrote:
Hm strange, try your regular user that you use on your Syno?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.
|
|
Running on latest DSM as well. You could alternatively run a basic reindex on your folder. You can schedule it via task scheduler:
That would save you from tampering around with the touch and changing exif data back. There is one final solution I can think of and that is if you want to fix this once and for all, including other containers/services you may be running. For that you'd need to install the inotify-tools package from the SynoCommunity and create a little script. You'll find more details here: |
|
Yeah but the index runs through the whole folder and it’s massive. The touch can be executed in the same timeframe as the download schedule and is easier on the resources then…
I’m still wondering how it works for you and doesn’t in my case.
… On 12 May 2022, at 23:45, onestix ***@***.***> wrote:
Running on latest DSM as well.
You could alternatively run a basic reindex on your folder. You can schedule it via task scheduler:
/var/packages/SynologyPhotos/target/usr/bin/synofoto-bin-index-tool -t basic_reindex -i /var/services/homes/USERNAME/Photos
That would save you from tampering around with the touch and changing exif data back.
There is one final solution I can think of and that is if you want to fix this once and for all, including other containers/services you may be running. For that you'd need to install the inotify-tools package from the SynoCommunity and create a little script. You'll find more details here:
letorbi/synoindexwatcher#51 (comment)
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.
|
|
I found this the other day: https://github.com/zeichensatz/SynologyPhotosAPI It details an API available on the Synology which has a couple of Indexing methods (SYNO.FotoTeam.Index and SYNO.Foto.Index). It may be possible to pass the name of the file that needs indexing in to the API to trigger the index of the individual file. |
I believe this would only let you index the full folder again.. |
I am now using this on a schedule: This prevents previous folders to be indexed. |
This way can solve the problem
sh /volume1/web/icloudpd.sh
#!/bin/bash
# Author: Howardnm
# #####-----------#####-----------#####-----------#####-----------#####-----------#####
# config
down_directory="/volume1/homes/howard/Drive/Moments/icloud" # Photo download directory (照片下载目录)
Script_cache_file="/volume1/web" # Script cache file storage (脚本缓存文件存放区)
days=2 # check files from the last 2 days (扫描前2天的照片)
synophoto_index_tool="/var/packages/SynologyMoments/target/usr/bin/synophoto-bin-index-tool" # DSM 6.2 moments tool
#synophoto_index_tool="/var/packages/SynologyPhotos/target/usr/bin/synofoto-bin-index-tool" # mybe DSM 7.0 ? (i don't know)
# #####-----------#####-----------#####-----------#####-----------#####-----------#####
# execution area
search_file(){
find ${down_directory} -mtime -${days} -name '*.*' > ${Script_cache_file}/log.txt
sleep 2s
sed '/@eaDir/d' ${Script_cache_file}/log.txt > ${Script_cache_file}/log_update.txt
sleep 2s
num_files=$(cat ${Script_cache_file}/log_update.txt | wc -l)
echo "[Find ${num_files} files from the last ${days} days]"
}
search_file
file=1
while(( $file<=$num_files ))
do
file_path=$(sed -n "${file}p" ${Script_cache_file}/log_update.txt)
echo "[$file]$file_path"
${synophoto_index_tool} -t basic_reindex -i ${file_path}
sleep 1s
let "file++"
done
echo "[------Finish------]"
|
|
Hi, I'm just going to close this off a it seems that this issue needs to be handled outside of the container. Thanks. |
|
Thanks @boredazfcuk. If you could please however leave the synology_photos_app_fix in place? It is working for me with this little development that does the touch. Appreciate it and thanks again! |
|
No worries. I've made a note in the code not to obsolete it. Just to remind me further down the line... |
|
I think the KEY POINT is the permission. After the icloudpd docker completed the task, i found the permission of folder for the photos is drwxr-x--- i use if you restart the docker of icloudpd, every photo changed to question mark. |
This is really a great code that helps me easily download photos from Icloud.
My issue is synology moments can't show photos after it completed 'rebuild index'. If I move the photos to another location then move back, it can show.
I tried add User/UID/Group/Group ID predicate, it changed user from 1000 to my name, but still fail to show phto.
Can you help me to check the issue? Thanks!
The text was updated successfully, but these errors were encountered: