DSM7 (Synology Photo) compatibility

[dtypo]

Hello letorbi :)
I've used your great python module with profit since 1 year now and no problem at all until yesterday, when I decided to update to DSM7.
Considering the integration of Python3 in DSM, I've reinstalled all my python modules from the scratch, including SynoIndexWatcher.
Unfortunately, giving a simple python -m synoindexwatcher produces the usual output:

2021-09-07 23:04:36,266 INFO Adding watch for path: /volume1/music
2021-09-07 23:04:42,417 INFO Adding watch for path: /volume1/photo
2021-09-07 23:04:45,144 INFO Adding watch for path: /volume1/video
2021-09-07 23:04:45,145 INFO Waiting for media file changes...

but the images automatically uploaded in /volume1/photo (via syncthing) don't appear in my Synology Photo app.
The only way I can see the images is by manual reindexing with the button showed in the screenshot.

The exact same mechanism (syncthing+synoindexwatcher) worked like a charm with Photo Station and Moments.
So I suppose this is strictly related to the new indexing mechanism used by Synology Photo.

More info HERE and HERE.

I hope you can give an eye and solve this problem.
Thanks in advance :)

[letorbi]

Hej dtypo,

thanks for the report. I didn't have the time to upgrade my DiskStation, so I didn't ran into this problem, yet.

I don't know when I will have time to upgrade to DSM7, but I hope it will happen within the next three weeks. As soon as I know more about the issue I will write here.

Thanks again,
Torben

[danyfire]

I Confirm I'm facing with the same issue as @dtypo after DSM7 update. The indexing stopped working, but I didn't go as far as reinstalling every modules yet.
Good luck, thank you again for your great tool.

[danyfire]

Hello any news on that issue ? Synchronisation still not working since DSMUpdate

[letorbi]

I plan to work in the issue from 18th to 20th of October, but please don't take this timespan for granted as work and family might have other plans for me ;)

[letorbi]

Hello,

I've finally started to work on the issue 🥳 The upgrade to DSM 7 went through and I did some first test with some surprising results:

• First of all I've found put, that the synoindexwatcher module has to be reinstalled after the update, because the whole Python environment seems be replaced during the upgrade process.
• However, Synoindex Watcher does not seem be be required anymore! I've added files via sftp, Syncthing and the Terminal and every change was reflected in the media index immediately - all without Synoindex Watcher running. It seems that the Synology guys implemented some similar functionality themself...

I haven't tested Synology Photo so far, but if my findings are right, it would IMHO mean that the best solution would be to leave Synoindex Watcher as it is and create a new auto-update tool for Synology Photo. This could be based on Synoindex Watcher, but I'm not sure, if this is best way, since the whole indexing seems to work different there.

👉 Can someone please confirm that Synoindex Watcher is not required to auto-update the media index on DSM 7?

[danyfire]

Hello,

for me I haven't done that part "the synoindexwatcher module has to be reinstalled after the update," .
I'm a average user not too advanced.
Every time I mess around with synoindexwatcher, I end up restarting everything from scratch.
For me synoindexwatcher used to be able to deal with the indexing related to the automatic process that I use to "download tvshows"
In my case it's the app Sonarr. Once a episode is downloaded using the app "Transmission", sonarr moves the episode from the download folder to another episode folder, and that's when synoindexwatcher. kicks in to index media index.
Unfortunately since the upgrade that process isn't working anymore.

[dtypo]

Hello,

I've finally started to work on the issue 🥳 The upgrade to DSM 7 went through and I did some first test with some surprising results:

• First of all I've found put, that the synoindexwatcher module has to be reinstalled after the update, because the whole Python environment seems be replaced during the upgrade process.
• However, Synoindex Watcher does not seem be be required anymore! I've added files via sftp, Syncthing and the Terminal and every change was reflected in the media index immediately - all without Synoindex Watcher running. It seems that the Synology guys implemented some similar functionality themself...

I haven't tested Synology Photo so far, but if my findings are right, it would IMHO mean that the best solution would be to leave Synoindex Watcher as it is and create a new auto-update tool for Synology Photo. This could be based on Synoindex Watcher, but I'm not sure, if this is best way, since the whole indexing seems to work different there.

👉 Can someone please confirm that Synoindex Watcher is not required to auto-update the media index on DSM 7?

Hello letorbi!
As I said previously (and just like you), I had to reinstall synoindexwatcher after the update to DSM7 because the python environment has changed a lot.
Speaking about the feedback you need...
Giving the fact that I CANNOT see any picture uploadad in /photo (via Syncthing) in the app Synology Photo, could you please remind me how to verify if a file has been correctlty inserted in the media index?
I don't know if this can be of any help (I don't remember how this worked on DSM6 before I installed synoindexwatcher), but I definetely CAN see the files uploaded via Syncthing in File Station.

[letorbi]

Thanks for the feedback :)

@danyfire Can you see the new TV-Show files in your video-folder? If you see new files, but they aren't part of the media-index, Sonarr seems to circumvent the new auto-updating mechanism somehow. In that case, it should be enough to re-install Synoindex Watcher simply by following the install-instructions in the readme. If the files are not present in the folder, the problem must be with Transmission or Sonarr.

@dtypo It seems that Synology Photo does not use the media-index to find photos anymore. Because of this Synology Photo can not be used to check, whether a file has been added to the media-index. I personally check the media-index via the Media Server package, which presents all files in the media-index to UPnP/DLNA-clients (I'm using DMS Explorer on my Android, but VLC should also work). Basically I just open the App, connect to my DiskStation and browse the files. If you prefer a more low level approach, you could also use the psql command on you DiskStation. For example, the command psql mediaserver -c "SELECT path FROM music" lists all files, which where added as music to the media-index. Similar commands work for photo and video.

[danyfire]

oki fixed for me thx

[letorbi]

oki fixed for me thx

Great to hear. Do you still have to use Synoindex Watcher to have files added to the media-index or was it a problem with Transmission or Sonarr?

[danyfire]

I still need Synoindex Watcher I think. I didn't test to uninstall it. All I can say indexing wasn't working after updating dsm. And after I reinstalled Synoindex it was working fine.

…

On Thu, Oct 21, 2021, 12:15 Torben ***@***.***> wrote: oki fixed for me thx Great to hear. Do you still have to use Synoindex Watcher to have files added to the media-index or was it a problem with Transmission or Sonarr? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#51 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYRCBF7FKP5U7AV6CZRJWLUH7R3HANCNFSM5DTHUJCA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[letorbi]

Just to keep you updated: I'm still on this issue.

Currently I'm trying to move my files from my DiskStation to another Location, in order to perform a factory reset. Once that is done, I will analyze the media indexing behavior on a vanilla DSM7 system. My goal is to confirm, that any files added to a media folder are automatically indexed - even without Synoindex Watcher installed.

If that's the case, I will leave Synoindex Watcher as it is and create a new tool for adding photo-files to the Synology Photo index instead. If Synoindex Watcher is still required on DSM7 systems, I will integrate the Synology Photo auto-indexing functionality into it.

[systematicguy]

I want to chime in about the fact, that some means of manual nudge is needed, if files are landing via rsync.
My setup in a nutshell: I have files landing in /homes/webdav/incoming/folder[1-4]
3 of those folders are being continuously rsynced (combined with inotifywait) to /homes/myuser/Photos/personal/folder[1-3]
1 of those folders is being continuously rsynced to /photo/shared/folder4
Just like @dtypo I need to do a reindex from the gui or some automated way, to have those pics show up.

My temporary solution is Task Scheduler: two scheduled tasks, performed every 8 hours, both with root:
for the personal stuff:
/var/packages/SynologyPhotos/target/usr/bin/synofoto-bin-index-tool -t basic_reindex -i /volume1/homes/myuser/Photos
and the other one:
/var/packages/SynologyPhotos/target/usr/bin/synofoto-bin-index-tool -t basic_reindex -i /volume1/photo

Not ideal as it is basically a reindex. (You cannot go deeper and say /volume1/photo/shared, because it somehow messes up the index structure, causing some nested folder /volume1/photo/shared/shared on the gui which appears to be empty on the gui, but once you delete it, you actually delete /volume1/photo/shared).

[misery]

I'm using inotify-tools. It works good.

https://synocommunity.com/package/inotify-tools

Login with SSH.

notify.sh

#!/bin/bash

inotifywait \
  /var/services/homes/USERNAME/Photos/DCIM \
  -r \
  --monitor \
  -e moved_to \
  --timefmt '%Y-%m-%dT%H:%M:%S' \
  --format '%T %w %f %e' \
  --exclude "@eaDir" \
| while read datetime dir filename event; do
  echo "Event: $datetime $dir$file $event"
  touch "${dir}${filename}"
done

notify.service

[Unit]
Description=inotify
After=network.target

[Service]
Type=simple
User=USERNAME
ExecStart=/var/services/homes/USERNAME/systemd/notify.sh
Restart=on-failure

[Install]
WantedBy=multi-user.target

Start and enable

sudo systemctl enable /var/services/homes/USERNAME/systemd/notify.service
sudo systemctl start /var/services/homes/USERNAME/systemd/notify.service

Known issue: You need to delete/move files with Synology Photos/File Manager. If you delete/move files from "Container" it won't update the index.

[onestix]

I'm using inotify-tools. It works good.

https://synocommunity.com/package/inotify-tools

Login with SSH.

notify.sh

#!/bin/bash

inotifywait \
  /var/services/homes/USERNAME/Photos/DCIM \
  -r \
  --monitor \
  -e moved_to \
  --timefmt '%Y-%m-%dT%H:%M:%S' \
  --format '%T %w %f %e' \
  --exclude "@eaDir" \
| while read datetime dir filename event; do
  echo "Event: $datetime $dir$file $event"
  touch "${dir}${filename}"
done

notify.service

[Unit]
Description=inotify
After=network.target

[Service]
Type=simple
User=USERNAME
ExecStart=/var/services/homes/USERNAME/systemd/notify.sh
Restart=on-failure

[Install]
WantedBy=multi-user.target

Start and enable

sudo systemctl enable /var/services/homes/USERNAME/systemd/notify.service
sudo systemctl start /var/services/homes/USERNAME/systemd/notify.service

Known issue: You need to delete/move files with Synology Photos/File Manager. If you delete/move files from "Container" it won't update the index.

@misery - this worked for me, thanks! My next challenge were the images that are being deleted by my docker container. I tried the following "synofoto-bin-index-tool" Syno service (basic_reindex) which I am running as a scheduled task. This seems to be removing all non-existent files/pictures from the Photos app:

/var/packages/SynologyPhotos/target/usr/bin/synofoto-bin-index-tool -t basic_reindex -i /var/services/homes/USERNAME/Photos

Cheers

[letorbi]

According to the last comments in #57, Docker might be the reason why files are not added to the media index by Synology's indexing tool.

I won't be able to investigate this before September, so it would be helpful, if anyone can test whether "normal" file-operations (like cp or mv) inside a Docker container also don't trigger Synology's indexing tool.

PS: Sorry for the long delays. I have quite a busy life right now and not much time for side-projects.

[coolhand72]

Hi @letorbi, thank you so much for all your work on this. I'm curious if you have ever forwarded this information to synology? I am currently running into the same issues and it does appear to me the issue revolves around media indexing of folders when those changes are made from inside a docker container.

[coolhand72]

I received this explanation from Synology Support today:

The process which listens for changes, and activates the auto indexing service does not directly monitor the disc for file changes. It instead communicates with other services to determine when a change is made, and then activates the indexing process. Unfortunately, changes to the disk via docker is not one of the things this process currently monitors.

I did submit is as a feature request.

[KingOfBaboon]

confirmed this script won't help to get Synology photo auto index synced file from docker syncthing.

But thanks to all the info here, I have a solution now to get the auto index working:
https://forum.syncthing.net/t/share-my-solution-to-get-synology-photos-auto-index-working-with-syncthing/21410

use the smb instead of direct mount within docker.
This can be a solution for all issues caused by Synology won't auto index my file created via docker.