Images and tracks in group folders are not scanned

[jurgenhaas]

As my images still didn't show up after a couple of days I've tried the OCC command to scan images and tracks and they find some images for the users. But most images on our family NC installation are stored in group folders and none of them gets scanned.

Are you planning to offer that too?

[jancborchardt]

With group folders, do you mean folders other people shared to you? Or created with the https://github.com/nextcloud/groupfolders app?

[jurgenhaas]

I mean group folders.

[Klakla2]

But of course the same problem occurs with shared folders too.
I think an additional setting whether to scan shared folders or not is needed - exactly like it is done in the settings of the gpxpod app.

[Klakla2]

I propose to change the tiltle of the issue to

Images and tracks in group folders and shared folders are not scanned.

[julien-nc]

Photos in folders "explicitly" shared with you should be scanned and file hooks should also handle them (when a photo file is created/edited/deleted in such folders). If that does not work, there is a bug.

If a folder is shared with a user group (implicitly with you), file hooks won't work but manual scan should work as they appear when exploring your storage.

I don't know about adding an option. As we're supposed to avoid scan performance issues with automatic background scanning + file hooks in Maps, I think seeing photo files shared with you should always happen. There is no point in disabling it here, or is there?

[Klakla2]

Explicitly shared tracks appear, but that's not our use case.

I do not understand your statement on 'manual scan' - can this be triggered in GUI or only via occ command? If the latter is true this is not posible for 99% of the users.

I see no point in disabling the scan of files explicitly shared, but - for our use case - we would require an option to also scan implicitly shared folders.

Thank you for this beautiful app and your patience.

[julien-nc]

Explicitly shared tracks appear, but that's not our use case.

Let me rephrase:
I'm not saying things are fine, just explaining what works and what does not. For the moment, file hooks don't work for folders/files shared with a group. Everything should work fine if files/folders are shared with users.

It's not a choice, just something that is a bit challenging and not implemented. Adding an option to enable it is not possible for the moment because it's not implemented 😉.

[Klakla2]

Ok for me. I do not know, whether it is related, but I saw following error in Log repeatedly: [PHP] Error: Division by zero at /html/owncloud/apps/maps/lib/Service/PhotofilesService.php#420 GET /cron.php from 89.12.216.19 at 2019-09-15T19:06:26+00:00 Julien Veyssier <notifications@github.com> schrieb am So., 15. Sep. 2019 21:21:

…

Explicitly shared tracks appear, but that's not our use case. Let me rephrase: I'm not saying things are fine, just explaining what works and what does not. For the moment, file hooks don't work for folders/files shared with a group. Everything should work fine if files/folders are shared with users. It's not a choice, just something that is a bit challenging and not implemented. Adding an option to enable it is not possible for the moment because it's not implemented 😉. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#123?email_source=notifications&email_token=AEMERLRXVIY4SHDTSFJ7GLDQJ2DKJA5CNFSM4ITOSNA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6XXGVA#issuecomment-531592020>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEMERLTOLFXHBUDJMVQIHGLQJ2DKJANCNFSM4ITOSNAQ> .

[jancborchardt]

And just to clarify from the design side: If you can see the stuff in Files, then you should also see it in Maps. No additional setting should be needed. :) (But as @eneiluj said of course stuff has to be implemented.)

[Bloodsoul]

As I understand the comments, this should generally be possible, but it isn't. Then I don't get the idea of why it is labeled with a milestone that seem to say something like "We're not sure, if we want to do something about it at all".
Does it need additional work what must be done in the nextcloud core? Maybe then it should be linked to an issue filed there?

[jancborchardt]

@Bloodsoul this might have been because the milestone was given before we named the milestones. But in any case the "0.4 maybe someday" is a placeholder for anything after a possible 0.3 cause we can’t really look that far in the future. :)

Regarding what is needed, @eneiluj explained it a bit above. If someone is interested in helping with the implementation I’m sure he can explain it more. :)

[Bloodsoul]

Thanks for clarification and changing it to 0.3. Maybe the milestone needs a better name then ;) But that's off-topic here.

[slackfan]

In case the external storage feature of Nextcloud is not used, a workaround could be to change file apps/maps/lib/Service/PhotofilesService.php the way that

     private function gatherPhotoFiles ($folder, $recursive) {
         $notes = [];
         $nodes = $folder->getDirectoryListing();
         foreach ($nodes as $node) {
             if ($node->getType() === FileInfo::TYPE_FOLDER AND $recursive) {
                 // we don't explore external storages for which previews are disabled
                 if ($node->isMounted()) {
                     $options = $node->getMountPoint()->getOptions();
                     if (!$options['previews']) {
                         continue;
                     }
                 }
                 $notes = array_merge($notes, $this->gatherPhotoFiles($node, $recursive));
                 continue;
             }
             if ($this->isPhoto($node)) {
                 $notes[] = $node;
             }
         }
         return $notes;
     }

becomes

     private function gatherPhotoFiles ($folder, $recursive) {
         $notes = [];
         $nodes = $folder->getDirectoryListing();
         foreach ($nodes as $node) {
             if ($node->getType() === FileInfo::TYPE_FOLDER AND $recursive) {
                 $notes = array_merge($notes, $this->gatherPhotoFiles($node, $recursive));
                 continue;
             }
             if ($this->isPhoto($node)) {
                 $notes[] = $node;
             }
         }
         return $notes;
     }

At least I found no further flaws with that.

[Skip75]

Hello
If I understand well, this is not an issue, but a confirmed improvement/enhancement request for future releases ?

[gabrielklein]

Probably related
nextcloud/photos#129

[sirjaymond]

I do not know, whether it is related, but I saw following error in Log repeatedly: [PHP] Error: Division by zero at /html/owncloud/apps/maps/lib/Service/PhotofilesService.php#420 GET /cron.php from 89.12.216.19 at 2019-09-15T19:06:26+00:00 Julien Veyssier notifications@github.com schrieb am So., 15. Sep. 2019 21:21:
…

I'm also getting a similar error:

Division by zero at /var/www/html/custom_apps/maps/lib/Service/PhotofilesService.php#407

This error is currently appearing for every photo I have, but only after I've manually added them to the job queue using occ maps:scan-photos, and the cron job has run.

Is there any advice for fixing this issue?

[Chris-A-sc]

In case the external storage feature of Nextcloud is not used, a workaround could be to change file apps/maps/lib/Service/PhotofilesService.php the way that

     private function gatherPhotoFiles ($folder, $recursive) {
         $notes = [];
         $nodes = $folder->getDirectoryListing();
         foreach ($nodes as $node) {
             if ($node->getType() === FileInfo::TYPE_FOLDER AND $recursive) {
                 // we don't explore external storages for which previews are disabled
                 if ($node->isMounted()) {
                     $options = $node->getMountPoint()->getOptions();
                     if (!$options['previews']) {
                         continue;
                     }
                 }
                 $notes = array_merge($notes, $this->gatherPhotoFiles($node, $recursive));
                 continue;
             }
             if ($this->isPhoto($node)) {
                 $notes[] = $node;
             }
         }
         return $notes;
     }

becomes

     private function gatherPhotoFiles ($folder, $recursive) {
         $notes = [];
         $nodes = $folder->getDirectoryListing();
         foreach ($nodes as $node) {
             if ($node->getType() === FileInfo::TYPE_FOLDER AND $recursive) {
                 $notes = array_merge($notes, $this->gatherPhotoFiles($node, $recursive));
                 continue;
             }
             if ($this->isPhoto($node)) {
                 $notes[] = $node;
             }
         }
         return $notes;
     }

At least I found no further flaws with that.

This partially solves the problem. However, new images will only show in the map if you start the maps:scan-photos command manually - as long as they are uploaded to a shared folder. Of course this is not ideal since depending on the amount of images, it can take a long time to do the scan.

Any advice how this can be avoided?

[slackfan]

I don't see how this should relate, but anyways, I have a cron job for the maps:scan-photos task.

[Chris-A-sc]

On some systems the rescan, no matter if started via cron or manually, of all photos will take a long time to finish. As I understood normally when you upload a photo, the maps app will only add the single image as a job and won't rescan everything. Which in my opinion would also be the preferred way for group folders.

[tacruc]

I don't see how this should relate, but anyways, I have a cron job for the maps:scan-photos task.

Just don't. It will rescan all pictures on your system and probably delay all other brackground jobs until it is finished. And at least on my picture collection this takes days.
It is not a problem doing it once, but you shouldn't do it via cron.

[tacruc]

So basically, the problem is that there is no equivalent to https://docs.nextcloud.com/server/stable/developer_manual/app/events.html#filesystem-root for group folders, or?
@rullzer @nickvergessen A'm I wrong here? I couldn't find anything in the event docs. Is it clear if those events are comming some day, or it there a more fundamental problem?

[nickvergessen]

Should be triggered as well. Maybe you can check the activity for differences?

[stuckinger]

How are the chances oft any progress here? I love the map feature for GPS info in exif. More than 95% oft my pictures are not scanned though because they are mostly stored in groupfolders (not external storage!).

I had no deep look into that code, but could it be possible to treat those as one user representing all group folders. Say rescan on occ giving 'group' as userid would scan group folders?
Still the need for the 'on upload' trigger... Where is the job trigger handling the single scan and why is it not triggered for groupfolders?

[iamklaus]

+1 I have the same request, should be too complicated, right?

[tacruc]

+1 I have the same request, should be too complicated, right?

Feel free happy for the PR.

[isaacolsen94]

Has there been a fix/improvement to enable to location of Group Folders? I followed this threat but it's a bit convoluted on if it is or isn't possible?

[Superharz]

I would also be interested in this feature.

[Superharz]

The Nextcloud Cookbook integration has an option which looks for me similaire to this feature: It lets you select where to store and look for recipes. It also allows selecting Group Folders as storage place.

[GHBLoos]

Hello,

I read the discussion, but I am not sure what the status is. In my case:

• Photos that are in a group folder do not show in the photo app under tab 'your photos' .
• They are shown under tab 'shared with you'.

Is that currently expected behaviour?

[tacruc]

Just tested on v0.2.5 and it worked for shares and groupfolders.

[alabiana]

Just tested on v0.2.5 and it worked for shares and groupfolders.

Is there anything special to do to get the images in the Map? I have many pictures in a groupfolder and none is shown. If i copy the pictures to a personal folder they get shown after the next cron run.

[Wicloz]

I'm using maps v1.0.2 and also don't see the photos from group folders.
I tried running php occ maps:scan-photos but this doesn't help. Doing so only shows photos owned by users in the console.

[tacruc]

@Wicloz is your cronjob running?

[Wicloz]

Background jobs are running properly in Cron mode. Moving the images to my personal folder causes them to appear on the map after a while.