Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Index: Support of symbolic links to files #1049

Closed
JoshMock opened this issue Feb 15, 2021 · 18 comments
Closed

Index: Support of symbolic links to files #1049

JoshMock opened this issue Feb 15, 2021 · 18 comments
Assignees
Labels
idea Feedback wanted / feature request released Available in the stable release

Comments

@JoshMock
Copy link

I use git-annex to manage and sync my photo library across machines. Each file is replaced with a symlink and stored elsewhere. When I point PhotoPrism at my main git-annex directory, they get imported as folders but not files.

Log example:

 2021-02-15 15:08:14 INFO folder: added album 21.36.04.Jpeg (path:2017/09/02/21.36.04.jpeg public:true)

I did verify that no symlinks are broken on the system where PhotoPrism is running.

version: 210211-b9595dd4-Linux-aarch64
OS: Raspbian 10 (arm_64bit=1 is set)
Running via docker-compose

@JoshMock JoshMock changed the title Symlinked files are not imported Symlinked files are imported as folders Feb 15, 2021
@lastzero
Copy link
Member

Sorry, symlinked files are currently not supported. Need to focus on other issues first.

@lastzero lastzero added the idea Feedback wanted / feature request label Feb 15, 2021
@lastzero
Copy link
Member

Also links can't be imported, just indexed. Only files can be imported.

@graciousgrey graciousgrey changed the title Symlinked files are imported as folders Support indexing of symlinked files Nov 2, 2021
@graciousgrey graciousgrey added the low-priority Everything nice to have but not so important label Nov 2, 2021
@jamesread
Copy link

After asking in chat, @graciousgrey kindly pointed me to this issue (fellow Git Annex user/lover here!). After a brief discussion with @lastzero, he kindly pointed out several complexities with symlinks that need to be considered (obviously not an exhaustive list);

  • A symlink check on each filesystem access would be necessary, to check that the correct mtime, etc are being used (sha1 fingerprinting should not make a difference)
  • Needs to be a mechanism for avoiding the scanning of infinite loops on the filesystem

In addition, I can think of;

  • How to represent "broken" symlinks in the UI?
  • Should the UI show a little shortcut icon over symlinked items to "help" with navigation?
  • Any complexities with hard links?
  • Any symlink complexities across other platforms than just Linux (which is what I use), eg, should Windows shortcuts be in scope for support?

@JoshMock
Copy link
Author

JoshMock commented Nov 5, 2021

I totally get why this is not a straightforward issue. Thanks for adding that helpful context for why, @jamesread!

In the meantime, I'm using git-annex-export to keep a raw copy of my photo collection on a spare drive, which is what I'm pointing Photoprism at. While I do think some way of supporting symlinks would still be useful, it's an okay workaround for now, at least for git-annex users.

@lastzero lastzero changed the title Support indexing of symlinked files Index: Support of symbolic links to files Jul 6, 2022
@lastzero lastzero self-assigned this Jul 6, 2022
lastzero added a commit that referenced this issue Jul 6, 2022


Signed-off-by: Michael Mayer <michael@photoprism.app>
lastzero added a commit that referenced this issue Jul 7, 2022
Signed-off-by: Michael Mayer <michael@photoprism.app>
@lastzero
Copy link
Member

lastzero commented Jul 7, 2022

@jamesread @JoshMock Would you mind testing these changes? Of course only if you are still interested 👍

@JoshMock
Copy link
Author

JoshMock commented Jul 7, 2022

@lastzero I'll see if I can set it up soon! Is there a Docker build for this change?

@lastzero
Copy link
Member

lastzero commented Jul 7, 2022

That'd be amazing! Yes, the preview build has just been updated. The release notes for the preview are not up to date in case you wonder, but the roadmap is.

@lastzero
Copy link
Member

lastzero commented Jul 8, 2022

The release notes for our preview build have been updated:

@jamesread
Copy link

Hey @lastzero , just grabbed the preview container image now, and the first results are looking pretty good. I have images showing up nicely.

I was fumbling for a bit because I have not actually used photoprism up until this point - it looks like I've got to index 500Gb of Photos for anything to show up in the Library -> Originals browser. That's not a problem, I want all their metadata indexed :-) It's just going to take a while.

Git annex frequently has "dead" symlinks (which means that the file is stored on another server) - this is the normal operation. However, is it possible to have the indexer index the file as being "corrupt", unreadable, or something like that? It whould be helpful if the user interface saw the file, even if it can't display anything about it.

@lastzero
Copy link
Member

lastzero commented Jul 8, 2022

AFAIK broken links should be skipped while indexing, but you should see an error or warning in the logs. If a link of an already indexed file breaks, it should be marked as missing in the files table just like any deleted file.

@JoshMock
Copy link
Author

JoshMock commented Jul 8, 2022

I just started up a fresh install on my RPi4 and it seems to be scanning my symlinks! I've got thousands of images and, you know, it's an RPi4, so I'll post more updates later about how it's going.

@JoshMock
Copy link
Author

JoshMock commented Jul 8, 2022

About 10% done working through the 25k photos in my library and from what I can tell it's treating symlinks exactly as if they were files. 🎉

This'll likely take a day or two to finish, and I want to see if it makes it to the point of facial recognition etc., but so far I'm satisfied.

@lastzero lastzero added released Available in the stable release and removed low-priority Everything nice to have but not so important labels Jul 29, 2022
lastzero added a commit that referenced this issue Sep 28, 2022
Signed-off-by: Michael Mayer <michael@photoprism.app>
@lastzero
Copy link
Member

@JoshMock @jamesread please note the optimizations in #2683. These improve indexing performance by (probably) skipping unneeded symbolic filename resolutions. It's hard for us to test because we don't use symlinks on our production instances and we don't have time to set up a suitable test environment at the moment.

@JoshMock
Copy link
Author

Thanks! Is there a Docker tag that includes that change?

@lastzero
Copy link
Member

lastzero commented Oct 2, 2022

Not yet, I'm sorry. The changes for the extended multi-user support are quite big, so it might take a while this time...

@StevenMapes
Copy link

StevenMapes commented Jul 14, 2023

I was looking on the here to see if symlinks were supported and I'm a little confused now as to whether they are or not. I see that this issue was closed and some issues were identified that would stop symlinks from being supported, but then I noticed further comments suggesting support had been added and @JoshMock commenting that symlinks were being picked up.

Is there anything particular that needs to be done to support this as whilst I'm running the latest release the symlink folders are ignored. I've tried symlinking to parent folders, to the folder containing images and to directly to images but they always seem to just be skipped when I try scanning for new images.

Or is a case that hard links work and soft links don't

@lastzero
Copy link
Member

@StevenMapes Symbolic links to files and directories within the originals folder are supported if they are accessible from the environment in which your instance is running. However, you cannot mount a symbolic link as a storage folder or use links within the storage folder.

👉 https://docs.photoprism.app/known-issues/#symbolic-links

@StevenMapes
Copy link

Thanks! I missed that part. I'll swap the files around then to exist within the Originals folder then I'll symlink them back to where I need them for other applications :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
idea Feedback wanted / feature request released Available in the stable release
Projects
Status: Release 🌈
Development

No branches or pull requests

5 participants