UX: Auto Import / Index

[p-baum]

I would prefer to automate the import of files that find their way to my Import directory. How about some kind of Import directory polling/watching?

[lastzero]

That's basically what happens when you use our integrated WebDAV sync feature. We think about adding more options / workers to run additional operations in the background, although it might impact performance for some users and consume additional energy (for example, because the CPU frequently wakes up from sleep).

[p-baum]

What/where is the integrated WebDAV sync feature?

[p-baum]

Actually the use case I have in mind here is for importing from the filesystem for which the WebDAV feature bears no relevance.

[lastzero]

Our copy command works from any local directory and will only import new files without modifying the source. You can easily run it with cron. For remote file systems / APIs, we will provide additional support in the future.

[p-baum]

Ok. I was hoping to avoid cron. Can of worms on docker.

[RAYs3T]

@p-baum You could define the cron on the host-system and pass the command to the docker. Using this method you wouldn't have to deal with modification to the original image / container.

For example:

*/10 * * * * docker exec <container-name> /bin/bash echo "This is executed in the container..."

[lastzero]

Reason we don't import / index originals automatically is that we don't know when files are added / modified. So it might happen that PhotoPrism tries to index a file that is currently being uploaded, and that will lead to an error (at best). Maybe not a big deal, but it's better to run indexing operations when you know other apps don't perform changes at the same time or at a time you know is safe.

[RyanRaw]

Reason we don't import / index originals automatically is that we don't know when files are added / modified. So it might happen that PhotoPrism tries to index a file that is currently being uploaded, and that will lead to an error (at best). Maybe not a big deal, but it's better to run indexing operations when you know other apps don't perform changes at the same time or at a time you know is safe.

Excuse me, I want to create a timed task to automatically index photos at 2 AM, how do I do it?

[RAYs3T]

@rainsuns If you're on linux you could create a cron job that executes a command in the docker container using docker exec

For example (on your host system) use crontab -e to edit your cronjobs.
Make sure to do this for either the root user (crontab -e -u root) or a user that is in the docker group so it can execute commands against the docker daemon.

Add a new line to the cron file

0 2 * * *    /usr/bin/docker exec -it <name_of_photo_prism_container> /photoprism/bin/photoprism index

Replace <name_of_photo_prism_container> with the name of your PP container (docker ps will tell you the name).

I'm using the docker flag -i so that you can see the output in the system logs.

Extra tip, for more special times here is a great site for cron timings

[thelioncape]

I think this would be a useful feature - similar to how the blackhole idea works with bittorrent clients.
The service would check a blackhole directory for photos, videos, etc. periodically (every few minutes? It's not an intensive operation to list files in a directory) and then move and process the files into folders in the photo library. I'm happy to try making something like this if it's something that's likely to be implemented

[Engr-AllanG]

I am currently trying to determine the best way to autosync as well, so I can have auto upload with my phone. I tried out Nextcloud but had issues with Nextcloud's auto upload and duplicate files. I trieid Seafile but looks like android auto uplaod is currently broken. Regardless I don't really want a duplicate library in Nextcloud. Next I tried FolderSync on android to upload via webdav directly to originals folder. But as discussed in this thread, you have to manually initiate indexing or use Cron or something. I suppose I could have the cron job run every few minutes, but having it auto scan a folder sounds better. I don't know anything about cron yet.

The other way I'm looking at doing it is in conjunction with FolderSync, creating a webdav server or adding webdav to my samba server and then using the already exiting photoprism sync-via-webdav feature. The downside to this is currently photoprism sync is limited to a minimum of once an hour. I'm trying to replace my phone photo app, and this requires 'instant' upload so that I could share via public link a photo/video right after taking the picture (which I do quite a bit, mostly for large videos)

[voarsh2]

0 2 * * * /usr/bin/docker exec -it <name_of_photo_prism_container> /photoprism/bin/photoprism index

This is helpful, I haven't looked into WebDav, but indexing for me, for my existing files on my NAS, seems a much easier way to go for me, so I think this should be documented/made more norm.

[lastzero]

This will automatically trigger indexing / importing after WebDAV uploads, deleting, renaming, editing, copying, and moving.

We still don't use a general file system change listener or simple timer because...

1. not all operating systems and file systems support notifications
2. these have a file count limit which some users may exceed
3. there is a certain risk this may cause feedback loops or bad performance, e.g. when auto importing to originals triggers the auto indexer (not needed since the importer already added new files to the index) or in combination with external apps and events
4. if we don't use a listener and just start indexing "randomly", there is a big risk of conflicts with other applications / user operations happening at the same time

I'm willing to look further into is when someone explains why 1 - 4 are no problems, and should better be ignored 👻

[lastzero]

Only watching the import folder may be easier as there typically shouldn't be as many files. We may still have to deal with users who import their complete, existing library with potentially millions of files. Those edge cases shouldn't break the app, or crash the operating system.

[swingstate]

Testing this now with Iphone + PhotoSync. After the upload, when should the photo become visible?

[lastzero]

After it has been imported. Auto refresh depending on your sort order. Try recently added.

[lastzero]

You can configure the delay using the config options listed in our docs.

[swingstate]

Thanks for the hint. Manual import worked well and now out import & index does too.

@ich777 Can you add

• PHOTOPRISM_AUTO_INDEX | --auto-index SECONDS | auto indexing safety delay in SECONDS (WebDAV)
• PHOTOPRISM_AUTO_IMPORT | --auto-import SECONDS | auto importing safety delay in SECONDS (WebDAV)

to the Unraid template?

I am using 60 seconds delay for import and 120 seconds for index as default values at the moment.

[lastzero]

Delay should depend on your network speed and file size. If you're uploading videos from China, use a longer delay. 60s should work for importing images locally. Our default is 15 minutes, so pretty conservative.

[swingstate]

good hint. I would upload locally only until 2FA is in place ;o)

[lastzero]

VPN FTW 🥁

[ich777]

@swingstate is this really needed?
If someone really needs this he can adds this on his own, what I can add and makes sense to me is a link to the Documentation since PhotoPrism will change much things because it's in heavy development from what I've seen.

[lastzero]

@ich777 Heavy development? We're implementing your feature requests.... to @swingstate's point: I think this is an important setting, as many users will want to index and import automatically... for this reason, we've implemented this on a weekend and didn't wait until next year 👍

[adyanth]

Planning to use this for calling from a Kubernetes CronJob.

import requests

baseurl = "https://photoprism.domain.com"

auth = {"username": "admin", "password": "pass"}
res = requests.post(
    f"{baseurl}/api/v1/session", json=auth, verify=False
)
res.raise_for_status()
session = {"X-Session-ID": res.headers["x-session-id"]}

data = {
    "path": "/", "convert": True,
    "rescan": False, "skipArchived": False
}
res = requests.post(
    f"{baseurl}/api/v1/index", json=data,
    headers=session, verify=False
)
res.raise_for_status()
print(res.json())

[mmontes11]

@adyanth I also used a CronJob for this, but calling the indexsubcommand instead:

https://github.com/mmontes11/charts/blob/main/charts/photoprism/templates/cronjob.yml#L36

[adyanth]

@mmontes11 looks like that would need an RWX volume mount for photoprism? I am using the sqlite DB, so that won't be a good idea for me. I preferred connecting to the existing photoprism service and call the index API.

Edit: This works well for me.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: "photoprism-cron"
spec:
  schedule: "@daily"
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 2
  successfulJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: cron
              image: python:alpine
              imagePullPolicy: IfNotPresent
              command:
                - sh
              args:
                - -c
                - "pip install requests; clear; python /scripts/index.py"
              envFrom:
                - secretRef:
                    name: photoprism
              volumeMounts:
                - mountPath: /scripts
                  name: scripts
          volumes:
            - name: scripts
              configMap:
                name: photoprism-scripts
          restartPolicy: Never
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: photoprism-scripts
data:
  index.py: |
    import os
    import requests

    baseurl = os.environ.get("PHOTOPRISM_BASE_URL", "http://photoprism:2342")

    auth = {
        "username": os.environ.get("PHOTOPRISM_ADMIN_USERNAME", "admin"),
        "password": os.environ["PHOTOPRISM_ADMIN_PASSWORD"]
    }
    res = requests.post(
        f"{baseurl}/api/v1/session", json=auth, verify=False
    )
    res.raise_for_status()
    session = {"X-Session-ID": res.headers["x-session-id"]}

    print("---> Logged in, indexing now...")

    data = {
        "path": os.environ.get("PHOTOPRISM_INDEX_PATH", "/"),
        "convert": os.environ.get("PHOTOPRISM_INDEX_CONVERT", "true") == "true",
        "rescan": os.environ.get("PHOTOPRISM_INDEX_RESCAN", "false") == "true",
        "skipArchived": os.environ.get("PHOTOPRISM_INDEX_SKIPARCHIVED", "false") == "true",
    }
    res = requests.post(
        f"{baseurl}/api/v1/index", json=data,
        headers=session, verify=False
    )
    res.raise_for_status()
    print(res.json())
    print("Complete")

[mgrove36]

I've been using docker exec -it photoprism /photoprism/bin/photoprism import which has stopped working with the error stat /photoprism/bin/photoprism: no such file or directory Does anyone know what the path for automating the import call should be now?

@SiJiL82 you can find out with docker exec -it which photoprism. I believe it's now /opt/photoprism/bin/photoprism

[azukaar]

I don't really understand the lengthy conversation about not simply supporting auto-importing / auto-indexing. Out of hundreds of self hosted media solution (Movies, images, musics, etc..) Photoprism is literally is only one software that exist that does not support this... Imagine having to manually hit scan everytime your library changes in Plex, Jellyfin, Airsonic or Navidrome?
All those software managed to do this quite easily day one, because it's not complex to scan a folder periodically and check for new / missing / edited files, and most of them even get away using FS watchers optionally for faster response.

This is honestly a huuuuge let down from a UX perspective and is honestly not really justified, especially when the answer ends up being "We can't do it because XYZ... But, oh! just do it yourself with some CRON job!". If the recommendation ends up being "scan periodically" anyway you might as well implement it, no?

[ITCMD]

I agree. I've made a script that indexes every 5 minutes if nobody is on one of the game servers sharing the hardware, but it's less than ideal.

Takes 1-10 minutes for a photo I take to show up in my Photoprism library. I honestly am confused why this isn't being treated as a very high priority issue.

[ITCMD]

I think someone said that indexing should work if it's uploaded over webdav. I do want to confirm that this is not true. Photoprism will not index photos uploaded over webdev until an index command is run to docker.

[lastzero]

Yes, automatic indexing and import will work if you use WebDAV. Since our users have a wide variety of file systems and operating systems, we will not release something that has only been tested with the default file system on Linux, as this could result in a huge number of support requests that we will not be able to handle. Also, we are currently working on multi-user support, which many are also waiting for. Feel free to support us by contributing or becoming a sponsor if you haven't signed up yet.

[azukaar]

The very vast majority of the system supports file watchers, but that's beside the point, since you don't actually need watchers to simply trigger a periodic scan right?
The "solution in the middle" / "low effort solution" I was proposing was simply having the option to have periodic scan without the sketchy CRON setup behind it.

[lastzero]

Why Cron? It's a few lines of YAML and we even provide a tested example: https://dl.photoprism.app/docker/scheduler/

[ITCMD]

Yes, automatic indexing and import will work if you use WebDAV. Since our users have a wide variety of file systems and operating systems, we will not release something that has only been tested with the default file system on Linux, as this could result in a huge number of support requests that we will not be able to handle. Also, we are currently working on multi-user support, which many are also waiting for. Feel free to support us by contributing or becoming a sponsor if you haven't signed up yet.

Like I said though, automatic indexing does not occur for me when I upload via webdav.

Is there some setting I need to enable for that to work?

[graciousgrey]

@ITCMD it's triggered after a safety delay that can be configured with

PHOTOPRISM_AUTO_INDEX

https://docs.photoprism.app/user-guide/library/originals/#automatic-indexing

In case this does not work feel free to open a bug report following this template --> https://photoprism.app/kb/reporting-bugs

[graciousgrey]

We understand that it is much more convenient to set up job automation (e.g. for indexing) in the user interface than to use our examples from the documentation. The same is true for some other config options ( Even if there is not yet an issue for every option).
We have this on our radar, and the long term goal is for the end user to be able to do as much as possible through the GUI. It's just a matter of development resources and time.

Feel free to leave a 👍 there: #2495

[azukaar]

Why Cron? It's a few lines of YAML and we even provide a tested example: https://dl.photoprism.app/docker/scheduler/

Whether you use CRON or another scheduler, doesnt change the DIY aspect of it, esp. for people not using your docker-compose right away / at all.

Feel free to leave a 👍 there: #2495

Done! Here's also a good inspiration. Navidrome is one of the best performing (if not the best) software in that category. This makes it much cleaner, no need for external scheduler, just a simple setup

https://github.com/navidrome/navidrome/blob/master/scheduler/scheduler.go

[lastzero]

Thank you! We appreciate your feedback. As Theresa said, we are aware that many details can and should be improved. However, we cannot and should not work on everything at the same time. In general, we try to focus on features and improvements for which no other tools or workarounds exist. I fully agree with you that implementing a built-in scheduler or watcher is technically not a problem and makes perfect sense once we get to it.

Without having tested Navidrome (looks great!): Finding the right strategy for indexing photos and videos is generally much harder than for audio files, especially if you don't just have single JPEGs. Music files usually don't have extremely complex metadata split across files and formats. Also, probably 99.99% of users never edit them. While WAV could be considered a RAW format of sorts, most users only have a single file per track, which is also often nicely organized into albums.

In practice, a single "photo" may consist of 3 or more files, some of which may be created hours later - e.g. manually in Photoshop - and may have a completely different size or use (such as JPEG, RAW, and XMP, which is not itself an image, so indexing it simply because it was uploaded or otherwise added makes little sense):

• https://docs.photoprism.app/getting-started/faq/#what-are-sidecar-files-and-where-do-i-find-them

[mfed3]

I am giving you honest feedback here, not supporting an auto import feature makes this already niche product useless to 99.9% of people who would ever possibly use it. No one but a techie on linux would ever be able to figure out how to set up cron jobs, so therefore I can never recommend this software to any friends or family for it to work out of the box in a home server. Sometimes developers have no common sense in understanding the priority should be to listen to what the community wants not what you want to develop, otherwise your product will die on the vine. Due to the attitude I see here I am most likely going to be shutting down my docker container and putting this product in the "tried it, doesn't meet my needs" bin. Good luck on your next project.

[lastzero]

You can use WebDAV for auto index & import:
https://docs.photoprism.app/user-guide/sync/webdav/

Background jobs for other use cases and less experienced users will be added later.

Please don't use GitHub Issues for questions and general comments. We provide other forums for that. Thank you! ❤️

[mfed3]

I'm going to go ahead and call out your bull**** right now. I am essentially expressing to you as a customer that I am experiencing this same issue, and would like this feature that you closed implemented. Please don't use GitHub Issues to try to one up people and give yourself a false sense of superiority when you simply can't take the criticism. Thank you!

[heikomat]

@mfed3 to be fair

1. you're not a customer, as this is free, open source software. No one owes you anything here.
2. There are other, non-closed issues regarding this topic.
3. @lastzero doesn't say things won't be improved upon. Just that these developments wont happen right now, but probably at some other time.
4. Setting up any server-application requires some tech-skills. That's why non-techies like to use things like google photos, because that is already setup for them. If you want to host your own, you need to learn how to do that.

As @lastzero said, please use issues to discuss issues with the software. You can use for example the community chat for other questions or comments

[lastzero]

@mfed3 👉 https://photoprism.app/code-of-conduct

[evertinker]

Just leave out the path.

docker exec -it photoprism photoprism import

is there a way to call import and specify the option of 'move' vs 'copy'? When I call it the way you describe it seems to ignore my config file/option checked in web interface and default to 'move'.

[lastzero]

docker exec -it photoprism photoprism help

[evertinker]

thanks, that's what I needed!

[graciousgrey]

Related open issues:
#2608
#2495

[arendtio]

Thanks for the Ofelia example, I will try that 👍

In addition, I would like to express my opinion, that I would love to see some kind of scheduling of the index job as part of Photoprism. Sure, I would prefer some fancy real-time filesystem change notification system, but simply having the option to let the job run once per day would be completely sufficient. And if making it part of the software is a complete no-go please add a hint in the documentation referencing the Ofelia example.

To give some context: I have a hard drive which is my repository for all pictures from multiple devices. Some transfer their pictures automatically via ssh, others get transferred manually. The repository is also available via NFS for desktop computers and Photoprism has read-only access. Therefore, neither the import functionality nor the WebDAV interface seem to be viable options. Nevertheless, I would like to see yesterday's photos in Photoprism if they got transferred overnight.

[lastzero]

Once we have a budget to spend, we are happy to provide all of this! 🥳

[arendtio]

I ended up using Chadburn (a fork of Ofelia) because I like the label syntax better (no need for the container name; no extra config file; matches the style of the Traefik configuration) and Chadburn does "not require a dummy task on the Chadburn container", which Ofelia does... However, Chadburn uses the standard 5-part cron syntax instead of the 6-part syntax that Ofelia uses for the scheduling format. In addition, it might be helpful to know that the timings are UTC timings by default.

An example of the labels can be found here: https://www.reddit.com/r/photoprism/comments/ygug5j/scheduler_with_ofelia/

[lastzero]

That could be something to contribute to the Advanced section in the Getting Started docs!