Raspberry Pi 4 crashes while indexing

[breakingflower]

Installation details

I followed the official installation guide here.

Hardware: RPi 4 4gb @ 2100 MHz with active cooling
OS: Ubuntu Server 20.10 arm64
The server has all images stored on an ssd on which the OS runs.

Running everything as services in compose.

Compose file

  photoprism:
    # Use photoprism/photoprism-arm64:preview instead for testing preview builds:
    image: photoprism/photoprism-arm64:latest
    container_name: photoprism
    # Only enable automatic restarts once your installation is properly
    # configured as it otherwise may get stuck in a restart loop:
    # https://docs.photoprism.org/getting-started/faq/#why-is-photoprism-getting-stuck-in-a-restart-loop
    restart: unless-stopped
    security_opt:
      - seccomp:unconfined
      - apparmor:unconfined
    ports:
      - ${PHOTOPRISM_PORT}:2342 # [local port]:[container port]
    environment:
      PHOTOPRISM_ADMIN_PASSWORD: "REDACTED"          # PLEASE CHANGE: Your initial admin password (min 4 characters)
      PHOTOPRISM_HTTP_PORT: 2342                     # Built-in Web server port
      PHOTOPRISM_HTTP_COMPRESSION: "none"            # Improves transfer speed and bandwidth utilization (none or gzip)
      PHOTOPRISM_WORKERS: 2                          # Limits the number of indexing workers to reduce system load
      PHOTOPRISM_DEBUG: "false"                      # Run in debug mode (shows additional log messages)
      PHOTOPRISM_PUBLIC: "false"                     # No authentication required (disables password protection)
      PHOTOPRISM_READONLY: "false"                   # Don't modify originals directory (reduced functionality)
      PHOTOPRISM_EXPERIMENTAL: "false"               # Enables experimental features
      PHOTOPRISM_DISABLE_WEBDAV: "false"             # Disables built-in WebDAV server
      PHOTOPRISM_DISABLE_SETTINGS: "false"           # Disables Settings in Web UI
      PHOTOPRISM_DISABLE_TENSORFLOW: "true"         # Disables using TensorFlow for image classification
      PHOTOPRISM_DARKTABLE_PRESETS: "true"           # Enables Darktable presets and disables concurrent RAW conversion
      PHOTOPRISM_FFMPEG_ENCODER: "h264_v4l2m2m"    # FFmpeg AVC encoder for video transcoding (default: libx264)
      # PHOTOPRISM_FFMPEG_BUFFERS: "64"              # FFmpeg capture buffers (default: 32)
      PHOTOPRISM_DETECT_NSFW: "false"                # Flag photos as private that MAY be offensive
      PHOTOPRISM_UPLOAD_NSFW: "true"                 # Allow uploads that MAY be offensive
      # PHOTOPRISM_DATABASE_DRIVER: "sqlite"         # SQLite is an embedded database that doesn't require a server
      PHOTOPRISM_DATABASE_DRIVER: "mysql"            # Use MariaDB (or MySQL) instead of SQLite for improved performance
      PHOTOPRISM_DATABASE_SERVER: "photoprism_db:3306"     # MariaDB database server (hostname:port)
      PHOTOPRISM_DATABASE_NAME: "photoprism"         # MariaDB database schema name
      PHOTOPRISM_DATABASE_USER: "photoprism"         # MariaDB database user name
      PHOTOPRISM_DATABASE_PASSWORD: "${PICTURES_DB_USER_PW}"       # MariaDB database user password
      PHOTOPRISM_SITE_URL: "REDACTED"  # Public PhotoPrism URL
      #PHOTOPRISM_SITE_URL: "http://localhost:2342/"  # Public PhotoPrism URL
      PHOTOPRISM_SITE_TITLE: REDACTED"
      PHOTOPRISM_SITE_CAPTION: "REDACTED"
      PHOTOPRISM_SITE_DESCRIPTION: ""
      PHOTOPRISM_SITE_AUTHOR: "REDACTED"
      PHOTOPRISM_WORKERS: 1
      # You may optionally set a user / group id using environment variables if your Docker version or NAS does not
      # support this natively (see next example):
      #UID: ${PUID}
      #GID: ${PGID}
      # UMASK: 0000
    # Uncomment and edit the following line to set a specific user / group id (native):
    user: "${PUID}:${PGID}"
    # For hardware AVC transcoding using the h264_v4l2m2m encoder:
    devices:
      - "/dev/video11:/dev/video11"
    volumes:
      # Your photo and video files ([local path]:[container path]):
      - "${SSD_MEDIA_ROOT}/photoprism:/photoprism/originals"
      - "${SSD_MEDIA_ROOT}/photoprism_import:/photoprism/import"
      # Multiple folders can be indexed by mounting them as sub-folders of /photoprism/originals:
      # - "/mnt/Family:/photoprism/originals/Family"    # [folder_1]:/photoprism/originals/[folder_1]
      # - "/mnt/Friends:/photoprism/originals/Friends"  # [folder_2]:/photoprism/originals/[folder_2]
      # Mounting an import folder is optional (see docs):
      # - "~/Import:/photoprism/import"
      # Permanent storage for settings, index & sidecar files (DON'T REMOVE):
      - "${SSD_APPDATA_ROOT}/photoprism:/photoprism/storage"
    depends_on:
      - photoprism_db

  photoprism_db:
    image: arm64v8/mariadb:10.5
    container_name: photoprism_db
    restart: unless-stopped
    security_opt:
      - seccomp:unconfined
      - apparmor:unconfined
    command: mysqld --transaction-isolation=READ-COMMITTED --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --max-connections=512 --innodb-rollback-on-timeout=OFF --innodb-lock-wait-timeout=50
    volumes: # Don't remove permanent storage for index database files!
      - "${SSD_APPDATA_ROOT}/photoprism_db:/var/lib/mysql"
    environment:
      MYSQL_ROOT_PASSWORD: ${PICTURES_DB_ROOT_PW}
      MYSQL_DATABASE: photoprism
      MYSQL_USER: photoprism
      MYSQL_PASSWORD: ${PICTURES_DB_USER_PW}

The Problem

The server crashes when importing larger (e.g. > 3) amounts of files. Have tried reducing the amount but manually uploading 3 files every time is tedious. Have to physically reset the server after such a crash.

I suspect this is related to the thumbnail creation module, but is there a way to stage the thumbnail creation so that it takes longer, but doesn't crash?

Steps I've taken to solve this

Everything on the documentation page, including:

• Disabling TF
• Creating a swap partition (8gb)
• set PHOTOPRISM_WORKERS=1
• Disable hardware decoding by commenting out PHOTOPRISM_FFMPEG_ENCODER

Extra info

Below image shows an ssh connection on the moment the server crashes.

• Left: info messages from photoprism docker logs -f photoprism.
• Middle: swapon -s to see swap usage. It can be observed that swap is not entirely used.
• Right: vcgencmd measure_temp - a module to measure temperatures on a raspberrypi CPU.
  

[kvalev]

Just FYI, setting swap would not help you anyway, as docker does not take swap memory into account, at least not per default. What you can try and do is to exclude larger files from being reindexed using PHOTOPRISM_ORIGINALS_LIMIT (see #1135). You would have to docker-compose down your instance for the setting to take effect.

[breakingflower]

Thanks for the tip about the swap, I will look into using the --memory-swap parameter.

w.r.t. the limit value, it does not seem to change anything, as most of my images are low-medium quality (<10mb) and videos <50mb. I think this is more related to the amount of images being thumbnailed at the same time, although I have no real way to confirm this.

[lastzero]

When there is only one worker, then only one file should be processed at a time. You may run photoprism config in a terminal to see the current config values so that you can be sure the values set via environment variables are actually used and it's not a config problem.

[lastzero]

Note that the swapon -s parameter is deprecated. Not sure if it makes a difference, but it's better to use free -h instead. Shows you all current memory usage and not just swap.

[breakingflower]

I decided to dig into the log files of the server and found that there was a recurring error related to cgroups, which was thrown by the docker service.

After googling, I found a solution enable cgroups for memory in my /boot/firmware/cmdline.txt:
cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1.
The error dissappeared afterwards.

I did the usual sudo apt update && sudo apt upgrade && docker-compose pull && docker-compose up -d and am no longer experiencing crashes.

To test I have added a large (900 images) library to photoprism with no negative observed effects. In the end I can't comment on what solved the issue, but in my opinion we can close this.

[lastzero]

@graciousgrey should add this to our docs

[breakingflower]

Would be good to have at least one instance of another user facing the same issue and resolving with the same solution.

I would reverse the cgroup trick to verify if this was actually the solution, but currently have too many live services to test. Perhaps it was a simple docker update that did it? Unsure.

[kavemang]

Would be good to have at least one instance of another user facing the same issue and resolving with the same solution.

I would reverse the cgroup trick to verify if this was actually the solution, but currently have too many live services to test. Perhaps it was a simple docker update that did it? Unsure.

I just had to implement the settings you gave here to get my indexing working again on a RPI4, that said I have no idea why indexing stopped working to begin with as it was working back in october during my last photo dump

[lastzero]

From what I understood reading related blog posts and documentation,cgroups are only needed if you want Docker to restrict memory or swap usage:

• https://serverfault.com/questions/790318/cannot-enable-cgroup-enable-memory-swapaccount-1-on-gce-debian-jessie-instance
• https://fabianlee.org/tag/cgroup_enable/

This isn't part of our default configuration. Did you change the configuration to limit memory usage? Do you use Kubernetes or any other container management system that might do this implicitly?

[lastzero]

If you set a hard memory limit for PhotoPrism (or all Docker containers in general) and it then comes across a large file that temporarily needs more memory while indexing, this would also explain why the container is killed and it looks like PhotoPrism has "crashed".

[lastzero]

Updated our config examples to include the note:

• https://dl.photoprism.org/docker/arm64/docker-compose.yml
• https://dl.photoprism.org/docker/armv7/docker-compose.yml

[lastzero]

Getting Started docs have been updated as well:

• https://docs.photoprism.org/getting-started/raspberry-pi/

[kavemang]

From what I understood reading related blog posts and documentation,cgroups are only needed if you want Docker to restrict memory or swap usage:

• https://serverfault.com/questions/790318/cannot-enable-cgroup-enable-memory-swapaccount-1-on-gce-debian-jessie-instance
• https://fabianlee.org/tag/cgroup_enable/

This isn't part of our default configuration. Did you change the configuration to limit memory usage? Do you use Kubernetes or any other container management system that might do this implicitly?

at the moment I have no restrictions on my prism container, although my RPI4 is the 4GB ram model not 8

the only difference between these photos and my oct dumb that I have found so far is that the latest photos were vertical frame (an older relative took them) and were from a different camera model although I don't think they were particularly large resolution

[lastzero]

But now it's working flawlessly?

[kavemang]

But now it's working flawlessly?

Yesterday it crashed with the same error:
/entrypoint.sh: line 94: 9 Killed "$@"
but all it was doing was:

level=info msg="index: updating counts"

level=info msg="index: updating covers"

I've also noticed it seems to be doing that ^ every few seconds 24/7 is that normal if no one is viewing the webfront?

[lastzero]

Is this our latest release? Log level should be debug for this now. Depends on the wakeup interval which is configurable.

[kavemang]

im on latest branch, here is some of my settings, any recommendations for running from a RPI4 with 4GB ram which is rooted and booting on an SSD?

PS: my workers is set to 0 but not sure why it defaulted to that to start, im guessing I can set it to 4 as the pi has 4 physical cores?

[lastzero]

photoprism/photoprism:latest?

See https://docs.photoprism.org/getting-started/raspberry-pi/

[kavemang]

photoprism/photoprism:latest?

See https://docs.photoprism.org/getting-started/raspberry-pi/

yes latest, thanks I've added "arm_64bit=1" to my firmware config and redeployed with debug turned on, will follow up if I get useful results/ crash logging