mobile: fall back to mtime rather than ctime when no EXIF date is present

[Atemu]

The bug

Luckily, I was still testing things out but when I added other local albums on my phone, the photos' dates weren't their mtime or exif dates but the mtime of the directory containing the photos.

Edit: Far more detailed pdate below.

The OS that Immich Server is running on

NixOS

Version of Immich Server

v1.94.1

Version of Immich Mobile App

v1.94.1

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

# Modified version of https://github.com/immich-app/immich/releases/download/v1.93.3/docker-compose.yml
version: "3.8"
name: immich
services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "immich"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: 'no'
    logging:
      driver: json-file
  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "microservices"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: 'no'
    logging:
      driver: json-file
  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: 'no'
    logging:
      driver: json-file
  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:c5a607fb6e1bb15d32bbcf14db22787d19e428d59e31a5da67511b49bb0f1ccc
    restart: 'no'
    logging:
      driver: json-file
  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.1.11@sha256:0335a1a22f8c5dd1b697f14f079934f5152eaaa216c09b61e293be285491f8ee
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: 'no'
    logging:
      driver: json-file
volumes:
  pgdata: null
  model-cache: null

Your .env content

DB_DATABASE_NAME=immich
DB_HOSTNAME=immich_postgres
DB_PASSWORD=postgres
DB_USERNAME=postgres
IMMICH_VERSION=release
REDIS_HOSTNAME=immich_redis
UPLOAD_LOCATION=/var/lib/immich/

Reproduction steps

(Not sure if 1-3 matter but I added them for completeness sake; it's all I did with the app.)

1. Have existing photos uploaded from PC
2. Add local album which contains these photos too
3. Wait for sync
4. Add a few other albums with photos from varying dates
5. Let it sync
6. Photos of the other albums all have the same date

Additional information

I first thought it was the mtime of the newest file in the directory but it was the same date for multiple local albums; the date I restored a backup on my phone.

[Atemu]

I have investigated this bug further now and it still reproduces on v1.105.1.

This actually only affects a subset of my pictures and I may have found the distinguishing characteristic: They're PNGs.

It's really easy to repro by backing up your screenshots as those are typically PNGs.

Here are two pictures from the same directory taken by the same app one month apart:

$ exiftool IMG_20191104_094433.jpg | rg -i date
File Modification Date/Time     : 2019:11:04 09:44:33+01:00
File Access Date/Time           : 2024:05:24 21:14:44+02:00
File Inode Change Date/Time     : 2023:09:30 13:41:49+02:00
Modify Date                     : 2019:11:04 09:44:33
Date/Time Original              : 2019:11:04 09:44:33
Create Date                     : 2019:11:04 09:44:33
Create Date                     : 2019:11:04 09:44:33.422233
Date/Time Original              : 2019:11:04 09:44:33.422233
Modify Date                     : 2019:11:04 09:44:33.422233
$ exiftool IMG_20191222_180405.png | rg -i date # (this is not on the device)
File Modification Date/Time     : 2019:12:22 18:04:22+01:00
File Access Date/Time           : 2024:05:24 21:10:49+02:00
File Inode Change Date/Time     : 2023:09:30 13:41:56+02:00
$ adb shell ls -ld /data/media/0/DCIM/OpenCamera/
drwxrwsr-x 2 media_rw media_rw 24576 2024-05-24 20:58 /data/media/0/DCIM/OpenCamera/
$ adb shell stat /data/media/0/DCIM/OpenCamera/IMG_20191222_180405.png
  File: /data/media/0/DCIM/OpenCamera/IMG_20191222_180405.png
  Size: 20386474         Blocks: 39872   IO Blocks: 512  regular file
Device: fd05h/64773d     Inode: 137855   Links: 1        Device type: 0,0
Access: (0664/-rw-rw-r--)       Uid: ( 1023/media_rw)   Gid: ( 1057/media_image)
Access: 2019-12-22 18:04:22.000000000 +0100
Modify: 2019-12-22 18:04:22.000000000 +0100
Change: 2023-11-26 11:44:02.336289663 +0100

Apparently, I had configured the camera app to use PNG at some point between taking these. Also apparently the PNGs OpenCamera spat out don't have EXIF metadata.

The JPG gets the correct time assigned in Immich while the PNG gets assigned its ctime. As you can tell by the file name, the mtime is correct (the atime too but please don't use that). I'd expect Immich to take the

I assume this is an error in the fallback path when no EXIF data exists for the date.

Another notable fact here is that these images were manually restored after I had to wipe my phone's userdata partition which explains why ctime != mtime. This bug may not surface under "normal" conditions where ctime should almost always equal mtime.

Still, the distinction between mtime and ctime exits precisely for cases like these and should be honoured.

The bug only occurs in the app (Android, can't test iOS); uploading IMG_20191222_180405.png to the web UI works as expected.

This appears to be the nature for this bug: ctimes are used rather than mtimes when there is no exif data.

This was supposedly fixed for the server already in #4191.

Now that I know the likely root cause, this is probably a duplicate of #1861. I'll leave it up to you to either re-open that issue or continue discussing the bug here.

In that issue, @fyfrey raised the point that users may prefer ctime over mtime.

To this I'd like to raise the argument that ctime really does not make sense for such a scenario as it's updates when anything about a file changes, including its metadata its metadata in every sense; even the location in the filesystem. This perception might be rooted in the assumption that the "c" in ctime stands for "creation". The Wikipedia article on ctime clears that up:

It is tempting to believe that ctime originally meant creation time;[13] however, while early Unix did have modification and creation times, the latter was changed to be access time before there was any C structure in which to call anything ctime. The file systems retained just the access time (atime) and modification time (mtime) through 6th edition Unix. The ctime timestamp was added in the file system restructuring that occurred with Version 7 Unix, and has always referred to inode change time. It is updated any time file metadata stored in the inode changes, such as file permissions, file ownership, and creation and deletion of hard links. POSIX also mandates ctime (last status change) update with nonzero write() (file modification).[14] In some implementations, ctime is affected by renaming a file, despite filenames not being stored in inodes: Both original Unix, which implemented a renaming by making a link (updating ctime) and then unlinking the old name (updating ctime again) and modern Linux tend to do this.

Unlike atime and mtime, ctime cannot be set to an arbitrary value with utime(), as used by the touch utility, for example. Instead, when utime() is used, or for any other change to the inode other than an update to atime caused by accessing the file, the ctime value is set to the current time.

I do not believe anyone could want the time they renamed an image or moved it into another directory to be used as the canonical "creation time" of the picture. mtime is the sane fallback.

[fyfrey]

Very valuable find of the actual ctime meaning.
However, I'm not sure Android actually uses file ctime. It has a creation time in the database that we are using in the app. That database should be the creation of the database entry.
What the server does to a file without EXIF is a different story. Best: it should take the information from app that is send during upload.

[Atemu]

However, I'm not sure Android actually uses file ctime. It has a creation time in the database that we are using in the app. That database should be the creation of the database entry.

I'm not sure I follow. Where is this database located, who controls it and where does its data come from?

The source of truth for file metadata is always the filesystem. However that info may be pulled through the layers of abstraction, we want the mtime/modification time associated with the files in the filesystem, not any other time.

Some modern filesystems implement a "birth time" metadata attribute but even that probably shouldn't be used though as a file's birth isn't preserved when copying etc. which are actions that should not change a picture's "creation time".

mtime says when a file's contents were last modified and is commonly preserved when copying or can be configured to be preserved. It's intended to be controllable by users. ctime and birth time are filesystem-internal and cannot be modified.

What the server does to a file without EXIF is a different story. Best: it should take the information from app that is send during upload.

The server part actually works fine. The bug only occurs when you upload these pictures using the app.

If you upload through the web UI, the dates are correct and, from that point on, it works flawlessly with the app too.

The problem is that the app sends the wrong info during upload.

[fyfrey]

I'm not sure I follow. Where is this database located, who controls it and where does its data come from?

Android has a internal database for it's media assets. The photo manager library we use, queries that database and we use the returned values. We can probably use their modified time, but this will possibly break other user's data.

The server part actually works fine. The bug only occurs when you upload these pictures using the app.
If you upload through the web UI, the dates are correct and, from that point on, it works flawlessly with the app too.
The problem is that the app sends the wrong info during upload.

Very interesting. So in your instance using file mtime (from the webbrowser) works fine. We should probably try to align the behavior of uploads of the same file from the web/app. We'll need to discuss internally what is the most robust, typically best way for assets without EXIF.