[BUG] iOS app - Broken upload when multiple photos have same name and timestamp.

[xick]

The bug

IOS App Version: tryied 1.77.0 and 1.78.0
Server: both 1.77.0 and 1.78.0
Hi Everyone, Today I tryed 3 times to upload my phone library using IOS App, everything went fine except I noticed that a consistent number of photos didn't upload succesfully. The broken photos shows a grey thumbnails in the web client, and can't not be downloaded (it will report a immich server fail).
By Investigating further I noticed that this behavior happens when photos had same timestamps (shot at the same minute) and they have been modified, resulting in a file named 'FullSizeRender.jpg' (this is a common behaviour for ios but immich should rename these files before sending them to the server I think).
This most likely happens when files in the upload queue presents same timestamp and name. As result only one shot in this queue is preserved ( random? Or Maybe will pick the first in order of elaboration)

An easy way to replicate this is by taking 2 pairs of shots taken at the same minute, and apply any modification in ios gallery (like a crop). Now open Immich app and trigger a backup. Browsing the gallery from the web will show weird behaviours , like photos taking place of others, grey thumbnails, incorrect metadata, impossibility to download or mismatch.** .
From IOS app at first glance it seems the uploads went fine, but it's just displaying the files on local

The OS that Immich Server is running on

Turnkey core inside Proxmox

Version of Immich Server

v1.78.0

Version of Immich Mobile App

v1.78.0

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - stack.env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - stack.env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      # remove this to get debug messages
      - GLOG_minloglevel=1
    volumes:
      - tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - stack.env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    depends_on:
      - immich-server
      - immich-web
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata:

Your .env content

UPLOAD_LOCATION=/mnt/data/docker/immich
IMMICH_VERSION=release
TYPESENSE_API_KEY=**
DB_PASSWORD=**
DB_HOSTNAME=**
DB_USERNAME=**
DB_DATABASE_NAME=**
REDIS_HOSTNAME=**

Reproduction steps

An easy way to **replicate** this is by taking 2 pairs of shots taken at the same minute, and apply any modification in ios gallery (like a crop). Now open Immich app and trigger a backup. **Weird behaviours could happen, like photos taking place of others, grey thumbnails, incorrect metadata, impossibility to download or mismatch.** .

Additional information

No response

[etnoy]

I have also seen this, can confirm

[VenomousViper27]

I've seen this as well on iOS, I've noticed that the problem file always seemed to be named FullSizeRender.jpg but wasn't sure what iOS was doing to name the file that way.

[xick]

It's very annoying, for now I'll keep backup from app disabled entirely.
Btw It may be an issue of the flutter_photo_manager library used. #fluttercandies/flutter_photo_manager#976

[xick]

I submitted a pull request to flutter_photo_manager, it’s waiting for reviews. Can confirm it addresses this issue

[awonglk]

Can confirm this is an issue for me.
Hopefully the fix will address any files with the same name, not just "FullSizeRender".
And a way for us to retroactively fix this as well once fix is in place.

I have a LOT of missing "FullSizeRender" photos that have not been backed up from my iPhone because of this bug.

[xick]

Can confirm this is an issue for me.

Hopefully the fix will address any files with the same name, not just "FullSizeRender".

And a way for us to retroactively fix this as well once fix is in place.

I have a LOT of missing "FullSizeRender" photos that have not been backed up from my iPhone because of this bug.

If a file has the same name immich it will check the timestamp, so in theory situations where media have same timestamps and name are very unlikely. Maybe we should check what happens when trying to upload media with same filenames but without timestamps.

[maslanypotwor]

I saw that the PR on the flutter GitHub was merged!
Does Immich needs to bump flutter version (once they release new one) or there is some additional work to be done?

PS. Thanks for tackling this!

[xick]

I saw that the PR on the flutter GitHub was merged!
Does Immich needs to bump flutter version (once they release new one) or there is some additional work to be done?

PS. Thanks for tackling this!

Immich auto fetch all last flutter libraries releases on pub.dev when a new version is out, so no additional work should be required.
In the next days I'll check what can be done with previously uploaded assets affected

[alextran1502]

@xick Thank you for your work on this.

We have to wait for a new release from photo_manager library to resolve this issue.

[ftruzzi]

Hi, and thanks for fixing this!

When the a new version of the Immich app is released, will it be enough to delete all FullSizeRender.jpg and FullSizeRender.mov files from the server so that the app can upload the original files again?

[awonglk]

I would like clarifications to this as well. The instruction on the* v1.82 release note* is: *You can fix the older assets by removing them from your immich server and reuploading the files, and they will have the original name.* Is it a matter of simply: 1. Use the metadata search feature in Immich with "m" keyword to search for FullSizeRender, i.e., m: FullSizeRender Above should locate all the photos of that name 2. Delete the files found above in Immich GUI I believe this should cleanly delete both the physical file and the metadata associated with it. 3, Run "Sync" from mobile app. This should pickup the 'missing' photos and back them up with original name this time. Is that correct?

…

On Mon, Oct 16, 2023 at 10:54 PM Francesco Truzzi ***@***.***> wrote: Hi, and thanks for fixing this! When the a new version of the Immich app is released, will it be enough to delete all FullSizeRender.jpg and FullSizeRender.mov files from the server so that the app can upload the original files again? — Reply to this email directly, view it on GitHub <#4088 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5EPFCPEU4Q5MJWVKHPWG3X7VDA3AVCNFSM6AAAAAA4XGLKECVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRUGY3DCOJUGE> . You are receiving this because you commented.Message ID: ***@***.***>

[xick]

I would like clarifications to this as well. The instruction on the* v1.82 release note* is: You can fix the older assets by removing them from your immich server and reuploading the files, and they will have the original name. Is it a matter of simply: 1. Use the metadata search feature in Immich with "m" keyword to search for FullSizeRender, i.e., m: FullSizeRender Above should locate all the photos of that name 2. Delete the files found above in Immich GUI I believe this should cleanly delete both the physical file and the metadata associated with it. 3, Run "Sync" from mobile app. This should pickup the 'missing' photos and back them up with original name this time. Is that correct?
…
On Mon, Oct 16, 2023 at 10:54 PM Francesco Truzzi @.> wrote: Hi, and thanks for fixing this! When the a new version of the Immich app is released, will it be enough to delete all FullSizeRender.jpg and FullSizeRender.mov files from the server so that the app can upload the original files again? — Reply to this email directly, view it on GitHub <#4088 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5EPFCPEU4Q5MJWVKHPWG3X7VDA3AVCNFSM6AAAAAA4XGLKECVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRUGY3DCOJUGE . You are receiving this because you commented.Message ID: @.>

Just tested, this is the way to go. You also have to delete items from Trash before sync again.
Unfortunatly I miss the chance to reproduce the corruption file bug again. When retrying I was in 1.79.1 so maybe the related issue was already resolved in server by this release.
If someone has lost original files and they're corrupted, could try to reupload them by looking in the 'upload' folder, they're should be there. Can someone confirm?