Error loading images (No thumbnail generation after April)

[etokheim]

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

Related issues: #12736 #18523 #18965

Any help is greatly appreciated! (And thanks for working on this awesome project!) 😃

---

I started getting these in April, and patiently waited for an update to fix it. However its still an issue, even after migrating to vectorchord and updating immich to the latest version (1.135.3).

How or why the issue started is hard to say. This has happened consistently with all images after April, 29th, but there's also a few littered around the timeline (altough very few, so that could be a separate issue).

Version History
Installed 1.135.3 on 18. juli 2025
Installed 1.134.0 on 18. juli 2025
Installed 1.133.1 on 31. mai 2025
Installed 1.132.3 on 1. mai 2025
Installed 1.131.3 on 13. apr. 2025

As you can see above, I didn't upgrade Immich around April 29th.

Clues

1. Live photos plays when hovering over the play button

   • 

2. I can download the image and both the live photo and the photo are downloaded and works fine (so it's probably an issue with thumbnail generation)

3. Videos plays fine, but have no thumbnail preview (instead shows the now familiar Error loading image)

4. Clicking `Refresh thumbnails` from the overflow menu (`...`) doesn't do anything (apart from sending a `POST` to the server. Can't see anything else in the logs at least). Click to see related log entries:

     [Nest] 17  - 07/19/2025, 11:04:03 AM   DEBUG [Api:LoggingInterceptor~iuufhkle] GET /api/assets/11bafb53-83e3-4107-9ee8-e4d8cf0529f6 200 4.06ms 10.71.128.64
     [Nest] 17  - 07/19/2025, 11:04:04 AM   DEBUG [Api:LoggingInterceptor~uo1c7ol2] GET /api/assets/11bafb53-83e3-4107-9ee8-e4d8cf0529f6/thumbnail?size=preview 200 2.60ms 10.71.128.64
     [Nest] 17  - 07/19/2025, 11:04:04 AM   DEBUG [Api:GlobalExceptionFilter~uo1c7ol2] HttpException(404): {"message":"Asset media not found","error":"Not Found","statusCode":404}
     [Nest] 17  - 07/19/2025, 11:04:04 AM   DEBUG [Api:LoggingInterceptor~du3a7e0q] GET /api/albums?assetId=11bafb53-83e3-4107-9ee8-e4d8cf0529f6 200 12.82ms 10.71.128.64
     [Nest] 17  - 07/19/2025, 11:04:04 AM   DEBUG [Api:LoggingInterceptor~pspr2x8h] GET /api/albums?assetId=11bafb53-83e3-4107-9ee8-e4d8cf0529f6 200 10.34ms 10.71.128.64
     [Nest] 17  - 07/19/2025, 11:04:04 AM   DEBUG [Api:LoggingInterceptor~rmhmf39c] GET /api/server/ping 200 0.27ms ::ffff:10.0.0.10
     [Nest] 17  - 07/19/2025, 11:04:04 AM   DEBUG [Api:LoggingInterceptor~gzu9s4vt] GET /api/server/ping 200 0.17ms ::ffff:10.0.0.10
     [Nest] 17  - 07/19/2025, 11:04:08 AM   DEBUG [Api:LoggingInterceptor~j10kpsof] POST /api/assets/jobs 204 2.70ms 10.71.128.64
     [Nest] 17  - 07/19/2025, 11:04:14 AM   DEBUG [Api:LoggingInterceptor~b7ht5gld] GET /api/server/ping 200 0.20ms ::ffff:10.0.0.10
     [Nest] 17  - 07/19/2025, 11:04:14 AM   DEBUG [Api:LoggingInterceptor~ggge33v9] GET /api/server/ping 200 0.13ms ::ffff:10.0.0.10

5. Issues with machine learning requests

   Don't know if it's related, but as you can see, I have quite a few of these Machine learning request to "http://sande-immich-server-machine-learning:3003" failed: fetch failed.

   I can curl the url with no problems from within the immich server container:

   root@sande-immich-server-54447d8658-lbspz:/usr/src/app# curl http://sande-immich-server-machine-learning:3003
   {"message":"Immich ML"}

   It doesn't quite make sense to me that there is an issue with the machine learning, as it does successfully detect faces. Isn't that machine learning based?: 1 faces detected in upload/thumbs/5355fc70-db04-4fb6-b336-cbe03e7fe794/a4/ac/a4ac8410-985d-40fe-83c4-f7ff07cf5cb2-preview.jpeg

   [Nest] 7  - 07/19/2025, 9:45:51 AM     LOG [Microservices:LibraryService] Checking 78956 asset(s) against import paths and exclusion patterns in library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:53 AM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://sande-immich-server-machine-learning:3003" failed: fetch failed
   [Nest] 7  - 07/19/2025, 9:45:53 AM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://sande-immich-server-machine-learning:3003" failed: fetch failed
   [Nest] 7  - 07/19/2025, 9:45:53 AM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://sande-immich-server-machine-learning:3003" failed: fetch failed
   [Nest] 7  - 07/19/2025, 9:45:53 AM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://sande-immich-server-machine-learning:3003" failed: fetch failed
   [Nest] 17  - 07/19/2025, 9:45:54 AM   DEBUG [Api:LoggingInterceptor~mt66u28k] GET /api/server/ping 200 0.66ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:45:54 AM   DEBUG [Api:LoggingInterceptor~lm9p1y3g] GET /api/server/ping 200 0.79ms ::ffff:10.0.0.10
   [Nest] 7  - 07/19/2025, 9:45:54 AM   DEBUG [Microservices:VersionService] Running version check
   [Nest] 7  - 07/19/2025, 9:45:55 AM   DEBUG [Microservices:PersonService] 0 faces detected in upload/thumbs/5355fc70-db04-4fb6-b336-cbe03e7fe794/c1/eb/c1eb08de-004a-4208-99f4-6a475e49199f-preview.jpeg
   [Nest] 7  - 07/19/2025, 9:45:55 AM   DEBUG [Microservices:PersonService] 0 faces detected in upload/thumbs/5355fc70-db04-4fb6-b336-cbe03e7fe794/79/46/7946c92e-f75d-42ef-a8d6-23cace40956a-preview.jpeg
   [Nest] 7  - 07/19/2025, 9:45:56 AM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://sande-immich-server-machine-learning:3003" failed: fetch failed
   [Nest] 7  - 07/19/2025, 9:45:57 AM   DEBUG [Microservices:PersonService] 0 faces detected in upload/thumbs/5355fc70-db04-4fb6-b336-cbe03e7fe794/f0/3b/f03b6787-bb98-48fb-91fd-0d7f28bc4530-preview.jpeg
   [Nest] 7  - 07/19/2025, 9:45:57 AM   DEBUG [Microservices:PersonService] 0 faces detected in upload/thumbs/5355fc70-db04-4fb6-b336-cbe03e7fe794/58/b1/58b1f103-aa8f-4002-8122-c581f2393823-preview.jpeg
   [Nest] 7  - 07/19/2025, 9:45:58 AM   DEBUG [Microservices:PersonService] 0 faces detected in upload/thumbs/5355fc70-db04-4fb6-b336-cbe03e7fe794/f9/a0/f9a045bd-caa8-4851-8bf2-50c5a5e6f457-preview.jpeg
   [Nest] 7  - 07/19/2025, 9:45:58 AM   DEBUG [Microservices:PersonService] 0 faces detected in upload/thumbs/5355fc70-db04-4fb6-b336-cbe03e7fe794/14/b2/14b287a1-5afc-46ab-a000-69682bd4b8cc-preview.jpeg
   [Nest] 7  - 07/19/2025, 9:45:58 AM   DEBUG [Microservices:PersonService] 1 faces detected in upload/thumbs/5355fc70-db04-4fb6-b336-cbe03e7fe794/a4/ac/a4ac8410-985d-40fe-83c4-f7ff07cf5cb2-preview.jpeg

6. Log output after starting the server:

   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   Error: connect ETIMEDOUT
       at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
       at Object.onceWrapper (node:events:632:28)
       at Socket.emit (node:events:518:28)
       at Socket._onTimeout (node:net:609:8)
       at listOnTimeout (node:internal/timers:594:17)
       at process.processTimers (node:internal/timers:529:7) {
     errorno: 'ETIMEDOUT',
     code: 'ETIMEDOUT',
     syscall: 'connect'
   }
   [Nest] 17  - 07/19/2025, 9:45:37 AM     LOG [Api:EventRepository] Websocket Connect:    VNManMOdGjgQm4RpAAAB
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:MapRepository] Initializing metadata repository
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:MetadataService] Initialized local reverse geocoder
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:ServerService] Feature Flags: {
     "smartSearch": true,
     "facialRecognition": true,
     "duplicateDetection": true,
     "map": true,
     "reverseGeocoding": true,
     "importFaces": false,
     "sidecar": true,
     "search": true,
     "trash": true,
     "oauth": false,
     "oauthAutoLaunch": false,
     "passwordLogin": true,
     "configFile": false,
     "email": true
   }
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:StorageService] Verifying system mount folder checks, current state: {"mountFiles":true,"mountChecks":{"thumbs":true,"upload":true,"backups":true,"library":true,"profile":true,"encoded-video":true}}
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:StorageService] Successfully verified system mount folder checks
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:SystemConfigService] LogLevel=debug (set via system config)
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:NestFactory] Starting Nest application...
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:InstanceLoader] BullModule dependencies initialized
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:InstanceLoader] ClsModule dependencies initialized
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:InstanceLoader] ClsCommonModule dependencies initialized
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:InstanceLoader] OpenTelemetryModule dependencies initialized
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:InstanceLoader] KyselyModule dependencies initialized
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:InstanceLoader] KyselyCoreModule dependencies initialized
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:InstanceLoader] DiscoveryModule dependencies initialized
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:InstanceLoader] OpenTelemetryCoreModule dependencies initialized
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:InstanceLoader] ClsRootModule dependencies initialized
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:InstanceLoader] BullModule dependencies initialized
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:InstanceLoader] BullModule dependencies initialized
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:InstanceLoader] MicroservicesModule dependencies initialized
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:ClsModule] Mounting ClsMiddleware to /
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:NestApplication] Nest application successfully started
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:Bootstrap] Immich Microservices is running [v1.135.3] [production]
   [Nest] 7  - 07/19/2025, 9:45:38 AM     LOG [Microservices:SessionService] Deleted 0 expired session tokens
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:VersionService] Running version check
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:VersionService] Running version check
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:VersionService] Running version check
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:VersionService] Running version check
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:BackupService] Database Backup Started
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:VersionService] Running version check
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:38 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM   DEBUG [Microservices:LibraryService] Validating import paths for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM   DEBUG [Microservices:LibraryService] Validating import paths for library 82be8a48-dbf3-494f-bded-2ec80e41d50d...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:39 AM     LOG [Microservices:LibraryService] Starting disk crawl of 1 import path(s) for library 82be8a48-dbf3-494f-bded-2ec80e41d50d...
   [Nest] 17  - 07/19/2025, 9:45:39 AM     LOG [Api:EventRepository] Websocket Connect:    u1sYZTxE0mjWxStDAAAD
   [Nest] 7  - 07/19/2025, 9:45:41 AM     LOG [Microservices:BackupService] Database Backup Starting. Database Version: 16
   [Nest] 7  - 07/19/2025, 9:45:41 AM   DEBUG [Microservices:VersionService] Running version check
   [Nest] 7  - 07/19/2025, 9:45:41 AM   DEBUG [Microservices:VersionService] Running version check
   [Nest] 7  - 07/19/2025, 9:45:41 AM   DEBUG [Microservices:VersionService] Running version check
   [Nest] 7  - 07/19/2025, 9:45:41 AM   DEBUG [Microservices:VersionService] Running version check
   [Nest] 7  - 07/19/2025, 9:45:42 AM     LOG [Microservices:LibraryService] Crawled 994 file(s) so far: 0 of current batch of 994 will be imported to library 82be8a48-dbf3-494f-bded-2ec80e41d50d...
   [Nest] 7  - 07/19/2025, 9:45:42 AM     LOG [Microservices:LibraryService] Finished disk crawl, 994 file(s) found on disk and queued 0 file(s) for import into 82be8a48-dbf3-494f-bded-2ec80e41d50d
   [Nest] 17  - 07/19/2025, 9:45:44 AM   DEBUG [Api:LoggingInterceptor~u3jk5m7d] GET /api/server/ping 200 44.17ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:45:44 AM   DEBUG [Api:LoggingInterceptor~y95s0w46] GET /api/server/ping 200 0.53ms ::ffff:10.0.0.10
   [Nest] 7  - 07/19/2025, 9:45:51 AM     LOG [Microservices:LibraryService] Checking 78956 asset(s) against import paths and exclusion patterns in library cd90d414-b016-4e44-b85c-a7403ea1a120...
   [Nest] 7  - 07/19/2025, 9:45:53 AM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://sande-immich-server-machine-learning:3003" failed: fetch failed
   [Nest] 7  - 07/19/2025, 9:45:53 AM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://sande-immich-server-machine-learning:3003" failed: fetch failed
   [Nest] 7  - 07/19/2025, 9:45:53 AM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://sande-immich-server-machine-learning:3003" failed: fetch failed
   [Nest] 7  - 07/19/2025, 9:45:53 AM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://sande-immich-server-machine-learning:3003" failed: fetch failed
   [Nest] 17  - 07/19/2025, 9:45:54 AM   DEBUG [Api:LoggingInterceptor~mt66u28k] GET /api/server/ping 200 0.66ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:45:54 AM   DEBUG [Api:LoggingInterceptor~lm9p1y3g] GET /api/server/ping 200 0.79ms ::ffff:10.0.0.10
   [Nest] 7  - 07/19/2025, 9:45:54 AM   DEBUG [Microservices:VersionService] Running version check
   [Nest] 7  - 07/19/2025, 9:45:55 AM   DEBUG [Microservices:PersonService] 0 faces detected in upload/thumbs/5355fc70-db04-4fb6-b336-cbe03e7fe794/c1/eb/c1eb08de-004a-4208-99f4-6a475e49199f-preview.jpeg
   [Nest] 7  - 07/19/2025, 9:45:55 AM   DEBUG [Microservices:PersonService] 0 faces detected in upload/thumbs/5355fc70-db04-4fb6-b336-cbe03e7fe794/79/46/7946c92e-f75d-42ef-a8d6-23cace40956a-preview.jpeg
   [Nest] 7  - 07/19/2025, 9:45:56 AM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://sande-immich-server-machine-learning:3003" failed: fetch failed
   [Nest] 7  - 07/19/2025, 9:45:57 AM   DEBUG [Microservices:PersonService] 0 faces detected in upload/thumbs/5355fc70-db04-4fb6-b336-cbe03e7fe794/f0/3b/f03b6787-bb98-48fb-91fd-0d7f28bc4530-preview.jpeg
   [Nest] 7  - 07/19/2025, 9:45:57 AM   DEBUG [Microservices:PersonService] 0 faces detected in upload/thumbs/5355fc70-db04-4fb6-b336-cbe03e7fe794/58/b1/58b1f103-aa8f-4002-8122-c581f2393823-preview.jpeg
   [Nest] 7  - 07/19/2025, 9:45:58 AM   DEBUG [Microservices:PersonService] 0 faces detected in upload/thumbs/5355fc70-db04-4fb6-b336-cbe03e7fe794/f9/a0/f9a045bd-caa8-4851-8bf2-50c5a5e6f457-preview.jpeg
   [Nest] 7  - 07/19/2025, 9:45:58 AM   DEBUG [Microservices:PersonService] 0 faces detected in upload/thumbs/5355fc70-db04-4fb6-b336-cbe03e7fe794/14/b2/14b287a1-5afc-46ab-a000-69682bd4b8cc-preview.jpeg
   [Nest] 7  - 07/19/2025, 9:45:58 AM   DEBUG [Microservices:PersonService] 1 faces detected in upload/thumbs/5355fc70-db04-4fb6-b336-cbe03e7fe794/a4/ac/a4ac8410-985d-40fe-83c4-f7ff07cf5cb2-preview.jpeg
   [Nest] 17  - 07/19/2025, 9:46:04 AM   DEBUG [Api:LoggingInterceptor~yuq2wefu] GET /api/server/ping 200 0.32ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:46:04 AM   DEBUG [Api:LoggingInterceptor~tobc6aug] GET /api/server/ping 200 0.34ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:46:14 AM   DEBUG [Api:LoggingInterceptor~7qdlc158] GET /api/server/ping 200 0.49ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:46:14 AM   DEBUG [Api:LoggingInterceptor~nptp4yy1] GET /api/server/ping 200 0.31ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:46:24 AM   DEBUG [Api:LoggingInterceptor~39zxy8so] GET /api/server/ping 200 0.29ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:46:24 AM   DEBUG [Api:LoggingInterceptor~3q3jx9xh] GET /api/server/ping 200 0.27ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:46:34 AM   DEBUG [Api:LoggingInterceptor~f051qgz6] GET /api/server/ping 200 0.28ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:46:34 AM   DEBUG [Api:LoggingInterceptor~vwp35wjg] GET /api/server/ping 200 0.39ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:46:44 AM   DEBUG [Api:LoggingInterceptor~6jhabtzi] GET /api/server/ping 200 0.28ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:46:44 AM   DEBUG [Api:LoggingInterceptor~x16vazvh] GET /api/server/ping 200 0.26ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:46:54 AM   DEBUG [Api:LoggingInterceptor~nyn37exc] GET /api/server/ping 200 0.25ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:46:54 AM   DEBUG [Api:LoggingInterceptor~z2n940uj] GET /api/server/ping 200 0.32ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:47:04 AM   DEBUG [Api:LoggingInterceptor~eb47sj8i] GET /api/server/ping 200 0.25ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:47:04 AM   DEBUG [Api:LoggingInterceptor~fqq7t27b] GET /api/server/ping 200 0.26ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:47:14 AM   DEBUG [Api:LoggingInterceptor~ye17o0wt] GET /api/server/ping 200 0.24ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:47:14 AM   DEBUG [Api:LoggingInterceptor~jg51ejoi] GET /api/server/ping 200 0.26ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:47:24 AM   DEBUG [Api:LoggingInterceptor~ioq2yjyu] GET /api/server/ping 200 0.30ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:47:24 AM   DEBUG [Api:LoggingInterceptor~tk3i38aa] GET /api/server/ping 200 0.26ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:47:34 AM   DEBUG [Api:LoggingInterceptor~tdbon7hu] GET /api/server/ping 200 0.21ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:47:34 AM   DEBUG [Api:LoggingInterceptor~m1wwtbmv] GET /api/server/ping 200 0.22ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:47:44 AM   DEBUG [Api:LoggingInterceptor~oeeja1o9] GET /api/server/ping 200 0.39ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:47:44 AM   DEBUG [Api:LoggingInterceptor~drxhfj2o] GET /api/server/ping 200 0.33ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:47:54 AM   DEBUG [Api:LoggingInterceptor~4fnzx4bc] GET /api/server/ping 200 0.25ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:47:54 AM   DEBUG [Api:LoggingInterceptor~qvnoorz1] GET /api/server/ping 200 0.29ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:48:04 AM   DEBUG [Api:LoggingInterceptor~dq30j12a] GET /api/server/ping 200 0.28ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:48:04 AM   DEBUG [Api:LoggingInterceptor~wlw0iv1j] GET /api/server/ping 200 0.23ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:48:14 AM   DEBUG [Api:LoggingInterceptor~arxnvehj] GET /api/server/ping 200 0.21ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:48:14 AM   DEBUG [Api:LoggingInterceptor~m1p3bm38] GET /api/server/ping 200 0.25ms ::ffff:10.0.0.10
   [Nest] 7  - 07/19/2025, 9:48:18 AM     LOG [Microservices:BackupService] Database Backup Success
   [Nest] 7  - 07/19/2025, 9:48:18 AM   DEBUG [Microservices:BackupService] Database Backup Cleanup Started
   [Nest] 17  - 07/19/2025, 9:48:24 AM   DEBUG [Api:LoggingInterceptor~li7k1mbs] GET /api/server/ping 200 0.26ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:48:24 AM   DEBUG [Api:LoggingInterceptor~wtvls6ai] GET /api/server/ping 200 0.21ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:48:34 AM   DEBUG [Api:LoggingInterceptor~9loomgts] GET /api/server/ping 200 0.17ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:48:34 AM   DEBUG [Api:LoggingInterceptor~zefdvonw] GET /api/server/ping 200 0.17ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:48:44 AM   DEBUG [Api:LoggingInterceptor~6hxdcizj] GET /api/server/ping 200 0.25ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:48:44 AM   DEBUG [Api:LoggingInterceptor~97u62fmg] GET /api/server/ping 200 0.17ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:48:54 AM   DEBUG [Api:LoggingInterceptor~g4rrnl9h] GET /api/server/ping 200 0.19ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:48:54 AM   DEBUG [Api:LoggingInterceptor~l9yww8on] GET /api/server/ping 200 0.18ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:49:04 AM   DEBUG [Api:LoggingInterceptor~zqy3uveh] GET /api/server/ping 200 0.26ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:49:04 AM   DEBUG [Api:LoggingInterceptor~gxg4l7us] GET /api/server/ping 200 0.21ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:49:14 AM   DEBUG [Api:LoggingInterceptor~l0rekelk] GET /api/server/ping 200 0.25ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:49:14 AM   DEBUG [Api:LoggingInterceptor~o0nt5nq2] GET /api/server/ping 200 0.25ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:49:21 AM   DEBUG [Api:LoggingInterceptor~nn6nygia] GET /api/timeline/bucket?timeBucket=2011-07-01T00%3A00%3A00.000Z&visibility=timeline&withPartners=true&withStacked=true 200 2024.67ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~25v9urst] GET /api/assets/5b670b0a-5b18-4fba-aebb-8460cd09eec6/thumbnail?size=thumbnail&c=X%2BgNDQJ2eI%2BY2Hh4iIdXR7WgnEMF 200 569.39ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~rzt9w99o] GET /api/assets/99467961-20a1-4b51-974a-832d9586b9e0/thumbnail?size=thumbnail&c=U%2FgRDYB2anh%2Fl4d7eIeIh%2FzWZDDF 200 581.08ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~trna1n2m] GET /api/assets/7249f19c-3d99-4c14-bdec-9f5a6e9f4c46/thumbnail?size=thumbnail&c=HdkJFYKweohLiHd5eHiXiLbA6dj3 200 571.93ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~1dafypm7] GET /api/assets/72096930-293a-4235-94ab-92acf8b4d3dc/thumbnail?size=thumbnail&c=YOgNDQJ2eI%2BY13h4h3ZHN6WwrTMF 200 563.38ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~voufue2z] GET /api/assets/75f637f8-7339-4b40-8af6-7e60ec77645f/thumbnail?size=thumbnail&c=XekJDYJ3qIaPh3h2iHd3h%2FmMX%2FpY 200 650.98ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~7kvqi5uq] GET /api/assets/904df037-8504-4431-8d21-0f615c7132a4/thumbnail?size=thumbnail&c=m9kFDYSImJaPh3eYh3eGd9%2Bpbvpm 200 644.02ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~jzdfndic] GET /api/assets/381ef0a9-8d00-401e-a8a0-13542d1ce475/thumbnail?size=thumbnail&c=pRgWHYaQaGmCeYeYd3h4mI959Jl3 200 18.51ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~7cxoqwis] GET /api/assets/c6c1966a-7cbc-467b-85cd-88c605b4d636/thumbnail?size=thumbnail&c=mtkFDYJPi3eNlYp3d5iYeY8mXAc6 200 58.17ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~1v06u898] GET /api/assets/b8dde53c-68a4-4efc-9408-5ca1bcb31140/thumbnail?size=thumbnail&c=aDkGJQTPIn6HGntZh2Nmac886fuQ 200 98.24ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~nslbmcej] GET /api/assets/a7fbd876-4489-4635-8a80-03c89c713538/thumbnail?size=thumbnail&c=mskFFYAhiokIsnlYmLepipgQ5ZDu 200 159.26ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~h6zzmjyg] GET /api/assets/9a5f7587-73c2-44cc-a5b9-4979e8d2cff0/thumbnail?size=thumbnail&c=9CgKHYYtd2iAd4hlh1Z3dsCMCc2W 200 55.75ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~l3kfagak] GET /api/assets/0214ba49-8daf-4a24-b0d3-fc3d69e3f22b/thumbnail?size=thumbnail&c=9hgKFYIKWXh2WIeQiHiWlvZcXM%2B1 200 106.22ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~h74i3a13] GET /api/assets/692e1f0d-4ebc-4842-bf5f-8252c59e4045/thumbnail?size=thumbnail&c=MCkGJYRri4VzVpdgqtdadqKvKPua 200 72.27ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~gh1y4qur] GET /api/assets/419d4d58-5c44-40c5-9dc8-cf0e37a15da6/thumbnail?size=thumbnail&c=IDkGFYJPanJkL5c4Wth0Zu8N%2Fcyi 200 84.54ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~o6rsit5t] GET /api/assets/685ef837-6a40-4726-9eaa-4de9a514ea53/thumbnail?size=thumbnail&c=LVkOJYyPdXaElneJZmmHdtJPC%2FyT 200 89.53ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~3h117k65] GET /api/assets/50749bd4-4965-4dad-880c-607b5c47ae04/thumbnail?size=thumbnail&c=WVkGFQZ%2FiZKhV2kmamuZhn2Q5gp2 200 117.53ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~5r0tpied] GET /api/assets/771a877f-6f00-45f2-bb03-306af2aeaade/thumbnail?size=thumbnail&c=YzkOHYRQaHh3ZniFiIeHiDaPqvG4 200 215.04ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~pmgrx0ug] GET /api/assets/f32f22a6-b9e6-44d1-8ba6-aaffe4698e32/thumbnail?size=thumbnail&c=X0kKFQIgeHhpp5moV5l4ePSabt%2F7 200 49.38ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~9ylauhmb] GET /api/assets/9314396f-d095-46f8-beba-2fe86bca3529/thumbnail?size=thumbnail&c=0SgGJYb7pXdWaXl6dHeaefVnVI9G 200 76.88ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~7pxg5t8q] GET /api/assets/2c6eb31e-2b42-4047-899e-56956664ed5f/thumbnail?size=thumbnail&c=mlkGFYRwR3uUeYuIm5WIes9e%2BMhG 200 86.34ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~045he6r8] GET /api/assets/ec945d56-e5d8-40f6-8d56-e3f2eaa4478e/thumbnail?size=thumbnail&c=YjkOHYSQa2mXlni5Z1eIlW98%2BuW4 200 46.37ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~iautninb] GET /api/assets/bb8ee71f-802e-4359-aaac-75bf2c9f4aad/thumbnail?size=thumbnail&c=mEkGFYJChMN%2FZUaGWziqgfO2en%2Bq 200 61.17ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~svm7iy6o] GET /api/assets/d3f46576-e40b-4813-ac38-6fa4fdf4e806/thumbnail?size=thumbnail&c=oUkKFYKvWEeZm3kymrlld7ggnwny 200 96.76ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~3w253a5a] GET /api/assets/ebc3bb66-ccaa-4972-b43a-8b51329d0385/thumbnail?size=thumbnail&c=VkkGFYIHiMqYu4qom4l4iWNVD0jz 200 60.91ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~f3fekw7o] GET /api/assets/cb0218df-e678-4028-a888-142289e86f07/thumbnail?size=thumbnail&c=W0kGHQRid0yICKx4jKV2d5ufyviq 200 141.92ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:23 AM   DEBUG [Api:LoggingInterceptor~m4ofp7fc] GET /api/assets/e3e72dca-6e2c-4e02-9b81-09dc4f8f0711/thumbnail?size=thumbnail&c=WkkKFYJWe5hPh4dXaGV4mD%2Bl%2FiT7 200 86.53ms 10.71.128.64
   [Nest] 17  - 07/19/2025, 9:49:24 AM   DEBUG [Api:LoggingInterceptor~sqys76k7] GET /api/server/ping 200 0.23ms ::ffff:10.0.0.10
   [Nest] 17  - 07/19/2025, 9:49:24 AM   DEBUG [Api:LoggingInterceptor~rxu26a9q] GET /api/server/ping 200 0.26ms ::ffff:10.0.0.10

7. Points of interest from the mobile app (TLDR: Thumbnails visible since it uses the local thumbnails. Turning on the `Prefer external images` makes the app look as broken as the web version):

   1. Don't know if it's related, but my backed up images (Sikkerhetskopier: 1189) has been higher than the total number of images on the device (Total: 1058) for a long time. Isn't that weird?
      • 
   2. From the mobile app:
      • 
   3. From the mobile app after turning on the advanced setting: Prefer external images. (The videos that does have previews are not backed up).
   4. 
   5. 
   6. 

The OS that Immich Server is running on

Ubuntu 24.04 -> Microk8s -> Helm

Version of Immich Server

v1.135.3

Version of Immich Mobile App

1.135.3

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content Helm values

Open

## This chart relies on the common library chart from bjw-s
## You can find it at https://github.com/bjw-s/helm-charts/tree/main/charts/library/common
## Refer there for more detail about the supported values

# These entries are shared between all the Immich components

env:
  DB_HOSTNAME: "immich-rw.cloudnative-pg"
  # -- You should provide your own secret outside of this helm-chart and use `postgresql.global.postgresql.auth.existingSecret` to provide credentials to the postgresql instance
  DB_PASSWORD: [very secret]
  #IMMICH_MACHINE_LEARNING_URL: '{{ printf "http://%s-machine-learning:3003" .Release.Name }}'

image:
  tag: v1.135.3

nodeSelector:
  location: sande
  kubernetes.io/hostname: node1

immich:
  metrics:
    # Enabling this will create the service monitors needed to monitor immich with the prometheus operator
    enabled: false
  persistence:
    # Main data store for all photos shared between different components.
    library:
      # Automatically creating the library volume is not supported by this chart
      # You have to specify an existing PVC to use
      existingClaim: immich-library

  # configuration is immich-config.json converted to yaml
  # ref: https://immich.app/docs/install/config-file/
  #
  configuration: {}
    # trash:
    #   enabled: false
    #   days: 30
    # storageTemplate:
    #   enabled: true
    #   template: "{{y}}/{{y}}-{{MM}}-{{dd}}/{{filename}}"

# Dependencies

postgresql:
  enabled: false

redis:
  enabled: true
  architecture: standalone
  auth:
    enabled: false
  master:
    nodeSelector:
      location: sande
      kubernetes.io/hostname: node1
    persistence:
      enabled: true
      #size: 11Gi
      # Optional: Set this to pvc to avoid downloading the ML models every start.
      type: pvc
      # storageClass: your-class
      mountPath: bitnami/postgresql
      existingClaim: immich-redis

# Immich components

server:
  enabled: true
  image:
    repository: ghcr.io/immich-app/immich-server
    pullPolicy: Always
  persistence:
    camera:
      enabled: true
      existingClaim: sande.immich.erling
      mountPath: /usr/src/app/camera
    anne:
      enabled: true
      existingClaim: sande.immich.anne
      mountPath: /usr/src/app/anne

machine-learning:
  enabled: true
  image:
    repository: ghcr.io/immich-app/immich-machine-learning
    #tag: release-cuda # Use Hardware-acceleration
    pullPolicy: Always
  env:
    TRANSFORMERS_CACHE: /cache
  persistence:
    cache:
      enabled: true
      # Optional: Set this to pvc to avoid downloading the ML models every start.
      type: pvc
      existingClaim: immich-machine-learning-cache
  resources:
    requests:
      # Hardware acceleration using an Intel iGPU w/ QuickSync and
      # using intel-gpu-plugin (https://github.com/intel/intel-device-plugins-for-kubernetes)
      # gpu.intel.com/i915: 1
      #nvidia.com/gpu: 1
      # cpu: 200m
      # memory: 256Mi
    limits:
      # Hardware acceleration using an Intel iGPU w/ QuickSync and
      # using intel-gpu-plugin (https://github.com/intel/intel-device-plugins-for-kubernetes)
      # gpu.intel.com/i915: 1
      #nvidia.com/gpu: 1
      # memory: 4096Mi

Other

Other

Your .env content

None

Reproduction steps

Don't know

Relevant log output

Additional information

No response

[seldmceen]

Following. Same issue but using docker in unraid

[HansAndreManfredson]

I'm experiencing the same problem on my instance. Version v1.135.3. I hope I'm not mixing up problems, but perhaps there's a dependency."

Error 1:
immich_server            | [Nest] 17  - 07/21/2025, 8:59:43 AM   DEBUG [Api:GlobalExceptionFilter~hc8h1fj8] HttpException(404): {"message":"Asset media not found","error":"Not Found","statusCode":404}

----------

Error 2: 

immich_server            | [Nest] 7  - 07/21/2025, 8:43:45 AM    WARN [Microservices:Error: File not found - /usr/src/app/upload/upload/2ac44631-32fb-439a-bdc2-16252a5377e6/fe/07/fe07d3ac-ed06-49d7-9038-214c0ab3e6c5.JPG.xmp
immich_server            |     at ReadTask._ExifToolTask_parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:76:71)
immich_server            |     at ReadTask.parser (/usr/src/app/node_modules/exiftool-vendored/dist/ExifToolTask.js:47:167)
immich_server            |     at ReadTask._Task_resolve (/usr/src/app/node_modules/batch-cluster/dist/Task.js:146:40)
immich_server            |     at runNextTicks (node:internal/process/task_queues:65:5)
immich_server            |     at listOnTimeout (node:internal/timers:555:9)
immich_server            |     at process.processTimers (node:internal/timers:529:7)] Error reading exif data (upload/upload/2ac44631-32fb-439a-bdc2-16252a5377e6/fe/07/fe07d3ac-ed06-49d7-9038-214c0ab3e6c5.JPG.xmp): Error: File not found - /usr/src/app/upload/upload/2ac44631-32fb-439a-bdc2-16252a5377e6/fe/07/fe07d3ac-ed06-49d7-9038-214c0ab3e6c5.JPG.xmp

[skatsubo]

Hey @etokheim, thanks for the detailed report.

1. Redis connection errors in logs:

Error: connect ETIMEDOUT
at Socket. (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
syscall: 'connect'

Is redis healthy and listening on port 6379? Do you have more details on redis state?

2. Redis definition in helm - is it correct? Asking just because of (somewhat unexpected for me):

redis:
...
mountPath: bitnami/postgresql

Redis is required for jobs to work, AFAIK, no redis -> no jobs -> no thumbnail generation.

[etokheim]

Umm, @skatsubo, I gotta say I'm puzzled too – probably found it somewhere. Or, maybe I messed it up when moving Postgres out of the helm values?

I don't have proper version control for the values, but the mountPath has been the same since at least August of last year – so I guess it at least has been working fine?

Doing a quick check of the helm manifest, I can see that Bitnami's Redis image is used (I guess it's this one). In the readme, they mention mapping /path/to/redis-persistence:/bitnami/redis/data in order to persist the database.

Which path do you map in order to persist the Redis database?

---

Regarding the timedout messages – I just assumed they were due to the server starting up before the redis server is ready. As you can see, they only show up in the beginning of the log. Should I look into it further? The Redis server seems to be running fine, at least.

Redis log (click to expand)

kc logs --follow sande-immich-server-redis-master-0
1:C 20 Jul 2025 20:19:06.719 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 20 Jul 2025 20:19:06.719 * Redis version=7.4.3, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 20 Jul 2025 20:19:06.719 * Configuration loaded
1:M 20 Jul 2025 20:19:06.719 * monotonic clock: POSIX clock_gettime
1:M 20 Jul 2025 20:19:06.720 * Running mode=standalone, port=6379.
1:M 20 Jul 2025 20:19:06.720 * Server initialized
1:M 20 Jul 2025 20:19:07.647 * Reading RDB base file on AOF loading...
1:M 20 Jul 2025 20:19:07.647 * Loading RDB produced by version 7.4.3
1:M 20 Jul 2025 20:19:07.647 * RDB age 80126 seconds
1:M 20 Jul 2025 20:19:07.647 * RDB memory usage when created 163.90 Mb
1:M 20 Jul 2025 20:19:07.647 * RDB is base AOF
1:M 20 Jul 2025 20:19:14.007 * Done loading RDB, keys loaded: 78896, keys expired: 0.
1:M 20 Jul 2025 20:19:14.007 * DB loaded from base file appendonly.aof.1209.base.rdb: 7.225 seconds
1:M 20 Jul 2025 20:19:23.171 * DB loaded from incr file appendonly.aof.1490.incr.aof: 9.163 seconds
1:M 20 Jul 2025 20:19:23.171 * DB loaded from append only file: 16.390 seconds
1:M 20 Jul 2025 20:19:23.173 * Opening AOF incr file appendonly.aof.1490.incr.aof on server start
1:M 20 Jul 2025 20:19:23.173 * Ready to accept connections tcp
1:M 21 Jul 2025 05:18:56.264 * Starting automatic rewriting of AOF on 100% growth
1:M 21 Jul 2025 05:18:56.366 * Creating AOF incr file appendonly.aof.1491.incr.aof on background rewrite
1:M 21 Jul 2025 05:18:56.371 * Background append only file rewriting started by pid 142432
142432:C 21 Jul 2025 05:19:13.721 * Successfully created the temporary AOF base file temp-rewriteaof-bg-142432.aof
142432:C 21 Jul 2025 05:19:13.723 * Fork CoW for AOF rewrite: current 3 MB, peak 3 MB, average 2 MB
1:M 21 Jul 2025 05:19:13.808 * Background AOF rewrite terminated with success
1:M 21 Jul 2025 05:19:13.808 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-142432.aof into appendonly.aof.1210.base.rdb
1:M 21 Jul 2025 05:19:13.965 * Removing the history file appendonly.aof.1490.incr.aof in the background
1:M 21 Jul 2025 05:19:13.965 * Removing the history file appendonly.aof.1209.base.rdb in the background
1:M 21 Jul 2025 05:19:14.042 * Background AOF rewrite finished successfully
1:M 21 Jul 2025 20:29:48.911 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 21 Jul 2025 20:30:12.028 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 21 Jul 2025 21:33:15.229 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.

Thanks by the way, @skatsubo! Appreciate any help! 😄

[skatsubo]

Regarding the timedout messages – I just assumed they were due to the server starting up before the redis server is ready. As you can see, they only show up in the beginning of the log.

That totally makes sense! I overlooked it. So, then Redis is a red herring.

What we can do to finally remove Redis from the equation:

• Backup its files somewhere
• Then purge/recreate its PVC / mount path

If that does not fix the issue after triggering jobs in Immich, then we at least know that Redis is not related to it. If that does help, we will have a backup to investigate further.

Good luck!

(Btw, I use plain docker / compose setup, no k8s.)

[bo0tzz]

Backup its files somewhere

nb the redis data is pretty unimportant, so it's not really necessary to back it up.

[skatsubo]

What else to check:

1. IMMICH_LOG_LEVEL=verbose instead of debug.

2. Thumbnail/preview records in the database.

select * from assets a 
join asset_files af on a.id = af."assetId" 
join asset_job_status ajs on a.id = ajs."assetId" 
where a."originalPath" ilike '%my_image.jpg';

3. Thumbnail/preview files on disk.

4. HTTP 404 in browser developer tools and/or server logs.

[5aji]

FWIW I encountered this recently and had to add --pids-limit=1048576 to my container to allow it to spawn more threads. The error I would see was libgomp: Thread creation failed: Resource temporarily unavailable. I think this is a separate issue though, but putting this here in case other folks find it. Once I did this I could actually finish the Generate Thumbnails job.

[skatsubo]

The error I would see was libgomp: Thread creation failed: Resource temporarily unavailable. I think this is a separate issue though

Thanks for sharing @5aji. I agree that libgomp threads is probably not the cause of this issue. Anyway, good to know.
Btw, I found one libgomp-related question on Discord: Crash after Thread creation failed.

[yilin101]

Exactly the same situation as you, this problem has been bothering me since the beginning of April, I tried to create a brand new immich with the same configuration, the same external pictures, and everything worked fine. But the old immich just doesn't work, and I'm trying to find a way to fix it because I don't want to reset the albums and characters! I've been waiting for months and still no solution, sad!

[skatsubo]

Hey folks, @etokheim @yilin101 and whoever has that issue!
Would be super nice if you can gather more info from your systems, see below. Let me know if it's too much (haha). Also if it feels more convenient we can troubleshoot it interactively on Immich Discord.

I suggest to enable extended monitoring, upload an image to trigger thumbnails generation, fetch related recent records from Postgres/Redis.

Troubleshooting steps

Outline

1. Enable verbose logging in Immich.
2. Enable activity watching in Redis.
3. Upload an image, view it in UI, refresh thumbnails from the three-dots overflow menu "..."
4. Save Immich and Redis logs from steps 1 and 2.
5. Fetch recent records from Postgres.
6. Fetch recent records from Redis.

Do this during quiet time when there is no other users/activity to reduce noise in logs.

Then share the logs and records output.

See below step-by-step instructions for docker compose setup.

• For Kubernetes should be adjusted accordingly.
• For Synology you need SSH access, I assume.

---

1. Enable verbose logging in Immich server.

Add to your .env file:

IMMICH_LOG_LEVEL=verbose

Apply:

docker compose up -d

2. Enable activity watching in Redis.

Watch all activity related to thumbnails (it includes constant refresh/noise from BullMQ):

docker exec -ti immich_redis redis-cli monitor | grep thumbnailGeneration

3. Upload an image and refresh its thumbnails

• Upload an image through web.
• Wait a bit (~10 sec).
• Open the image in UI.
• Refresh its thumbnails from the overflow menu ("...")

4. Save Immich and Redis logs from steps 1 and 2.

Redis:

• Stop the "redis-cli" command with Ctrl-C.
• Copy and save its output.

Immich:

• Run docker logs -t immich_server
• Copy and save its output.

5. Fetch recent records from Postgres.

Connect to the database:

docker exec -ti --user postgres immich_postgres  psql -Ppager=no --expanded -d immich

Query 1: recent records with missing thumbnailAt.

SELECT * FROM asset_job_status ajs
JOIN assets a ON a.id = ajs."assetId"
WHERE "thumbnailAt" IS NULL
ORDER BY "metadataExtractedAt" DESC LIMIT 4;

Query 2: recent records having thumbnailAt.

SELECT * FROM asset_job_status ajs
JOIN assets a ON a.id = ajs."assetId"
JOIN asset_files af ON a.id = af."assetId"
ORDER BY "thumbnailAt" DESC, "metadataExtractedAt" DESC LIMIT 4;

If you have your image file name, try also more specific query after substituting file name at the end:

select * from assets a 
left join asset_files af on a.id = af."assetId" 
left join asset_job_status ajs on a.id = ajs."assetId" 
where a."originalFileName" = 'MY_UPLOADED_IMAGE.JPG';

6. Fetch recent records from Redis.

Fetch last 30 events for the thumbnails job. (The command is quite long because it transforms Redis output into more readable form)

docker exec immich_redis redis-cli --raw XREVRANGE "immich_bull:thumbnailGeneration:events" + - COUNT 30 | \
  awk '/^[0-9]+-[0-9]+$/ { if (NR > 1) print output ; output = "ID=" $0 ; next }
  { if (!key) { key = $0 } else { output = output ", " key "=" $0 ; key = "" } }
  END { print output }'

Finally: post/share your findings

There will be 4 outputs:

• Immich logs from docker compose logs (step 1)
• Redis "monitor" (step 2)
• Postgres records (step 5)
• Redis records (step 6)

[rdelcorro]

I am in the same boat with a new deployment. I do see redis traffic and there is data in the DB but no thumb. Jobs are stuck in their queues. External libraries crawl and find the files but nothing happens after (can't even see the images without thumbs on those.

Server logs: https://pastebin.ramirodelcorro.com/rZwxrWHC
Redis logs: https://pastebin.ramirodelcorro.com/AfsYCDhM#redislogs

Q1 / Q2 : https://pastebin.ramirodelcorro.com/QJDafd6b

Noting on asset_file:
immich=# SELECT * FROM asset_file; (0 rows)

Asset table: https://pastebin.ramirodelcorro.com/aGoNqdVj

Redis events: https://pastebin.ramirodelcorro.com/iopPM2Qx

Thanks!

[skatsubo]

Hey folks!
I'm converting this issue to discussion to have threads.
(Another issue to be created after we get better understanding of what's going on here)