I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.
The bugRelated issues: #12736 #18523 #18965 Any help is greatly appreciated! (And thanks for working on this awesome project!) 😃 
I started getting these in April, and patiently waited for an update to fix it. However its still an issue, even after migrating to vectorchord and updating immich to the latest version (1.135.3). How or why the issue started is hard to say. This has happened consistently with all images after April, 29th, but there's also a few littered around the timeline (altough very few, so that could be a separate issue). Version History
Installed 1.135.3 on 18. juli 2025
Installed 1.134.0 on 18. juli 2025
Installed 1.133.1 on 31. mai 2025
Installed 1.132.3 on 1. mai 2025
Installed 1.131.3 on 13. apr. 2025As you can see above, I didn't upgrade Immich around April 29th. Clues
The OS that Immich Server is running onUbuntu 24.04 -> Microk8s -> Helm Version of Immich Serverv1.135.3 Version of Immich Mobile App1.135.3 Platform with the issue
|
Replies: 20 comments 46 replies
|
Following. Same issue but using docker in unraid |
|
I'm experiencing the same problem on my instance. Version v1.135.3. I hope I'm not mixing up problems, but perhaps there's a dependency." |
|
Hey @etokheim, thanks for the detailed report.
Is redis healthy and listening on port 6379? Do you have more details on redis state?
Redis is required for jobs to work, AFAIK, no redis -> no jobs -> no thumbnail generation. |
|
Umm, @skatsubo, I gotta say I'm puzzled too – probably found it somewhere. Or, maybe I messed it up when moving Postgres out of the helm values? I don't have proper version control for the values, but the Doing a quick check of the helm manifest, I can see that Bitnami's Redis image is used (I guess it's this one). In the readme, they mention mapping Which path do you map in order to persist the Redis database? Regarding the Redis log (click to expand)kc logs --follow sande-immich-server-redis-master-0
1:C 20 Jul 2025 20:19:06.719 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 20 Jul 2025 20:19:06.719 * Redis version=7.4.3, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 20 Jul 2025 20:19:06.719 * Configuration loaded
1:M 20 Jul 2025 20:19:06.719 * monotonic clock: POSIX clock_gettime
1:M 20 Jul 2025 20:19:06.720 * Running mode=standalone, port=6379.
1:M 20 Jul 2025 20:19:06.720 * Server initialized
1:M 20 Jul 2025 20:19:07.647 * Reading RDB base file on AOF loading...
1:M 20 Jul 2025 20:19:07.647 * Loading RDB produced by version 7.4.3
1:M 20 Jul 2025 20:19:07.647 * RDB age 80126 seconds
1:M 20 Jul 2025 20:19:07.647 * RDB memory usage when created 163.90 Mb
1:M 20 Jul 2025 20:19:07.647 * RDB is base AOF
1:M 20 Jul 2025 20:19:14.007 * Done loading RDB, keys loaded: 78896, keys expired: 0.
1:M 20 Jul 2025 20:19:14.007 * DB loaded from base file appendonly.aof.1209.base.rdb: 7.225 seconds
1:M 20 Jul 2025 20:19:23.171 * DB loaded from incr file appendonly.aof.1490.incr.aof: 9.163 seconds
1:M 20 Jul 2025 20:19:23.171 * DB loaded from append only file: 16.390 seconds
1:M 20 Jul 2025 20:19:23.173 * Opening AOF incr file appendonly.aof.1490.incr.aof on server start
1:M 20 Jul 2025 20:19:23.173 * Ready to accept connections tcp
1:M 21 Jul 2025 05:18:56.264 * Starting automatic rewriting of AOF on 100% growth
1:M 21 Jul 2025 05:18:56.366 * Creating AOF incr file appendonly.aof.1491.incr.aof on background rewrite
1:M 21 Jul 2025 05:18:56.371 * Background append only file rewriting started by pid 142432
142432:C 21 Jul 2025 05:19:13.721 * Successfully created the temporary AOF base file temp-rewriteaof-bg-142432.aof
142432:C 21 Jul 2025 05:19:13.723 * Fork CoW for AOF rewrite: current 3 MB, peak 3 MB, average 2 MB
1:M 21 Jul 2025 05:19:13.808 * Background AOF rewrite terminated with success
1:M 21 Jul 2025 05:19:13.808 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-142432.aof into appendonly.aof.1210.base.rdb
1:M 21 Jul 2025 05:19:13.965 * Removing the history file appendonly.aof.1490.incr.aof in the background
1:M 21 Jul 2025 05:19:13.965 * Removing the history file appendonly.aof.1209.base.rdb in the background
1:M 21 Jul 2025 05:19:14.042 * Background AOF rewrite finished successfully
1:M 21 Jul 2025 20:29:48.911 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 21 Jul 2025 20:30:12.028 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 21 Jul 2025 21:33:15.229 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
Thanks by the way, @skatsubo! Appreciate any help! 😄 |
That totally makes sense! I overlooked it. So, then Redis is a red herring. What we can do to finally remove Redis from the equation:
If that does not fix the issue after triggering jobs in Immich, then we at least know that Redis is not related to it. If that does help, we will have a backup to investigate further. Good luck! (Btw, I use plain docker / compose setup, no k8s.) |
|
Stopping Immich, deleting all of Redis’ data then starting up Immich again did the trick for me. Didn’t immediately realize it helped before it had run overnight. If you have the same issue, you can try hitting the Regenerate thumbnails button to check if purging Redis helped 😉 |
|
And as @bo0tzz said, the Redis data can be safely deleted:
|
Right. The idea was to inspect Redis data later to investigate the root cause. But if it's several GBs I doubt it's feasible :) Unless Redis "SAVE" can compact it |
nb the redis data is pretty unimportant, so it's not really necessary to back it up. |
|
What else to check:
select * from assets a
join asset_files af on a.id = af."assetId"
join asset_job_status ajs on a.id = ajs."assetId"
where a."originalPath" ilike '%my_image.jpg';
|
|
FWIW I encountered this recently and had to add |
Thanks for sharing @5aji. I agree that libgomp threads is probably not the cause of this issue. Anyway, good to know. |
|
Exactly the same situation as you, this problem has been bothering me since the beginning of April, I tried to create a brand new immich with the same configuration, the same external pictures, and everything worked fine. But the old immich just doesn't work, and I'm trying to find a way to fix it because I don't want to reset the albums and characters! I've been waiting for months and still no solution, sad! |
|
@yilin101 Could you try resetting Redis?
docker exec immich_redis redis-cli save
docker cp immich_redis:/data/dump.rdb redis-dump.rdb
docker compose stop redis immich-server
docker compose rm -vf redis
docker compose up -dIf that fixes the issue, please share your |
|
Hey folks, @etokheim @yilin101 and whoever has that issue! I suggest to enable extended monitoring, upload an image to trigger thumbnails generation, fetch related recent records from Postgres/Redis. Troubleshooting stepsOutline
Do this during quiet time when there is no other users/activity to reduce noise in logs. Then share the logs and records output. See below step-by-step instructions for docker compose setup.
1. Enable verbose logging in Immich server.Add to your IMMICH_LOG_LEVEL=verboseApply: docker compose up -d2. Enable activity watching in Redis.Watch all activity related to thumbnails (it includes constant refresh/noise from BullMQ): docker exec -ti immich_redis redis-cli monitor | grep thumbnailGeneration3. Upload an image and refresh its thumbnails
4. Save Immich and Redis logs from steps 1 and 2.Redis:
Immich:
5. Fetch recent records from Postgres.Connect to the database: docker exec -ti --user postgres immich_postgres psql -Ppager=no -Pexpanded=auto -d immichQuery 1: recent records with missing SELECT * FROM asset_job_status ajs
JOIN asset a ON a.id = ajs."assetId"
WHERE "thumbnailAt" IS NULL
ORDER BY "metadataExtractedAt" DESC LIMIT 4;Query 2: recent records having SELECT * FROM asset_job_status ajs
JOIN asset a ON a.id = ajs."assetId"
JOIN asset_file af ON a.id = af."assetId"
ORDER BY "thumbnailAt" DESC, "metadataExtractedAt" DESC LIMIT 4;If you have your image file name, try also more specific query after substituting file name at the end: select * from asset a
left join asset_file af on a.id = af."assetId"
left join asset_job_status ajs on a.id = ajs."assetId"
where a."originalFileName" = 'MY_UPLOADED_IMAGE.JPG';UPD. Edited queries to match table names in v1.136+. 6. Fetch recent records from Redis.Fetch last 30 events for the thumbnails job. (The command is quite long because it transforms Redis output into more readable form) docker exec immich_redis redis-cli --raw XREVRANGE "immich_bull:thumbnailGeneration:events" + - COUNT 30 | \
awk '/^[0-9]+-[0-9]+$/ { if (NR > 1) print output ; output = "ID=" $0 ; next }
{ if (!key) { key = $0 } else { output = output ", " key "=" $0 ; key = "" } }
END { print output }'Finally: post/share your findingsThere will be 4 outputs:
|
|
I am in the same boat with a new deployment. I do see redis traffic and there is data in the DB but no thumb. Jobs are stuck in their queues. External libraries crawl and find the files but nothing happens after (can't even see the images without thumbs on those. Server logs: https://pastebin.ramirodelcorro.com/rZwxrWHC Q1 / Q2 : https://pastebin.ramirodelcorro.com/QJDafd6b Noting on asset_file: Asset table: https://pastebin.ramirodelcorro.com/aGoNqdVj Redis events: https://pastebin.ramirodelcorro.com/iopPM2Qx 
Thanks! |
|
Thanks a lot @rdelcorro! I will have a closer look at these logs later today.
That's interesting. Wondering if your setup would allow us to reproduce the issue. A few preliminary questions:
|
|
Thanks for looking into this. My setup is pretty simple, a compose and no env file. Re creating redis seems to have worked. Attaching the backup Nothing fancy here beyond annotations for the reverse proxy: https://pastebin.ramirodelcorro.com/awdqkJF7 I attempted to restart the server but no luck multiple times. Jobs were stuck and never progressing. The only think I changed from the startup config was disabling face recognition but I ended up turning it back on just to make sure it was not causing this. Can you explain what memory limit should the ML container have vs the server? Interestingly I re created the stack and something similar happened but this time there were some "active" ones in the list but stuck. Deleted redis and started with a flood of "[Nest] 7 - 08/04/2025, 5:18:17 AM WARN [Microservices:MediaService] Thumbnail generation failed for asset e66d21ce-447c-4638-85b5-fa42199d6b48: not found Then it started processing normally. I will see what is the state of this tomorrow after the cron runs and report back |
That's good news! Hopefully your Redis dump will help us to find the root cause of the issue.
Have you checked CPU utilization by any chance? There were reports about Active jobs with 100% CPU usage - #14349.
Yeah, let's see. Either it was a weird bug in Immich+BullMQ+Redis or the issue is triggered by something in the photo collection.
From the docs
You can also search / ask community on Immich Discord or Reddit. |
|
|
@rdelcorro Could you save
You mean job concurrency settings? Or smth else? Btw, this one is probably more lightweight Redis reset command compared to docker exec immich_redis redis-cli FLUSHALL |
|
Hey folks! |
|
I've been experiencing the same thing since upgrading a few days ago. Although there's no problem generating thumbnails with the dropdown menu refresh option. Server verbose logs: Redis monitor: Postgres (probably an empty table because I manually went through and used the ... dropdown to refresh the thumbnails for recent images): Redis records: |
|
Alas, neither of those two restart/flush commands seems to have done anything. I can't figure out how to attach the redis file, as it's not a supported filetype for upload. ETA: If there's another way to share it, for the love of Bones McCoy also tell me how to redact sensitive info, please. I shouldn't be doing this stuff when sleep restricted. |
|
@Liziloo Hopefully we can handle the sensitive stuff with plain text export from Redis. Thank you for bringing this up.
So in your case the issue persists and is reproducible even with clean Redis state. Interesting, this is different from other cases. Nevertheless, if you don't mind, here is another intimidating script to save Redis records as plain text. So you can scroll through them and check for sensitive info, e.g. search for "jpg" and alike... I think file names/paths can be present in External Library scan jobs (if there are these jobs in waiting/active/failed state). docker exec -i immich_redis bash << 'EOF' > redis.txt
two_lines_to_kv='{ sub(/^[0-9]+\) /, "") ; if (NR % 2) { key = $0 } else { print key ": " $0 } }'
xrange_to_csv='/^[0-9]+-[0-9]+$/ { if (NR > 1) print output ; output = "id=" $0 ; next } { if (!key) { key = $0 } else { output = output ", " key "=" $0 ; key = "" } } END { print output }'
redis-cli --scan --pattern '*' | sort | while read -r key; do
type=$(redis-cli type "$key")
echo -e "\n==== $key ($type) ===="
case $type in
"string") redis-cli get "$key" ;;
"list") redis-cli lrange "$key" 0 -1 ;;
"set") redis-cli smembers "$key" ;;
"zset") redis-cli zrange "$key" 0 -1 withscores | awk "$two_lines_to_kv" ;;
"hash") redis-cli hgetall "$key" | awk "$two_lines_to_kv" ;;
"stream") redis-cli xrange "$key" - + | awk "$xrange_to_csv" ;;
*) echo "Unsupported: $type" ;;
esac
done
EOF |
|
I'll give that a try as soon as I'm at anything better than a cellphone. Meanwhile, is there any chance it has anything to do with the stack's internal networking vs network requests from outside the container network? For instance, could it be that the reason thumbnail generation works from the drop down be because it uses the program's external IP vs when the containers communicate amongst themselves they use internal addresses? I'm just wondering why my case is different. I'm running Immich behind both nginx proxy manager and authelia. So I'm actually just using "expose" instead of "ports" in my docker compose and then the reverse proxy resolves the IP from the container name and port. This has always worked fine, but I'm wondering. |
|
Ok, I definitely think it's an issue of what the containers use as internal IPs vs the external IPs, based on some experimenting last night. Attached are various logs from this morning. I let it run overnight after changing some things in terms of ports. Watching the logs for each container, it definitely affected their connectivity. Then I let it run overnight. When I got up, everything I uploaded yesterday now had thumbnails and was properly located in the library on the host file system. However, pretty much nothing runs from the Jobs panel, whereas last night when I bound more ports to my host system, some were running. Also, though the files from last night are good now, anything I've uploaded this morning doesn't have a thumbnail, is not in the proper library on the host system, and no longer corrects upon use of the dropdown menu. So I ran the troubleshooting steps you originally posted. Then pulled these logs. I'm also attaching the redacted compose and .env files that the server ran on overnight. |
|
Fixed. I had to use "expose" with the ports as in the .yml I attached forty minutes ago. Also, I bound the machine learning port to the host port and changed the machine learning URL in the GUI settings to match the host. Then I needed to use environment variables to change the redis and postgres hostnames to match the ones my system created automatically, thus:
Everything seems to run fine now on my system, at least. In fact, I no longer need to refresh the page after an upload to see the thumbnail render properly. |
Awesome! Some comments/questions on your recent comments/advancements. redis.txtLooking at 1. Waiting jobs
Example: 2900 facial recog jobs in waiting state 2. Processed jobs
==== immich_bull:metadataExtraction:events (stream) ====
id=1754611805521-0, event=paused
id=1754611805620-0, event=resumed
id=1754697670358-0, event=added, jobId=1, name=AssetExtractMetadata
id=1754697670358-1, event=waiting, jobId=1
id=1754697673872-0, event=active, jobId=1, prev=waiting
id=1754697674118-0, event=completed, jobId=1, returnvalue=null, prev=active
id=1754697674118-1, event=drainedContainer IPs and ports
Question Question I'm wondering if re-creation of containers (while reconfiguring ports) also somehow helped to recover it, at least for a night. |
|
Hi, I started an Immich instance for the first time yesterday, ran into this exact issue, and reading your discussion helped me establish the instance properly, following @Liziloo's post about setting the Redis and PostgreSQL endpoints in the .env file. (Thanks for that!) I think I have an idea about what's happening in our environments, because of an issue I saw a few days ago. When we name the service But because I also have a service named Last week I noticed lots of failed login attempts on both PostgreSQL instances of Outline and Linkwarden, from each other, and I realised this must be doing what DNS servers do, a round-robin between these two records. Then, and also yesterday, solution was to setup some other endpoint format for their DBs and Redis instances, to stop them reaching the other stack's container. @Liziloo can you confirm whether there's another stack with a service named A snippet of the keys from my |
Yes, there is another service named simply redis. |
|
@tiagotaino Thanks a lot for chiming in and for your detailed explanation! This exactly aligns with my hypothesis. As @Liziloo confirmed there are "duplicate" services in the sense you described above: both named
|
|
Thank you both for the clarification/illumination. Might explain a couple things I've seen going on. And the Immich team seriously does great work. |
|
Thanks @tiagotaino (and everyone else in the thread) I had the same issue since a while and never linked it to the fact that I deployed a global redis in the same docker network (backend) and it was conflicting with the redis of immich. My problem is sloved 😎 |
|
Hi, guys! Thanks for sharing your problems and how you're trying to solve them! Has been really useful 🙂 I've unfortunately not been able to dedicate any time to this before a late night yesterday. I didn't realize then, but apparently me messing around fixed the problem. I realized today that a lot of the missing thumbnails were generated overnight! The regeneration seemed to have stopped at some point or something, so I went into settings -> Jobs -> Generate thumbnails and clicked Things I tried that had no impactFixed the redis volume mapping (I did this a month ago. Didn't help)
Re-added env variables:
What fixed itNuked RedisPreviously my redis data took over 2.9G space and had a lot of files littered around:I have no name!@sande-immich-server-redis-master-0:/$ ls /data
appendonlydir temp-rewriteaof-169.aof temp-rewriteaof-228.aof temp-rewriteaof-35960.aof temp-rewriteaof-376454.aof temp-rewriteaof-379230.aof temp-rewriteaof-379913.aof temp-rewriteaof-69.aof
lost+found temp-rewriteaof-1707.aof temp-rewriteaof-229.aof temp-rewriteaof-362.aof temp-rewriteaof-376526.aof temp-rewriteaof-379235.aof temp-rewriteaof-379918.aof temp-rewriteaof-70242.aof
temp-rewriteaof-100.aof temp-rewriteaof-1711.aof temp-rewriteaof-2299.aof temp-rewriteaof-363.aof temp-rewriteaof-376815.aof temp-rewriteaof-379240.aof temp-rewriteaof-379930.aof temp-rewriteaof-705.aof
temp-rewriteaof-102.aof temp-rewriteaof-179.aof temp-rewriteaof-230.aof temp-rewriteaof-365543.aof temp-rewriteaof-376955.aof temp-rewriteaof-379251.aof temp-rewriteaof-380065.aof temp-rewriteaof-760943.aof
temp-rewriteaof-1049967.aof temp-rewriteaof-1816.aof temp-rewriteaof-231.aof temp-rewriteaof-371299.aof temp-rewriteaof-376997.aof temp-rewriteaof-379262.aof temp-rewriteaof-380079.aof temp-rewriteaof-807.aof
temp-rewriteaof-1068.aof temp-rewriteaof-182.aof temp-rewriteaof-232118.aof temp-rewriteaof-372539.aof temp-rewriteaof-377054.aof temp-rewriteaof-379348.aof temp-rewriteaof-380105.aof temp-rewriteaof-81.aof
temp-rewriteaof-109941.aof temp-rewriteaof-183262.aof temp-rewriteaof-233.aof temp-rewriteaof-373.aof temp-rewriteaof-377239.aof temp-rewriteaof-379376.aof temp-rewriteaof-380241.aof temp-rewriteaof-82.aof
temp-rewriteaof-117.aof temp-rewriteaof-1857.aof temp-rewriteaof-234.aof temp-rewriteaof-373011.aof temp-rewriteaof-377451.aof temp-rewriteaof-379383.aof temp-rewriteaof-380278.aof temp-rewriteaof-83.aof
temp-rewriteaof-1245.aof temp-rewriteaof-188.aof temp-rewriteaof-235.aof temp-rewriteaof-373224.aof temp-rewriteaof-378011.aof temp-rewriteaof-379391.aof temp-rewriteaof-380289.aof temp-rewriteaof-846.aof
temp-rewriteaof-1246.aof temp-rewriteaof-189.aof temp-rewriteaof-236.aof temp-rewriteaof-373608.aof temp-rewriteaof-378075.aof temp-rewriteaof-379399.aof temp-rewriteaof-380338.aof temp-rewriteaof-852.aof
temp-rewriteaof-127.aof temp-rewriteaof-190.aof temp-rewriteaof-238.aof temp-rewriteaof-373651.aof temp-rewriteaof-378114.aof temp-rewriteaof-379447.aof temp-rewriteaof-380340.aof temp-rewriteaof-894.aof
temp-rewriteaof-1328.aof temp-rewriteaof-1905.aof temp-rewriteaof-2388.aof temp-rewriteaof-373741.aof temp-rewriteaof-378370.aof temp-rewriteaof-379473.aof temp-rewriteaof-380382.aof temp-rewriteaof-8994.aof
temp-rewriteaof-133.aof temp-rewriteaof-1968.aof temp-rewriteaof-2409.aof temp-rewriteaof-373803.aof temp-rewriteaof-378530.aof temp-rewriteaof-379507.aof temp-rewriteaof-380492.aof temp-rewriteaof-91.aof
temp-rewriteaof-1332.aof temp-rewriteaof-200.aof temp-rewriteaof-255.aof temp-rewriteaof-373975.aof temp-rewriteaof-378662.aof temp-rewriteaof-379539.aof temp-rewriteaof-421.aof temp-rewriteaof-912.aof
temp-rewriteaof-1374.aof temp-rewriteaof-202951.aof temp-rewriteaof-256.aof temp-rewriteaof-374062.aof temp-rewriteaof-378971.aof temp-rewriteaof-379566.aof temp-rewriteaof-427383.aof temp-rewriteaof-915.aof
temp-rewriteaof-140.aof temp-rewriteaof-2075.aof temp-rewriteaof-260.aof temp-rewriteaof-374313.aof temp-rewriteaof-378988.aof temp-rewriteaof-379588.aof temp-rewriteaof-45296.aof temp-rewriteaof-95.aof
temp-rewriteaof-144.aof temp-rewriteaof-208.aof temp-rewriteaof-290.aof temp-rewriteaof-375105.aof temp-rewriteaof-379057.aof temp-rewriteaof-379637.aof temp-rewriteaof-45456.aof temp-rewriteaof-958.aof
temp-rewriteaof-146.aof temp-rewriteaof-209.aof temp-rewriteaof-295.aof temp-rewriteaof-375480.aof temp-rewriteaof-379072.aof temp-rewriteaof-379719.aof temp-rewriteaof-46587.aof temp-rewriteaof-960.aof
temp-rewriteaof-1512.aof temp-rewriteaof-210.aof temp-rewriteaof-299.aof temp-rewriteaof-375594.aof temp-rewriteaof-379122.aof temp-rewriteaof-379857.aof temp-rewriteaof-53.aof temp-rewriteaof-97.aof
temp-rewriteaof-1534.aof temp-rewriteaof-214.aof temp-rewriteaof-302.aof temp-rewriteaof-375991.aof temp-rewriteaof-379163.aof temp-rewriteaof-379877.aof temp-rewriteaof-54.aof temp-rewriteaof-98.aof
temp-rewriteaof-1600.aof temp-rewriteaof-2230.aof temp-rewriteaof-322.aof temp-rewriteaof-376224.aof temp-rewriteaof-379194.aof temp-rewriteaof-379884.aof temp-rewriteaof-541.aof temp-rewriteaof-99.aof
temp-rewriteaof-162.aof temp-rewriteaof-22522.aof temp-rewriteaof-324.aof temp-rewriteaof-376322.aof temp-rewriteaof-379201.aof temp-rewriteaof-379906.aof temp-rewriteaof-586.aof temp-rewriteaof-bg-496.aof
temp-rewriteaof-168.aof temp-rewriteaof-227.aof temp-rewriteaof-342.aof temp-rewriteaof-376372.aof temp-rewriteaof-379208.aof temp-rewriteaof-379912.aof temp-rewriteaof-64.aof
I have no name!@sande-immich-server-redis-master-0:/$ du -hs /data
2.9G /data
I have no name!@sande-immich-server-redis-master-0:/$ du -hs /data/appendonlydir/
102M /data/appendonlydir/
I have no name!@sande-immich-server-redis-master-0:/$ ls /data/appendonlydir/
appendonly.aof.1252.base.rdb appendonly.aof.1533.incr.aof appendonly.aof.manifestAlso the job page looked like this:
Starting Redis with a fresh volumeThe folder structure is now very clean with very few files. The job page also has no errors, 0 waiting and 0 active in all sections 🙂 Moving back to the old Redis volumeNow things have stopped working again. No new thumbnails after uploading new assets. This proves that the issue indeed lies with Redis. I really rather not spend any more time on this issue, but if anyone is interested in diving into it, I can provide some highlights from my ChatGPT assisted debugging attempt (I'm not really familiar with using Redis). (Sorry if this is lame): Debugging with ChatGPTI'm trying to debug why immich's thumbnail generation fails for me. Using "redis-cli monitor | grep thumbnailGeneration" while clicking "refresh thumbnails" from immich gives a lot of output from redis. Short snippet:
kc logs deploy/sande-immich-server-machine-learning
[08/21/25 20:27:13] INFO Starting gunicorn 23.0.0
[08/21/25 20:27:13] INFO Listening at: http://[::]:3003 (8)
[08/21/25 20:27:13] INFO Using worker: immich_ml.config.CustomUvicornWorker
[08/21/25 20:27:13] INFO Booting worker with pid: 9
[08/21/25 20:27:17] INFO Started server process [9]
[08/21/25 20:27:17] INFO Waiting for application startup.
[08/21/25 20:27:17] INFO Created in-memory cache with unloading after 300s of inactivity.
[08/21/25 20:27:17] INFO Initialized request thread pool with 12 threads.
[08/21/25 20:27:17] INFO Application startup complete.Should there be more activity in the machine-learning logs?
Trying to find error messages in Redis:redis-cli
127.0.0.1:6379> LRANGE immich_bull:thumbnailGeneration:failed 0 -1
(error) WRONGTYPE Operation against a key holding the wrong kind of value
TYPE immich_bull:thumbnailGeneration:failed
zset
ZREVRANGE immich_bull:thumbnailGeneration:failed 0 9 WITHSCORES
1) "61379"
2) "1755808105396"
3) "61380"
4) "1753823626582"
5) "61348"
6) "1753811608056"
7) "58713"
8) "1752873547757"
9) "58599"
10) "1752607422253"
11) "58598"
12) "1752607422253"
13) "58597"
14) "1752607422253"
15) "58596"
16) "1752607422253"
17) "58595"
18) "1752607422253"
19) "58594"
20) "1752607422253"
HGETALL immich_bull:thumbnailGeneration:61379
1) "stc"
2) "2"
3) "priority"
4) "0"
5) "finishedOn"
6) "1755808105396"
7) "timestamp"
8) "1746119386765"
9) "processedOn"
10) "1753823596566"
11) "delay"
12) "0"
13) "data"
14) "{\"id\":\"7b383c7e-06e0-4cba-bed5-cb20146722db\"}"
15) "ats"
16) "2"
17) "opts"
18) "{\"removeOnFail\":false,\"removeOnComplete\":true,\"delay\":0,\"attempts\":3}"
19) "failedReason"
20) "job stalled more than allowable limit"
21) "name"
22) "generate-thumbnails"
LRANGE immich_bull:thumbnailGeneration:active 0 -1
1) "61345"
2) "61346"
3) "61347"
4) "61349"
5) "61350"
6) "61351"
7) "61352"
8) "61353"
9) "61354"
10) "61355"
11) "61356"
12) "61357"
13) "61358"
14) "61359"
15) "61360"
16) "61361"
17) "61362"
18) "61363"
19) "61364"
20) "61365"
21) "61366"
22) "61367"
23) "61368"
24) "61369"
25) "61370"
26) "61371"
27) "61372"
28) "61373"
29) "61374"
30) "61375"
31) "61376"
32) "61377"
127.0.0.1:6379> SMEMBERS immich_bull:thumbnailGeneration:stalled
(empty array)
127.0.0.1:6379> LRANGE immich_bull:thumbnailGeneration:wait 0 20
1) "101666"
2) "101665"
3) "101664"
4) "101663"
5) "101662"
6) "101661"
7) "101660"
8) "101659"
9) "101658"
10) "101657"
11) "101656"
12) "101655"
13) "101654"
14) "101653"
15) "101652"
16) "101651"
17) "101650"
18) "101649"
19) "101648"
20) "101647"
21) "101646"
GET immich_bull:thumbnailGeneration:61345:lock
"799355fa-67bf-4b8d-b85c-3b73ccb75dda:31"
TYPE immich_bull:thumbnailGeneration:events
stream
XINFO STREAM immich_bull:thumbnailGeneration:events
1) "length"
2) (integer) 10064
3) "radix-tree-keys"
4) (integer) 125
5) "radix-tree-nodes"
6) (integer) 268
7) "last-generated-id"
8) "1755808676100-1"
9) "max-deleted-entry-id"
10) "0-0"
11) "entries-added"
12) (integer) 336900
13) "recorded-first-entry-id"
14) "1752873806366-0"
15) "groups"
16) (integer) 0
17) "first-entry"
18) 1) "1752873806366-0"
2) 1) "event"
2) "added"
3) "jobId"
4) "98705"
5) "name"
6) "generate-thumbnails"
19) "last-entry"
20) 1) "1755808676100-1"
2) 1) "event"
2) "waiting"
3) "jobId"
4) "101666"
XREVRANGE immich_bull:thumbnailGeneration:events - + COUNT 20
(empty array)Checking immich-server logsTailing the logs while refreshing thumbnails only outputs a single line: [Nest] 17 - 08/21/2025, 9:12:27 PM DEBUG [Api:LoggingInterceptor~b0vhxh25] POST /api/assets/jobs 204 63.09ms 10.71.128.64 I then confirmed that the container indeed have read, write and space.Some random checking in RedisLooking for an asset in postgres:I have now upgraded Immich to the latest version. Thanks everyone for the help! Special thanks to @skatsubo and @Liziloo 😄 PS: Please give a heads up if nuking your Redis data fixes the issue for anyone else as well! Also, should I close the issue? |
|
@etokheim That's good news! Thanks a lot for all the details. The debugging session is rather valuable too. Based on your observations I suspect an edge case or rare bug in BullMQ or Immich. It manifests probably at high load and then the jobs state (stored in Redis and read by BullMQ) becomes borken and then BullMQ/Immich are not able to move further with processing the queues.
Yes. |
|
HI, I have immich on 1.139.3 running as home assistant addon , with same problem , nuking the redis did not help "it is actually automatically done with each addon restart , my background is shallow so excuse my ignorance |
|
Hey @samimasad Is your issue still relevant / reproducible? If so:
If it's overwhelming, we can start with step 1 for now + whatever else you prefer. |
|
seems it is related to vips_NOVECTOR = 1, will try and report back for confirmation |
|
that did not help , what helped is change the thmubnails generations to as for the container i am using HA addon https://github.com/alexbelgium/hassio-addons/tree/master/immich |
|
correction : vips_NOVECTOR = 1 did stop the silent failures in my HA running on Rpi4 , nowI can create thumbnails again with WebP format |
Cool! Next Immich version will include the fix, then |
|
Similar (same?) issue for me since 11. Sept. Looking at my snapshot history and generated preview, it worked fine till (including) 1.137.3 System: Proxmox -> LXC -> dockge (docker-compose) The image shown has been uploaded manually. Todays images are both screenshots taken and uploaded by ShareX. 
It says Video, but its a pic. I can download it (filename: filename) and open it without problems with Paint. deleting the redis container and restarting didn't change anything Logs: Just reread every release notes for each version since 1.137.3 but couldn't see any steps I have missed. Thanks for any hint or help. |
|
@buyenne Could you re-check the files to make sure they actually are images or videos? Names like If you can share some file that is misdetected by Immich as |
|
@skatsubo - sure. I made a screenshot, it was automaticly uploaded to immich through shareX, the online exif tool gave me this: Since I didn't want to exclude, it beeing a possible error by ShareX I have checked when I made the last update. It was on the 19.08.2025 
I have just rechecked my uploader config in ShareX. It's identical with the config I made on the 12.08.2025 Slowly I tend to put the blame on ShareX / Windows. Edit: recap of the timeline to avoid the confusion Short explanation - uploads file with %filename% instead of screenshot_30092025.png. Drag&Drop from the local shareX folder works like a charm. PS: sorry if I hijacked this thread if the problem not beeing an immich issue. Didn't want to open another (same) issue with my knowledge from this morning. |
|
It finally works again. Quick solution. I removed the last line (highlighted part). Why did it work before? I have no idea. Why it stopped working right after an update? Pure coincidence? No idea either. |
|
Thank you for the update @buyenne, glad you have resolved it. Interesting, there was a change in |
|
Ok seems it was the change that broke the upload. Matched perfectly with my upgrade path. |
|
I just installed Immich last month with v1.141.1. It's been a pretty rough experience due to this. I tried the solutions here for resetting redis but any new images always start with broken thumbnails. This happened with my initial upload but i manually ran the thumbnail job to create them then it worked. Recently they've entirely stopped updating though. name: immich
services:
immich-server:
container_name: immich_server
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
# extends:
# file: hwaccel.transcoding.yml
# service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
volumes:
# This line reads the path from your .env file
- ${UPLOAD_LOCATION}:/data
- /etc/localtime:/etc/localtime:ro
env_file:
- .env
# The internal port is 2283. You don't need to expose this if using a proxy manager.
ports:
- '2283:2283'
depends_on:
- redis
- database
restart: always
networks:
- selfhost
immich-machine-learning:
container_name: immich_machine_learning
image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
volumes:
- ./immich-data/model-cache:/cache
env_file:
- .env
restart: always
networks:
- selfhost
redis:
container_name: immich_redis
image: docker.io/valkey/valkey:8-bookworm@sha256:a137a2b60aca1a75130022d6bb96af423fefae4eb55faf395732db3544803280
restart: always
networks:
- selfhost
database:
container_name: immich_postgres
image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:32324a2f41df5de9efe1af166b7008c3f55646f8d0e00d9550c16c9822366b4a
environment:
POSTGRES_PASSWORD: ${DB_PASSWORD}
POSTGRES_USER: ${DB_USERNAME}
POSTGRES_DB: ${DB_DATABASE_NAME}
POSTGRES_INITDB_ARGS: '--data-checksums'
volumes:
# This line reads the path from your .env file
- ${DB_DATA_LOCATION}:/var/lib/postgresql/data
shm_size: 128mb
restart: always
networks:
- selfhost
networks:
selfhost:
external: true Example env: # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables
# The location where your uploaded files are stored
UPLOAD_LOCATION=./library
# The location where your database files are stored. Network shares are not supported for the database
DB_DATA_LOCATION=./postgres
# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# TZ=Etc/UTC
# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release
# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=postgres
# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
# i tried consulting gemini and it suggested these two changes, which i tried, but didn't do anything useful for me
DB_HOSTNAME=immich_postgres
REDIS_HOSTNAME=immich_redispermissions: |
That totally makes sense! I overlooked it. So, then Redis is a red herring.
What we can do to finally remove Redis from the equation:
If that does not fix the issue after triggering jobs in Immich, then we at least know that Redis is not related to it. If that does help, we will have a backup to investigate further.
Good luck!
(Btw, I use plain docker / compose setup, no k8s.)