Large video file uploads get stuck at 0% for long periods of time.

[nexryai]

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

When uploading large video files, the file upload often gets stuck at 0% for long periods of time, and it takes a long time for the upload to fail with a 413 error.
The first file fails with 413, but the others get stuck at 0%.

It is unclear whether this is an Immich issue or a Cloudflare issue.
This causes extremely long upload times for directories containing large numbers of video files.
If this is not an Immich bug, then large video files that are clearly impossible to upload should be skipped instead of uploaded.
To avoid this issue, a feature is needed that allows server administrators to set a limit on the size of uploadable files and automatically skip those files.

The OS that Immich Server is running on

Ubuntu Server 24.04

Version of Immich Server

v2.2.3

Version of Immich Mobile App

Web Client

Platform with the issue

• Server
• Web
• Mobile

Device make and model

No response

Your docker-compose.yml content

Default configuration file without modifications

Your .env content

Default configuration file without modifications

Reproduction steps

1. Drag a bunch of large video files into the web client, the size of which is not supported by the reverse proxy.
2. The first file fails with 413, but the others get stuck at 0%.

...

Relevant log output

Additional information

No response

[skatsubo]

Hey

then large video files that are clearly impossible to upload should be skipped

Do you have a suggestion/idea/principle on how to establish that some assets "are clearly impossible to upload"? (Not a code suggestion, just algorithm/concept)

Re: Cloudflare. There is a section in FAQ https://docs.immich.app/FAQ/#why-are-only-photos-and-not-videos-being-uploaded-to-immich

If you are using Cloudflare Tunnel, please know that they set a maximum filesize of 100 MB that cannot be changed. At times, files larger than this may work, potentially up to 1 GB. However, the official limit is 100 MB. If you are having issues, we recommend switching to a different network deployment.

Also, you can search GH Issues and Discussions for "cloudflare large". A couple of recent threads:

• Android app cannot upload images to server, getting Payload Too Large error #23620
• [Feature] Chunk large files on upload. #23276

[nexryai]

I understand that the inability to upload large files is not an issue on Immich's side.

The main issue here is not the inability to upload large files, but rather that when uploads get stuck on a few specific files for some reason, the queue becomes occupy by the stuck files, making it extremely long to complete (or even impossible to complete) the upload of other files.

This is particularly problematic when uploading a directory containing many files, as you'll have to remove the video files and start the upload again.

Regardless of whether or not Cloudflare is used on the Immich server, reverse proxies (such as nginx) typically have some limit on the size of uploadable files. This is recommended from a security perspective. Modern high-resolution video files often far exceed these sizes.

Do you have a suggestion/idea/principle on how to establish that some assets are "clearly impossible to upload"? (Not a code suggestion, just an algorithm/concept)

If the administrator is aware of the reverse proxy's limitations (which is likely the case in most cases), they can set them in the control panel and have Immich's client-side processing skip uploading files that exceed the limit.
This would allow the upload to explicitly indicate that it exceeds the allowed size, rather than simply failing or getting stuck, which I think would also improve usability.

Alternatively, I would suggest implementing a workaround, such as a button to abort individual files during upload. This would allow users to skip over large files if they've accidentally added them to the upload queue.

[skatsubo]

Thank you for the explanation.

Currently Immich server and web UI properly handles explicit connection state changes such as HTTP 4XX and connection resets. It marks the asset upload attempt as failed: Unable to upload file in UI.
Immich does not implement special handling of more subtle signs, such as poor network conditions or "slow DoS" by an intermediate entity (I'm looking at CF).

So your suggestions above sound like a feature request(s). Broadly speaking: more sophisticated asset upload mechanism. In particular:

1. Smart handling of stalled/slow uploads to avoid blocking the upload queue by several slow items
2. Better informing about the reason for upload failure in the web UI (smth like that already done on mobile, see Android app cannot upload images to server, getting Payload Too Large error #23620 (comment))
3. Option "to abort individual files during upload"
4. Configurable option and client-side support to skip uploading files that exceed the defined limit

I suggest searching through the existing feature requests / discussions, issues, PRs (someone might already tried to implement one of your suggestions) and either comment there or create a new FR as you see fit.
Just keep in mind that the Immich dev team prefers more general concepts/features that cover multiple use cases - compared to a niche feature or a point solution for a narrow problem.
Personally, I like the ideas (1) and (2), while less convinced about (3) and (4).

For now I'm going to convert this topic into a Q-and-A discussion.

---

If you can reproduce the bug (stalled upload queue due to max body size limit) using a reverse proxy such as Nginx/etc instead of Cloudflare then please post the repro steps.

I did a quick test but wasn't able to reproduce it with a basic Caddy config (below), Firefox, uploading 10 large video files.

:8080 {
	request_body {
		max_size 1MB
	}
    reverse_proxy http://localhost:2283
}

[tgharring]

I am testing as well. I do use cloudflare. I decided to use Automatic URL switching and point to the local server address when on my home WiFi. I did confirm i am connecting thru the local ip, but the bug described in this discussion persists. So this should not be a Cloudflare error, I am not sure what else could be causing the bug. hmmm

[demonte21]

I do have the exact same issue. Using Cloudflare (but not Cloudflare Tunnel) No matter if i'm connecting through HTTPS ou through my local network (192.168.x.x) I cannot upload large MP4 file... get stuck at 0% (on my PC) . But on my phone, it seems to try to upload something since I see the phone wifi bandwith speed on the upper right saying i'm uploading at 54 mbps... The file never finally upload and try to upload for ever.`
Linux v2.2.3

[nexryai]

When I tried to reproduce this issue in Safari (macOS 26), I found that the request seemed to get stuck even though the browser's developer console showed a 413 error.

The following error was also output to the browser console:

Error calculating sha1 file=[FILENAME])
NotReadableError: The I/O read operation failed.

In cases where the upload would immediately fail with a 413 error (which is normal), a handleError was thrown and the upload would fail gracefully, rather than getting stuck.

This might not be a Cloudflare or reverse proxy issue, but rather a bug in Immich where it doesn't handle errors properly when hash calculation fails.
I tried uploading the same file multiple times, and sometimes it failed immediately and sometimes it got stuck with a NotReadableError. I don't know what conditions cause the NotReadableError to occur.

[nexryai]

I tried it in Google Chrome and it's stuck with the same error. The request doesn't fail with 413, but the upload doesn't progress past 0%.
I've opened an issue about this problem: #23936

[nexryai]

To investigate the problem, I wrote a simple HTTP server in Go that discards the content POSTed from XMLHttpRequest to /dev/null.
https://github.com/nexryai/large-POST-test-server/tree/main

In Google Chrome, it got stuck similarly to Immich. While Safari sometimes produced different results, it mostly got stuck in the same way.

Based on this, it's likely not a bug in Immich's program but rather a network problem caused by large POSTs. It's possibly related to Cloudflare's DDoS mitigation features, but we have limited knowledge about Cloudflare, so it's unclear. There are also comments that the same problem occurs in environments not using Cloudflare, which suggests it might not be a Cloudflare issue.
I looked at the Cloudflare Tunnel logs and the Cloudflare dashboard logs, but there were no logs of errors or attacks being mitigated.

I believe that implementing Chunked Uploads (#1674) is probably the only fundamental solution. Not using Cloudflare Tunnel is practically impossible, especially in environments without a static global IP or for those using an ISP that doesn't provide a dedicated global IPv4 address to individuals. While it can be avoided with workarounds like purchasing a VPS combined with Tailscale, this is undesirable because it costs money and introduces monthly bandwidth limitations.

[skatsubo]

@nexryai
Thanks a lot for your investigation!
Following your observations I was able to intermittently reproduce the stuck upload which is triggered when server responds with HTTP 4xx/5xx in the middle of upload.

So far I spotted the issue in the following synthetic scenarios where server responds with HTTP 413 or HTTP 401:

• HTTP 413 in a chain: Immich UI in Safari -> toxiproxy (limit bandwidth 1 MB/s) -> caddy (max size 100 MB) -> Immich server.
• HTTP 401 in a synthetic case: open Immich UI, then remove a browser session, try uploading several files. As the session is unauthorized, upload will fail anyway, but it gets stuck quite often too, usually on the second file.
• HTTP 401 using your HTML/JS from https://github.com/nexryai/large-POST-test-server and pointing it to http://localhost:2283/api/assets. With small files ~5MB it completes with Unauthorized errors, as expected. With bigger files 20 - 100 MB it gets stuck quite often. Confirmed: Safari, Chrome, Firefox.
• HTTP 401 using command-line NodeJS XHR and pointing it to http://localhost:2283/api/assets.

UPD regarding HTTP 401. I am able to reproduce the issue with HTTP 401 only against Immich in devcontainer. When VSCode/devcontainer forwards port 3000, it does not close the connection to client when server closes (FIN) and resets (RST) its side of a connection. So the client is unaware that server already closed and reset the connection. I wouldn't be surprised if smth similar happens in real network/setups when an intermediate entity/proxy "eats" some signals, therefore preventing client from detecting closed connection.

I agree that this edge case is not specific to Immich. Though improvements are possible to make Immich UI more robust against this issue, see a suggestion at the end of this comment.

---

Overall, the issue involves a lot of entities and is product of their combined behavior/interference:

• server
• client
• browser
• XHR
• HTTP
• TCP

During "normal" upload:

• XHR timely goes through its states https://xhr.spec.whatwg.org/#states and fires events https://xhr.spec.whatwg.org/#events. Eventually it emits load (fetch succeeded) and ends up in the state DONE (4) with a good HTTP response code. Then the promise uploadRequest is resolved. The upload is successful.
• If an error occurs during upload and if it is detected and signaled by XHR, then the promise gets rejected. An error handler is invoked. We see failed upload and error messages in console [handleError]: Unable to upload file. ApiError: Request Entity Too Large.

Basically, Immich client expects a ProgressEvent to be either "load" (fetch succeeded) or "error" (or alike).

During "abnormal" upload:

• neither happens, client/XHR will wait untill the Big Rip indefinitely, until smth fatal occurs, such as connection termination.

So this issue gets into a gray zone: the XHR fetch neither succeeded nor failed/errored.

---

The event sequence (based on additional event/state logs in console and inspecting network traffic):

1. Client begins XHR upload by POSTing data to /api/assets using fileUploader() and uploadRequest().
2. In the middle of upload server responds with HTTP 4xx/5xx.
3. Now, depending on XHR implementation and context (traffic, TCP state, timing) these event can be detected/reported or not:
   • Case A: XHR detects the state change and eventually finishes the fetch (success/failure/error) - ok.
   • Case B: XHR detects the state change but gets stuck in sending data and does not finish timely - stalled.
   • Case C: XHR does not detect the state change and gets stuck in sending data - stalled.

Why does it get stuck in cases B and C?
Looking at traffic dumps, this happens when XHR attempts to transfer a few more TCP segments, but gets no ACKs from server. Perhaps server already decided to not accept anything more and stopped consuming incoming data. TCP connection is still alive (at least from client point of view), but it has TCP Zero Window. Upon running several experiments I get a bunch of half-finished TCP connections (FIN_WAIT_2 in netstat).

How to fix/handle cases B and C?

• Case B "Detected the state change, yet stuck" can be handled by an additional event listener for readystatechange. If XHR transitions into readyState HEADERS_RECEIVED and status is not successful (e.g. status 4xx/5xx), then abort XHR. (Maybe with extra measures to not interfere with the case (a) because it also goes through this state).
• Case C "The state change is not detected" is more problematic.

[nexryai]

Thank you for the detailed explanation! This seems to be a much more complex and challenging issue than I had imagined.

I agree that Case B can likely be addressed by refining the XHR event handling, but I also agree that Case C is a particularly tricky problem. Handling requests that fall into Case C would require a mechanism to detect prolonged stalls, but it seems difficult to distinguish these from temporary network instability during a legitimate, slow upload.

[Sayrix]

I think that to address this issue the Immich app and server should split uploads into chunks, that way, it will make the upload more reliable by preventing very large files to timeout because of the time it takes to upload a file of several GB, and prevent proxy issues that has a limit on the size of the requests.