Upload times out for large files

[mardom1]

I am struggling to upload a large file as the upload endpoint times out after 30 seconds. I have tried uploading via DSC, web ui and simple curl. Each time after exactly 30s, the request times out with status code 503 and a "Response timed out" message.

At first, I thought the problem was with the reverse proxy, but after getting the same result with no reverse proxy in between, I think the problem might be with Docspell itself.

With my limited understanding of the code and Scala, my guess is that this option of the BlazeServerBuilder might be the cause.

  * @param responseHeaderTimeout: Time from when the request is made until a
  *    response line is generated before a 503 response is returned and the
  *    `HttpApp` is canceled

[eikek]

That looks like the correct parameter. This could be pushed to the config file, so users can change the value. But I'd like to also see what's going on, because 30s to get to a response should definitely be doable. When uploading files, they are only streamed into the database. How large is large in your case? Can you maybe provide an example file (and maybe some system specifics) so I can reproduce?

[mardom1]

The file is a scanned document and around 90 MB. My current (temporary) setup is, however, not helping either with Docspell being in a remote location and on slow hardware (old laptop with an Intel Core 2 Duo and only 4 GB of RAM).

A configuration option would be awesome until I can move Docspell back onto proper hardware.

[eikek]

Ok, that makes sense with a slow connection and large files. Depending on the content you might run in problems when processing this file on this hardware.

[falnos24865]

I'd to add this is happening to me as well. Any file around 100MB or more triggers this. My uploads are done on a local network to a moderately beefy machine (6 core, 32GB of RAM). After 30 - 40 seconds the upload times out.

[eikek]

What kind of file is it and do you have a reverse proxy in between? If it's a local link, it shouldn't run into the timeout mentioned above, I had hoped.

[mardom1]

Reading back my comments, it seems like I did not mention that I also tried uploading it locally from the same machine Docspell is running on (using dsc). So even without any network and reverse proxy in between it takes longer than 30 seconds and times out.

[eikek]

Ok I see. It could also be caused by an out-of-memory error. I didn't put extra effort in supporting large files. The request might be loaded into memory, not sure - have to check this.

[eikek]

I fixed the upload to allow large files and tried with a pdf with more than 1000 pages and ~140M size. It was uploaded fine and even processed, but took almost an hour to run through all the processing steps :-). The timeout can be configured now. But I think even when this is set to a higher value, at some point other timeouts could apply (browser, reverse proxy).

[falnos24865]

I have updated to 0.34.0 and am still seeing this. Where is the configuration for this timeout?

[eikek]

The default options are in the docs, see here at server-options.

[falnos24865]

I'm not sure the fix is fully working then. I set the DOCSPELL_SERVER_SERVER__OPTIONS_RESPONSE__TIMEOUT to various different values (from 1s to 1000s) and the option does not seem to affect anything. It always takes the same (roughly 30seconds) for a timeout to occur.

[eikek]

I think we need a new issue for this then. Is there by chance a reverse proxy in between that could also trigger a timeout? I can add some logging to see what is really set into the server configuration.