ResponseTooBig: how to push to file rather than memory?

[SamuelMarks]

I'm getting a ResponseTooBig error. I could adjust max_response, but then my memory usage would increase.

Could I instead 'stream' to a file, in chunks relative to speed (speed of disk IO, speed of network IO)?

[SergejJurecko]

Lower level streaming API should be used in this case. But yes write to file would also be a good feature.

[SamuelMarks]

@SergejJurecko On a related note, my download function is very slow, multiples slower than curl (which is multiples slower than aria2c).

Is there a trick to resolve this speed issue?

[SergejJurecko]

Debug is unfortunately crazy slow. I'm not sure what to do about that.

[SamuelMarks]

@SergejJurecko So you're saying it's these two which need removing, or hiding behind a compiler flag:

https://github.com/offscale/offregisters-lib/blob/134d216/src/download.rs#L50-L58
https://github.com/offscale/offregisters-lib/blob/134d216/src/download.rs#L16

[SergejJurecko]

No what I meant was running mio_httpc and not compiling with --release will be pretty slow.

[SamuelMarks]

Okay, well I'm looking at moving to hyper then, will see if I get the expected performance improvement

[helinwang]

@SamuelMarks I thought @SergejJurecko meant compiling with --release should be fast?

[helinwang]

@SergejJurecko thanks for the library! Do you have an idea of the performance of mio_httpc vs. libraries like hyper? I am trying to find a lightweight http client with low latency with ~100 concurrent requests.

[SergejJurecko]

My main goals were:

• based on mio

• easily integratable into other mio based apps

• safe

• not being tied to a single TLS implementation

• reasonably memory efficient

• not being too dependency heavy (native-pinning branch is also split into features so that one can further cut down on dependencies at the cost of functionality)

I have not measured performance against other libraries as I don't really care all that much. It is more then fast enough for anything I need it for. It is not really doing anything egregious that would be problematic. Allocations and data copying are both kept at a minimum. Individual calls are stored in a slab and accessed by index (as opposed to a hash table).

[SergejJurecko]

100 concurrent requests is nothing

[helinwang]

Great to know, thank you!
Btw, I care more about latency than throughput. But your answer is helpful nonetheless.