Upload file via cowboy

[comptekki]

I'm returning to this issue again:

#1433

I thought I had it working, but for some time now it has not. I've tried increasing the values in max_received_frame_rate, but the upload process quits at about 27%. Any idea what I can do to allow gig sized files to be uploaded to a cowboy server via websockets?

Thanks!

[essen]

Via Websocket? Not HTTP/2? Linked ticket is about the latter.

What error do you get?

[comptekki]

I don't see any error. My webapp shows a progress percent as it uploads, and it just quits at 27%. Smaller files work fine. It is websockets not http/2. Maybe the original fix was a fluke, if these parameters are only for http/2.

[comptekki]

A year ago, I was looking through the commits trying to find what might have changed in 2.7.0 from 2.6.3 to cause this issue. http/2 probably wasn't the issue.

[essen]

Is the terminate callback called, and if so what's the reason for termination? It's entirely possible that the client closes the connection.

[comptekki]

In my project, if I use 2.8.0 or 2.7.0, this issue happens. If I go back to 2.6.3, it uploads the file fine.

[comptekki]

Maybe there is a websockets change back in 2.7.0 I need to find. New parameters to use?

[essen]

I understand but you need to provide information or a reproducible test case for me to help.

[comptekki]

Right. Sorry. I'm looking through the commits in 2.7.0 to see what I can find. Thanks! I think you have an upload example in cowboy. Try a 200 gig file on it. Maybe you'll see what I'm seeing. I can't remember if that is a websockets example, though.

[comptekki]

No. That's not a websockets example.

[comptekki]

When I dig in to my code, I see I'm using code very similar to your upload example. So something may have changed in cowboy_req:read_part_body where I'm getting the data to write to disk. Would gen_tcp params affect any of this?

[comptekki]

I printed out the Req var and see: streamid => 23,version => 'HTTP/2'

Is that using http/2 to upload the file?

[comptekki]

I just switched my project back to cowboy 2.6.3 and the ~400meg file uploads fine and I don't see anything different with the Req var outputs.

[essen]

That's not Websocket then, you're uploading via HTTP/2. And read_part_body is about multipart as well, so not just a normal upload. It would help to have an example I can reproduce.

[comptekki]

I just tried the upload example in cowboy 2.8.0 and it failed to upload a 389mb file. The browser (firefox) comes up with a message: "The connection was reset" and in the terminal where I did: make run for the example it has a bunch of numbers, which is binary output then at the end it has:

106,198,157,216,36,38,151,176,118,124>>,

#{bindings => #{},body_length => 408836021,cert => undefined,has_body => true,headers => #{<<"accept">> => <<"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8">>,<<"accept-encoding">> => <<"gzip, deflate">>,<<"accept-language">> => <<"en-US,en;q=0.5">>,<<"connection">> => <<"keep-alive">>,<<"content-length">> => <<"408836021">>,<<"content-type">> => <<"multipart/form-data; boundary=---------------------------187645155928358590352183544006">>,<<"host">> => <<"localhost:8080">>,<<"origin">> => <<"http://localhost:8080">>,<<"referer">> => <<"http://localhost:8080/">>,<<"upgrade-insecure-requests">> => <<"1">>,<<"user-agent">> => <<"Mozilla/5.0 (X11; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0">>},host => <<"localhost">>,host_info => undefined,method => <<"POST">>,multipart => {<<"---------------------------187645155928358590352183544006">>,<<>>},path => <<"/upload">>,path_info => undefined,peer => {{127,0,0,1},54274},pid => <0.451.0>,port => 8080,qs => <<>>,ref => http,scheme => <<"http">>,sock => {{127,0,0,1},8080},streamid => 1,version => 'HTTP/1.1'}}} and stacktrace [{upload_h,init,2,[{file,"src/upload_h.erl"},{line,10}]},{cowboy_handler,execute,2,[{file,"src/cowboy_handler.erl"},{line,37}]},{cowboy_stream_h,execute,3,[{file,"src/cowboy_stream_h.erl"},{line,306}]},{cowboy_stream_h,request_process,3,[{file,"src/cowboy_stream_h.erl"},{line,295}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]

[comptekki]

Go back to cowboy 2.6.3, it works.

[essen]

OK I'll try it when I can and see.

[essen]

I would not expect the upload example to be able to accept a 389MB file considering it reads only up to 8MB. I doubt that it works even with 2.6.3, what most likely happens is simply that the way it stops after that differs.

[comptekki]

Hmm. You are right. I thought I had found something that showed the issue in going back and fourth between 2.8.0 and 2.6.3 in the example/upload. I'll need to look some more.

Thanks.

[comptekki]

OK. I have been looking through the docs to see how to increase this for the example. What would I change to increase the file size upload max? I have also been testing in my project that uses 2.8.0. It maxes out at a little over 100meg before it quits. I see that the cowboy_req:read_part_body(Req), calls stream_multipart, which calls read_body, which has a receive in it. I don't see what controls the max body size. I've tested different length, period, timeout. Those don't help sending a larger file. Thanks.

[comptekki]

I thought this would work:

{env => #{dispatch => Dispatch, length => 600000000}}

and

{env => #{dispatch => Dispatch, #{length => 600000000}}}

but length didn't change.

Did this (in myapp_app.erl) and saw the change in length size, but it didn't increase the upload size:

{"/upload", upload_handler, #{length => 600000000}}

[comptekki]

I tried:

{"/upload", upload_h, #{length => 600000000}}

in the upload cowboy example and it worked with my ~100meg file, but when I tried my test 389meg file, it ended up with:

make: *** [../../erlang.mk:7113: run] Killed

I don't see any message in my project. It just quits uploading. (my project, that is, not the cowboy example)

[comptekki]

I've been changing these:

read_body(Req, Opts) ->
Length = maps:get(length, Opts, 8000000),
Period = maps:get(period, Opts, 15000),
Timeout = maps:get(timeout, Opts, Period + 1000),

but this doesn't change the upload file size. Any pointers about what to change? Thanks

[comptekki]

I've been changing these:

read_body(Req, Opts) ->
Length = maps:get(length, Opts, 8000000),
Period = maps:get(period, Opts, 15000),
Timeout = maps:get(timeout, Opts, Period + 1000),

but this doesn't change the upload file size. Any pointers about what to change?

This is how I use upload in my project:

https://github.com/comptekki/esysman/blob/master/src/upload_handler.erl

Thanks

[comptekki]

I see what you mean now why the cowboy example will only work with 8mb. There is only one call to read_part_body, so no loop over remaining data, if larger. :)

Now I just need to figure out why the process quits at about 100meg file size when looping over read_part_body.

[comptekki]

I thought I had done this already. in appname_app.erl, I did this and it works:

#{env => #{dispatch => Dispatch},
max_received_frame_rate => {100000, 10000}}

[comptekki]

I think the problem was I was putting max_.... in the dispatch map.

[essen]

So the default value is still too small?

[comptekki]

Yes. I added another 0 to the first number so it wouldn't timeout for a more than 100meg file (tested with a 380meg file). Will test larger ones tomorrow.

[comptekki]

I tested an 800 file and that worked with max_received_frame_rate => {100000, 10000}}, I had to change it to max_received_frame_rate => {150000, 10000}} for a 1.2 gig file to upload. I'm happy with that!

[essen]

OK let's keep this open so the defaults can be increased again, or at least to have some explanation as to what's going on added to the guide.

[comptekki]

OK. Having an example of how to manually change default parameters, like max_received_frame_rate, would be helpful in the docs :)

[essen]

Closing as duplicate of #1523 and I think one other. I will probably increase the default again. Thanks!