nginx. stream cuts off after a certain point

[ghost]

i am using a portable bin and set up nginx as a proxy

when trying to read an infinite stream from the pipe server it cuts off at a certain point
seq inf only return up to about 393936. sends for less then a second

receiver error curl: (18) transfer closed with outstanding read data remaining
sender error curl: (55) OpenSSL SSL_write: Broken pipe, errno 32

2022/02/25 23:16:12 [error] 433215#433215: *288 upstream timed out (110: Connection timed out) while reading upstream, client: 72.178.17.200, server: x.grimmygrom.com, request: "PUT /testing HTTP/1.1", upstream: "http://127.0.0.1:8080/testing", host: "x.grimmygrom.com"

2022/02/25 23:16:12 [error] 433215#433215: *290 upstream prematurely closed connection while reading upstream, client: 72.178.17.200, server: x.grimmygrom.com, request: "GET /testing HTTP/1.1", upstream: "http://127.0.0.1:8080/testing", host: "x.grimmygrom.com"

my nginx config is like this

server {

        server_name x.grimmygrom.com;
        client_max_body_size 0;
        proxy_http_version      1.1; # default: 1.0, Streaming (chunked transfer encoding) requires at least 1.1.
        proxy_buffering         off;
        proxy_request_buffering off;

        location / {
                proxy_pass http://127.0.0.1:8080/;
        }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/x.grimmygrom.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/x.grimmygrom.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {

    if ($host = x.grimmygrom.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

        server_name x.grimmygrom.com;
        listen 80;
    return 404; # managed by Certbot
    
}

video of me showing the problem
https://youtu.be/WyqEju096so

thank you

[nwtgck]

Thank you for the detail report. I will see.

[nwtgck]

What happens when using HTTP instead HTTPS?

Can you ssh-login to the server of x.grimmygrom.com and do curl localhost:8080?

[ghost]

localhost:8080 works fine

[nwtgck]

What about localhost:80?

[ghost]

that goes to nginx. and i get a 404

[nwtgck]

What about curl -k https://localhost:443/testing? The -k option ignore the certificate.

[ghost]

that goes to nginx and i get 404

i set pipe server has port 4321 for https
and when i do curl -k https://localhost:4321/testing
that freezes and i get no output

[nwtgck]

It's OK. I mean that could you try seq inf | curl -kT- https://localhost:4321/testing and curl -k https://localhost:4321/testing like the video?

[ghost]

that works just fine.

[nwtgck]

I'm not sure why HTTPS port is 4321. The nginx configuration says 443. Do you have another reverse proxy for x.grimmygrom.com?

Can you publish https://localhost:4321 directly for x.grimmygrom.com?

[ghost]

i don't have another reverse proxy.
when i try launching pipe server with port 443. it gives me a error and says
[2022-03-06T02:25:41.483] [ERROR] default - on uncaughtException Error: listen EADDRINUSE: address already in use :::443

i had pipe server https on 4321 for testing

what do you mean by publish?

[nwtgck]

I guess the error message address already in use :::443 means that the port 443 is already used for nginx.

that goes to nginx and i get 404

#617 (comment)

Is https://localhost:443 https://x.grimmygrom.com:443?

[ghost]

yes they are both the same

[nwtgck]

If they are the same, doing seq inf | curl -kT- https://localhost:443/testing and curl -k https://localhost:443/testing are the same result as the video? I don't know why you got 404 error.

[ghost]

dose not work. i get error code 405 Not Allowed

my bad its not the same
i think this is due to it needed to be named x.grimmygrom.com so that localhost:443 to work

[nwtgck]

OK. I understand. You can attach -H "Host: x.grimmygrom.com" to the curl commands. Could you try?

[ghost]

that gives me the same error as in this video

streaming stops

[nwtgck]

Good. I'd like to reproduce on local.

Do you have pv command? Could you install it?

[ghost]

i dont? i dont know how to use pv

[nwtgck]

In my opinion, Nginx is very good software, but it is not good fit to HTTP streaming.

Could you try to add pv -qL 100k between seq inf and curl -kT- like below?

seq inf | pv -qL 100k | curl -kT- ...

That command limits transfer speed and it works on my local.

[ghost]

this seems to work and not die

[nwtgck]

Good!

I'd not like to limits transfer speed, but unfortunately I have only this solution for now.

Since Ngix has many options, some options may allow us do streaming well. Maybe it is better to ask Nginx.

[ghost]

thank you so much for the help.

this is very helpful.

[nwtgck]

Thanks. Your video on YouTube was very good for me to understand your situation.

[m2acgi]

Hi @ghost & @nwtgck, it seems that the directives proxy_http_version 1.1; and proxy_buffering off; is not required, but optional. I've test under almalinux 9.0 / nginx 1.20.1 / piping-server docker v1.12.6, there's no any error when send/recieve file >4GB. Do you know why ?

[nwtgck]

@m2acgi How did you send the big file? e.g. curl -T yourbigfile.dat https:///.

If you use curl -T yourbigfile.dat ..., the content-length is specified without chunked encoding. If content-length exists, Nginx should be handle without any error.

[m2acgi]

Yes, curl -T. Is there any nginx reverse proxy doc for piping-server?

[nwtgck]

curl -T- and curl -T <your filepath> are different. Which did you use?

curl -T- creates chunked encoding without content-length and curl -T <you filepath> creates a requet with content-length. With content-length, nginx should handle without any error.

[m2acgi]

I use the curl -T <filepath> in my test. Thank u for ur explain.