Connection hang up when calling ngx.location.capture after read request body in http2

[qleein]

Nginx with lastest lua-nginx-module

root@ubuntu17:/usr/local/nginx# ./sbin/nginx -V
nginx version: nginx/1.13.8
built by gcc 7.2.0 (Ubuntu 7.2.0-8ubuntu3) 
built with OpenSSL 1.0.2g  1 Mar 2016
TLS SNI support enabled
configure arguments: --with-debug --with-cc-opt='-g -O0' --with-ld-opt=-Wl,-rpath=/usr/local/lib/ --with-zlib=/home/kenan/code/zlib-1.2.11 --with-pcre=/home/kenan/code/pcre-8.41 --add-module=/home/kenan/code/lua-nginx-module --with-http_ssl_module --with-http_slice_module --with-http_v2_module --with-stream --with-stream_ssl_module --with-http_stub_status_module

root@ubuntu17:/usr/local/lua-nginx-module# git log
commit df1b0198ff0262c755029738058f5917a11a398a (HEAD, openresty/master)
Author: spacewander <spacewanderlzx@gmail.com>
Date:   Mon Jan 22 14:50:40 2018 +0800

The example nginx config file like this

user  root;
master_process off;
daemon off;

events {
use epoll;
}

http {
    server {
        listen 80 http2;
    
        location = /callback {
            internal;
            content_by_lua_block {
                ngx.print("fallback called")
            }
        }

        location / {
            rewrite_by_lua_block {
                ngx.req.read_body()
                --ngx.sleep(0.01)
                ngx.status = 200
                local res = ngx.location.capture('/callback')
                ngx.header["Content-Type"] = "text/html"
                ngx.print(res.body)
                return ngx.exit(ngx.status)
            }
            root html;
        }
    }
}

Send a http2 request by nghttp tool

$ echo hello > bodydata
$ nghttp http://127.0.0.1 -d bodydata -v
[  0.000] Connected
[  0.000] send SETTINGS frame <length=12, flags=0x00, stream_id=0>
          (niv=2)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
[  0.000] send PRIORITY frame <length=5, flags=0x00, stream_id=3>
          (dep_stream_id=0, weight=201, exclusive=0)
[  0.000] send PRIORITY frame <length=5, flags=0x00, stream_id=5>
          (dep_stream_id=0, weight=101, exclusive=0)
[  0.000] send PRIORITY frame <length=5, flags=0x00, stream_id=7>
          (dep_stream_id=0, weight=1, exclusive=0)
[  0.000] send PRIORITY frame <length=5, flags=0x00, stream_id=9>
          (dep_stream_id=7, weight=1, exclusive=0)
[  0.000] send PRIORITY frame <length=5, flags=0x00, stream_id=11>
          (dep_stream_id=3, weight=1, exclusive=0)
[  0.000] send HEADERS frame <length=39, flags=0x24, stream_id=13>
          ; END_HEADERS | PRIORITY
          (padlen=0, dep_stream_id=11, weight=16, exclusive=0)
          ; Open new stream
          :method: POST
          :path: /
          :scheme: http
          :authority: 127.0.0.1
          accept: */*
          accept-encoding: gzip, deflate
          user-agent: nghttp2/1.25.0
          content-length: 6
[  0.000] send DATA frame <length=6, flags=0x01, stream_id=13>
          ; END_STREAM
[  0.000] recv SETTINGS frame <length=18, flags=0x00, stream_id=0>
          (niv=3)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):128]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65536]
          [SETTINGS_MAX_FRAME_SIZE(0x05):16777215]
[  0.000] recv WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=0>
          (window_size_increment=2147418112)
[  0.000] recv SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  0.000] recv WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=13>
          (window_size_increment=2147418111)
[  0.000] send SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
^C
$ 

then the nghttp sending request to nginx, then block forever, no response was sent by nginx .
if I call ngx.sleep before ngx.location.capture, it works well.

Blow is my analysis.

In Nginx, if one created a subrequest, the new subrequest should been posted to queue(posted_requests in ngx_http_request_t structure), then been executed by calling ngx_http_run_posted_requests.

While here when calling ngx.location.capture, a subrequest was created and posted to the queue, but the function ngx_http_run_posted_requests would not been called, and the conneciton hang up without any response.

In nginx, ngx_http_read_client_request_body function is used to read the request body nonblockly, defined as this

ngx_int_t
ngx_http_read_client_request_body(ngx_http_request_t *r,
    ngx_http_client_body_handler_pt post_handler)

the post_handler is called after read request body done. In http2 mode, if you created subrequest in the post_handler, the subrequest would become zombie and never be executed.

In lua-nginx-module, ngx_http_read_client_request_body called in four positions, Testing the four position with following testsuits in Test-Nginx , all failed with this problems.

# vim:set ft= ts=4 sw=4 et fdm=marker:
use Test::Nginx::Socket::Lua;

#worker_connections(1014);
#master_process_enabled(1);
log_level('warn');

repeat_each(2);

plan tests => repeat_each() * (blocks() * 4);

#no_diff();
no_long_string();
#master_on();
#workers(2);
run_tests();

__DATA__

=== TEST 1: read buffered body and then subrequest
--- config
    location /foo {
        echo -n foo;
    }
    location = /test {
        content_by_lua '
            ngx.req.read_body()
            local res = ngx.location.capture("/foo");
            ngx.say(ngx.var.request_body)
            ngx.say("sub: ", res.body)
        ';
    }
--- http2
--- request
POST /test
hello, world
--- response_body
hello, world
sub: foo
--- no_error_log
[error]
[alert]

=== TEST 2: read body with ngx.location.capture in rewrite phase
--- config
    location /proxy {
        proxy_pass http://127.0.0.1:$server_port/hi;
    }
    location /hi {
        echo_request_body;
    }
    location /echo_body {
        lua_need_request_body on;
        rewrite_by_lua '
            ngx.say(ngx.var.request_body or "nil")
            local res = ngx.location.capture(
                "/proxy",
                { method = ngx.HTTP_POST,
                  body = ngx.var.request_body })

            ngx.print(res.body)
        ';
    }
--- http2
--- request eval
"POST /echo_body
hello\x00\x01\x02
world\x03\x04\xff"
--- response_body eval
"hello\x00\x01\x02
world\x03\x04\xff
"

=== TEST 2: read body with ngx.location.capture in access
--- config
    location /proxy {
        proxy_pass http://127.0.0.1:$server_port/hi;
    }
    location /hi {
        echo_request_body;
    }
    location /echo_body {
        lua_need_request_body on;
        access_by_lua '
            ngx.say(ngx.var.request_body or "nil")
            local res = ngx.location.capture(
                "/proxy",
                { method = ngx.HTTP_POST,
                  body = ngx.var.request_body })

            ngx.print(res.body)
        ';
    }
--- http2
--- request eval
"POST /echo_body
hello\x00\x01\x02
world\x03\x04\xff"
--- response_body eval
"hello\x00\x01\x02
world\x03\x04\xff
"

=== TEST 4: read body with ngx.location.capture in content
--- config
    location /proxy {
        proxy_pass http://127.0.0.1:$server_port/hi;
    }
    location /hi {
        echo_request_body;
    }
    location /echo_body {
        lua_need_request_body on;
        content_by_lua '
            ngx.say(ngx.var.request_body or "nil")
            local res = ngx.location.capture(
                "/proxy",
                { method = ngx.HTTP_POST,
                  body = ngx.var.request_body })

            ngx.print(res.body)
        ';
    }
--- http2
--- request eval
"POST /echo_body
hello\x00\x01\x02
world\x03\x04\xff"
--- response_body eval
"hello\x00\x01\x02
world\x03\x04\xff
"

The problem is here, but I'm not sure it should be fixed in lua-nginx-module ,or nginx project?

[agentzh]

@qlees The ngx.location.capture* API is known to be broken inside an http2 request. And there is no plan to add such support ourselves (though patches welcome from the community). I've committed a patch to the git repository to make those API functions throw an exception inside an http2 request.

[crasyangel]

@qlees It would be OK by adding "ngx_post_event(r->connection->read, &ngx_posted_events);" before lua_yield

[qleein]

@crasyangel In fact, adding ngx.sleep(0.01) before ngx.location.capture could avoid this in my project. while it's not recomended....

[agentzh]

@qlees If yielding helps, then ngx.sleep(0) would be much better than ngx.sleep(0.01).