Nginx Improved response body filter, issue #56 might be fixed #67

Merged
merged 1 commit into from Apr 9, 2013

Conversation

Projects
None yet
2 participants
@chaizhenhua
Contributor

chaizhenhua commented Apr 9, 2013

No description provided.

@brenosilva

This comment has been minimized.

Show comment
Hide comment
@brenosilva

brenosilva Apr 9, 2013

Contributor

Hello chaizhenhua,

NGX_HTTP_SSI_BUFFERED means we will process the response time to time without need complete buffered ?

Thanks

Contributor

brenosilva commented Apr 9, 2013

Hello chaizhenhua,

NGX_HTTP_SSI_BUFFERED means we will process the response time to time without need complete buffered ?

Thanks

@chaizhenhua

This comment has been minimized.

Show comment
Hide comment
@chaizhenhua

chaizhenhua Apr 9, 2013

Contributor

I think this flags tells nginx not to process the write eventimmediately(when fd is ready) . because there is nothing to send to client before all response body was processed, so we should delay the fd event.

Contributor

chaizhenhua commented Apr 9, 2013

I think this flags tells nginx not to process the write eventimmediately(when fd is ready) . because there is nothing to send to client before all response body was processed, so we should delay the fd event.

brenosilva added a commit that referenced this pull request Apr 9, 2013

Merge pull request #67 from chaizhenhua/remotes/trunk
Nginx Improved response body filter, issue #56 might be fixed

@brenosilva brenosilva merged commit 1919c3a into SpiderLabs:remotes/trunk Apr 9, 2013

@brenosilva

This comment has been minimized.

Show comment
Hide comment
@brenosilva

brenosilva Apr 9, 2013

Contributor

Applied Thanks!

Contributor

brenosilva commented Apr 9, 2013

Applied Thanks!

@brenosilva

This comment has been minimized.

Show comment
Hide comment
@brenosilva

brenosilva May 11, 2013

Hello man,

Looks like we still have the CPU issue. I inserted some debug messages and many requests are returning in ngx_http_modsecurity_body_filter() here:

if (in == NULL) {
    /* waiting for more buffer */
    r->buffered |= NGX_HTTP_SSI_BUFFERED;
    return NGX_AGAIN;
}

You can trigger is sending ab -n 1000 -c 10 http://your_ip/index.html (a html page with images).

Could this be the cause of 100% CPU issue ?

Thanks

Breno

Hello man,

Looks like we still have the CPU issue. I inserted some debug messages and many requests are returning in ngx_http_modsecurity_body_filter() here:

if (in == NULL) {
    /* waiting for more buffer */
    r->buffered |= NGX_HTTP_SSI_BUFFERED;
    return NGX_AGAIN;
}

You can trigger is sending ab -n 1000 -c 10 http://your_ip/index.html (a html page with images).

Could this be the cause of 100% CPU issue ?

Thanks

Breno

This comment has been minimized.

Show comment
Hide comment
@brenosilva

brenosilva May 11, 2013

Also try set worker_process 1

Looks like if you insert in the beginning of ngx_http_modsecurity_body_filter():

return ngx_http_next_body_filter(r, in);

The problem disappear. So looks like an issue in ngx_http_modsecurity_body_filter

Also try set worker_process 1

Looks like if you insert in the beginning of ngx_http_modsecurity_body_filter():

return ngx_http_next_body_filter(r, in);

The problem disappear. So looks like an issue in ngx_http_modsecurity_body_filter

This comment has been minimized.

Show comment
Hide comment
@brenosilva

brenosilva May 11, 2013

I inserted return NGX_AGAIN;

after if( in == NULL) { } return NGX_AGAIN; and the problem disappeared too. So looks like in == NULL is not the problem

Could be the move_chain_to_brigade() ?

I inserted return NGX_AGAIN;

after if( in == NULL) { } return NGX_AGAIN; and the problem disappeared too. So looks like in == NULL is not the problem

Could be the move_chain_to_brigade() ?

This comment has been minimized.

Show comment
Hide comment
@brenosilva

brenosilva May 11, 2013

Humm yes.. looks like the issue is calling move_chain_to_brigade()

I changed from:

rc = move_chain_to_brigade(in, ctx->brigade, r->pool, 0);

to

rc = move_chain_to_brigade(in, ctx->brigade, r->pool, 1);

Any idea how we can fix it ?

Humm yes.. looks like the issue is calling move_chain_to_brigade()

I changed from:

rc = move_chain_to_brigade(in, ctx->brigade, r->pool, 0);

to

rc = move_chain_to_brigade(in, ctx->brigade, r->pool, 1);

Any idea how we can fix it ?

This comment has been minimized.

Show comment
Hide comment
@brenosilva

brenosilva May 11, 2013

When the issue is triggered... the modsecurity debug.log stop here:

[11/May/2013:10:36:54 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Transformation completed in 1 usec.
[11/May/2013:10:36:54 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Executing operator "rx" with param "index" against REQUEST_URI.
[11/May/2013:10:36:54 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][9] Target value: "/catalog/index.html"
[11/May/2013:10:36:54 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Operator completed in 10 usec.
[11/May/2013:10:36:54 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][2] Warning. Pattern match "index" at REQUEST_URI. [file "/usr/local/nginx/conf/modsecurity.conf"] [line "236"] [id "10"]
[11/May/2013:10:36:54 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Rule returned 1.
[11/May/2013:10:36:54 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][9] Match -> mode NEXT_RULE.
[11/May/2013:10:36:54 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Hook insert_filter: Adding output filter (r a1239c0).

So ... it does not follow the execution like a normal transaction:

[11/May/2013:10:35:07 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Transformation completed in 1 usec.
[11/May/2013:10:35:07 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Executing operator "rx" with param "index" against REQUEST_URI.
[11/May/2013:10:35:07 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][9] Target value: "/catalog/index.html"
[11/May/2013:10:35:07 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Operator completed in 7 usec.
[11/May/2013:10:35:07 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][2] Warning. Pattern match "index" at REQUEST_URI. [file "/usr/local/nginx/conf/modsecurity.conf"] [line "236"] [id "10"]
[11/May/2013:10:35:07 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Rule returned 1.
[11/May/2013:10:35:07 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][9] Match -> mode NEXT_RULE.
[11/May/2013:10:35:07 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Hook insert_filter: Adding output filter (r a1239c0).
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Receiving output (f a0a8430, r a0a77c0).
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][4] Starting phase RESPONSE_HEADERS.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] This phase consists of 0 rule(s).
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Content Injection: Not enabled.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 3773 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 4096 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 4096 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 4096 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 4096 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 4096 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 4096 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 3594 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type EOS contains 0 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][4] Output filter: Completed receiving response body (buffered full - 31943 bytes).

When the issue is triggered... the modsecurity debug.log stop here:

[11/May/2013:10:36:54 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Transformation completed in 1 usec.
[11/May/2013:10:36:54 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Executing operator "rx" with param "index" against REQUEST_URI.
[11/May/2013:10:36:54 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][9] Target value: "/catalog/index.html"
[11/May/2013:10:36:54 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Operator completed in 10 usec.
[11/May/2013:10:36:54 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][2] Warning. Pattern match "index" at REQUEST_URI. [file "/usr/local/nginx/conf/modsecurity.conf"] [line "236"] [id "10"]
[11/May/2013:10:36:54 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Rule returned 1.
[11/May/2013:10:36:54 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][9] Match -> mode NEXT_RULE.
[11/May/2013:10:36:54 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Hook insert_filter: Adding output filter (r a1239c0).

So ... it does not follow the execution like a normal transaction:

[11/May/2013:10:35:07 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Transformation completed in 1 usec.
[11/May/2013:10:35:07 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Executing operator "rx" with param "index" against REQUEST_URI.
[11/May/2013:10:35:07 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][9] Target value: "/catalog/index.html"
[11/May/2013:10:35:07 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Operator completed in 7 usec.
[11/May/2013:10:35:07 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][2] Warning. Pattern match "index" at REQUEST_URI. [file "/usr/local/nginx/conf/modsecurity.conf"] [line "236"] [id "10"]
[11/May/2013:10:35:07 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Rule returned 1.
[11/May/2013:10:35:07 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][9] Match -> mode NEXT_RULE.
[11/May/2013:10:35:07 --0700] [/sid#a04f618][rid#a1239c0][/catalog/index.html][4] Hook insert_filter: Adding output filter (r a1239c0).
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Receiving output (f a0a8430, r a0a77c0).
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][4] Starting phase RESPONSE_HEADERS.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] This phase consists of 0 rule(s).
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Content Injection: Not enabled.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 3773 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 4096 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 4096 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 4096 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 4096 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 4096 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 4096 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 3594 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][9] Output filter: Bucket type EOS contains 0 bytes.
[11/May/2013:10:35:08 --0700] [/sid#a04f618][rid#a0a77c0][/catalog/index.html][4] Output filter: Completed receiving response body (buffered full - 31943 bytes).

This comment has been minimized.

Show comment
Hide comment
@chaizhenhua

chaizhenhua May 13, 2013

Owner

@brenosilva I will send the patch as soon as possiable. i haven't finish the patch yet.

Owner

chaizhenhua replied May 13, 2013

@brenosilva I will send the patch as soon as possiable. i haven't finish the patch yet.

This comment has been minimized.

Show comment
Hide comment
@brenosilva

brenosilva May 13, 2013

Sounds good man.

I still didn't detect the reason of memory leaks.. do you have any idea ?

Sounds good man.

I still didn't detect the reason of memory leaks.. do you have any idea ?

This comment has been minimized.

Show comment
Hide comment
@brenosilva

brenosilva May 13, 2013

I'm trying to create a new ngx_buf_t to send to brigade instead of chain->buf in move_chain_to_brigade.. to see if memory leaks disappears:

            buf = ngx_palloc(pool, sizeof(ngx_buf_t));
            if (buf == NULL) {
                    return NGX_ERROR;
            }

            len = chain->buf->end - chain->buf->start;
            buf->start = ngx_palloc(pool, len);
            ngx_memcpy(buf->start, chain->buf->start, len);

            buf->pos = buf->last = buf->start;
            buf->end = buf->start + len;
            buf->last_buf = chain->buf->last_buf;
            buf->last_in_chain = chain->buf->last_in_chain;

            e = ngx_buf_to_apr_bucket(buf, bb->p, bb->bucket_alloc);
            if (e == NULL) {
                    return NGX_ERROR;
            }

However i'm seeing in the modsecurity log:

[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type EOS contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][4] Output filter: Completed receiving response body (buffered full - 0 bytes).

I think i 'm missing some detail
Any idea ?

I'm trying to create a new ngx_buf_t to send to brigade instead of chain->buf in move_chain_to_brigade.. to see if memory leaks disappears:

            buf = ngx_palloc(pool, sizeof(ngx_buf_t));
            if (buf == NULL) {
                    return NGX_ERROR;
            }

            len = chain->buf->end - chain->buf->start;
            buf->start = ngx_palloc(pool, len);
            ngx_memcpy(buf->start, chain->buf->start, len);

            buf->pos = buf->last = buf->start;
            buf->end = buf->start + len;
            buf->last_buf = chain->buf->last_buf;
            buf->last_in_chain = chain->buf->last_in_chain;

            e = ngx_buf_to_apr_bucket(buf, bb->p, bb->bucket_alloc);
            if (e == NULL) {
                    return NGX_ERROR;
            }

However i'm seeing in the modsecurity log:

[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type NGINX contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][9] Output filter: Bucket type EOS contains 0 bytes.
[12/May/2013:23:26:17 --0700] [/sid#9af2618][rid#9b3b780][/catalog/index2.html][4] Output filter: Completed receiving response body (buffered full - 0 bytes).

I think i 'm missing some detail
Any idea ?

This comment has been minimized.

Show comment
Hide comment
Owner

chaizhenhua replied May 13, 2013

see this SpiderLabs#96

This comment has been minimized.

Show comment
Hide comment
@chaizhenhua

chaizhenhua May 13, 2013

Owner

buf->pos = buf->last = buf->start; this line should be buf->last = buf->end;

Owner

chaizhenhua replied May 13, 2013

buf->pos = buf->last = buf->start; this line should be buf->last = buf->end;

zimmerle added a commit that referenced this pull request Oct 11, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment