Skip to content

Commit

Permalink
updated the docs and done other releng work for the v0.07 release.
Browse files Browse the repository at this point in the history
  • Loading branch information
@agentzh
agentzh committed Dec 4, 2009
1 parent 1149cd5 commit fad343d
Show file tree
Hide file tree
Showing 2 changed files with 218 additions and 31 deletions.
148 changes: 128 additions & 20 deletions README
View file
Expand Up @@ -6,15 +6,21 @@ Name
installation instructions. installation instructions.


Version Version
This document describes chunkin-nginx-module v0.06 This document describes chunkin-nginx-module v0.07
(<http://github.com/agentzh/chunkin-nginx-module/downloads>) released on (<http://github.com/agentzh/chunkin-nginx-module/downloads>) released on
Nov 16, 2009. Dec 4, 2009.


Synopsis Synopsis
chunkin on; chunkin on;
location { ... } location { ... }
... ...


chunkin on;
location {
chunkin_keepalive on;
...
}

Description Description
This module adds HTTP 1.1 chunked This module adds HTTP 1.1 chunked
(<http://tools.ietf.org/html/rfc2616#section-3.6.1>) input support for (<http://tools.ietf.org/html/rfc2616#section-3.6.1>) input support for
Expand Down Expand Up @@ -44,17 +50,40 @@ Description
are followed partially. See Know Issues. are followed partially. See Know Issues.


WARNING: This module is in its very early phase in development and is WARNING: This module is in its very early phase in development and is
considered *highly experimental*. But you're encouraged to play and test considered *experimental*. But you're encouraged to play and test it in
it in your non-production environment and report any quirks that you your environment and report any quirks that you find.
find.


Efforts have been made to reduce data copying and dynamic memory This module is not supposed to be merged into the Nginx core because
I've used Ragel (<http://www.complang.org/ragel/>) to generate the
chunked encoding parser for joy :)

How it works
Nginx explicitly checks chunked "Transfer-Encoding" headers and absent
content length header in its very early phase. Well, as early as the
"ngx_http_process_request_header" function. So this module takes a
rather tricky approach. That is, use an output filter to intercept the
<code>411 Length Required</code> error page response issued by
ngx_http_process_request_header, fix things and finally issue an
internal redirect to the current location, thus starting from those
phases we all know and love, this time bypassing the horrible
"ngx_http_process_request_header" function.

In the "rewrite" phase of the newly created request, this module eagerly
reads in the chunked request body in a way similar to that of the
standard "ngx_http_read_client_request_body" function, but using its own
chunked parser generated by Ragel. The decoded request body will be put
into "r->request_body->bufs" and a corresponding "Content-Length" header
will be inserted into "r->headers_in".

Those modules using the standard "ngx_http_read_client_request_body"
function to read the request body will just work out of box because
"ngx_http_read_client_request_body" returns immediately when it sees
"r->request_body->bufs" already exists.

Special efforts have been made to reduce data copying and dynamic memory
allocation, thus unfortunately raising the risk of potential buffer allocation, thus unfortunately raising the risk of potential buffer
handling bugs caused by premature optimizations :P handling bugs caused by premature optimizations :P


This module is not supposed to be merged into the Nginx core because
I've used Ragel to generate the chunked encoding parser for joy :)

Directives Directives
chunkin chunkin
syntax: *chunkin on|off* syntax: *chunkin on|off*
Expand All @@ -65,14 +94,54 @@ Directives


Enables or disables this module's hooks. Enables or disables this module's hooks.


chunkin_keepalive
syntax: *chunkin_keepalive on|off*

default: *off*

context: *http, server, location, if*

Turns on or turns off HTTP 1.1 keep-alive and HTTP 1.1 pipelining
support.

Keep-alive without pipelining should be quite stable but pipelining
support is very preliminary, limited, and almost untested.

This directive was first introduced in the v0.07 release.

Technical note on the HTTP 1.1 pipeling support

The basic idea is to copy the bytes left by my chunked parser in
"r->request_body->buf" over into "r->header_in" so that nginx's
"ngx_http_set_keepalive" and "ngx_http_init_request" functions will pick
it up for the subsequent pipelined requests. When the request body is
small enough to be completely preread into the "r->header_in" buffer,
then no data copy is needed here -- just setting "r->header_in->pos"
correctly will suffice.

The only issue that remains is how to enlarge "r->header_in" when the
data left in "r->request_body->buf" is just too large to be hold in the
remaining room between "r->header_in->pos" and "r->header_in->end". For
now, this module will just give up and simply turn off "r->keepalive".

I know we can always use exactly the remaining room in "r->header_in" as
the buffer size when reading data from "c->recv", but's suboptimal when
the remaining room in "r->header_in" happens to be very small while
"r->request_body->buf" is quite large.

I haven't fully grokked all the details among "r->header_in",
"c->buffer", busy/free lists and those so-called "large header buffers".
Is there a clean and safe way to reallocate or extend the "r->header_in"
buffer?

Installation Installation
Grab the nginx source code from nginx.net (<http://nginx.net/>), for Grab the nginx source code from nginx.net (<http://nginx.net/>), for
example, the version 0.8.28 (see nginx compatibility), and then build example, the version 0.8.29 (see nginx compatibility), and then build
the source with this module: the source with this module:


$ wget 'http://sysoev.ru/nginx/nginx-0.8.28.tar.gz' $ wget 'http://sysoev.ru/nginx/nginx-0.8.29.tar.gz'
$ tar -xzvf nginx-0.8.28.tar.gz $ tar -xzvf nginx-0.8.29.tar.gz
$ cd nginx-0.8.28/ $ cd nginx-0.8.29/


# Here we assume you would install you nginx under /opt/nginx/. # Here we assume you would install you nginx under /opt/nginx/.
$ ./configure --prefix=/opt/nginx \ $ ./configure --prefix=/opt/nginx \
Expand All @@ -98,7 +167,7 @@ Installation
Compatibility Compatibility
The following versions of Nginx should work with this module: The following versions of Nginx should work with this module:


* 0.8.x (last tested version is 0.8.28) * 0.8.x (last tested version is 0.8.29)


* 0.7.x >= 0.7.21 (last tested version is 0.7.64) * 0.7.x >= 0.7.21 (last tested version is 0.7.64)


Expand All @@ -123,6 +192,44 @@ Source Repository
(<http://github.com/agentzh/chunkin-nginx-module>). (<http://github.com/agentzh/chunkin-nginx-module>).


ChangeLog ChangeLog
v0.07
* marked the disgarded 411 error page's output chain bufs as consumed
by setting "buf->pos = buf->last". (See this nginx-devel thread
(<http://nginx.org/pipermail/nginx-devel/2009-December/000025.html>)
for more details.)

* added the chunkin_keepalive directive which can enable HTTP 1.1
keep-alive and HTTP 1.1 pipelining, and defaults to "off".

* fixed the "alphtype" bug in the Ragel parser spec; which caused
rejection of non-ascii octets in the chunked data.

* added "Test::Nginx::Socket" to test our nginx module on the socket
level. Thanks J for his bug report.

* rewrote the bufs recycling part and preread-buf-to-rb-buf transition
part, also refactored the Ragel parser spec, thus eliminating lots
of serious bugs.

* provided better diagnostics in the error log message for "bad
chunked body" parsefails in the chunked parser. For example:

2009/12/02 17:35:52 [error] 32244#0: *1 bad chunked body (offset 7, near "4^M
hell <-- HERE o^M
0^M
^M
", marked by " <-- HERE ").
, client: 127.0.0.1, server: localhost, request: "POST /main
HTTP/1.1", host: "localhost"

* added some code to let the chunked parser handle special 0-size
chunks that are not the last chunk.

* fixed a connection leak bug regarding incorrect "r->main->count"
reference counter handling for nginx 0.8.11+ (well, the
"ngx_http_read_client_request_body" function in the nginx core also
has this issue, I'll report it later.)

v0.06 v0.06
* minor optimization: we won't traverse the output chain link if the * minor optimization: we won't traverse the output chain link if the
chain count is not large enough. chain count is not large enough.
Expand Down Expand Up @@ -158,9 +265,6 @@ Test Suite
enabled as well when building Nginx. enabled as well when building Nginx.


Known Issues Known Issues
* May not work properly if keep-alive is enabled in Nginx. Use it with
keep-alive at your own risk.

* May not work with certain 3rd party modules like the upload module * May not work with certain 3rd party modules like the upload module
(<http://www.grid.net.ru/nginx/upload.en.html>) because it (<http://www.grid.net.ru/nginx/upload.en.html>) because it
implements its own request body reading mechanism. implements its own request body reading mechanism.
Expand All @@ -171,15 +275,14 @@ Known Issues
* "client_body_in_file_only on" may *not* be obeyed for short contents * "client_body_in_file_only on" may *not* be obeyed for short contents
and fast network. and fast network.


* HTTP 1.1 pipelining may not fully work yet.

TODO TODO
* Fix the known issues. * Fix the known issues.


* Add support for trailers and chunked extensions as specified in the * Add support for trailers and chunked extensions as specified in the
RFC (<http://tools.ietf.org/html/rfc2616#section-3.6.1>) RFC (<http://tools.ietf.org/html/rfc2616#section-3.6.1>)


* Add tests for invalid chunked input by implementing
Test::Nginx::Socket in the test scaffold.

Getting involved Getting involved
You'll be very welcomed to submit patches to the author or just ask for You'll be very welcomed to submit patches to the author or just ask for
a commit bit to the source repository on GitHub. a commit bit to the source repository on GitHub.
Expand Down Expand Up @@ -243,6 +346,11 @@ See Also
(<http://agentzh.spaces.live.com/blog/cns!FF3A735632E41548!481.entry (<http://agentzh.spaces.live.com/blog/cns!FF3A735632E41548!481.entry
>) about this module's initial development. >) about this module's initial development.


* The thread discussing chunked input support on the nginx-devel
mailing list: "Chunked request body and HTTP header parser"
(<http://nginx.org/pipermail/nginx-devel/2009-December/000021.html>)
.

* The echo module for Nginx module's automated testing. * The echo module for Nginx module's automated testing.


* RFC 2616 - Chunked Transfer Coding * RFC 2616 - Chunked Transfer Coding
Expand Down
101 changes: 90 additions & 11 deletions doc/readme.wiki
View file
Expand Up @@ -6,14 +6,20 @@


= Version = = Version =


This document describes chunkin-nginx-module [http://github.com/agentzh/chunkin-nginx-module/downloads v0.06] released on Nov 16, 2009. This document describes chunkin-nginx-module [http://github.com/agentzh/chunkin-nginx-module/downloads v0.07] released on Dec 4, 2009.


= Synopsis = = Synopsis =


<geshi lang="nginx"> <geshi lang="nginx">
chunkin on; chunkin on;
location { ... } location { ... }
... ...

chunkin on;
location {
chunkin_keepalive on;
...
}
</geshi> </geshi>


= Description = = Description =
Expand All @@ -36,11 +42,25 @@ The core module's [[NginxHttpCoreModule#client_body_buffer_size|client_body_buff


The [[NginxHttpCoreModule#client_body_in_file_only|client_body_in_file_only]] and [[NginxHttpCoreModule#client_body_in_single_buffer|client_body_in_single_buffer]] settings are followed partially. See [[#Known Issues|Know Issues]]. The [[NginxHttpCoreModule#client_body_in_file_only|client_body_in_file_only]] and [[NginxHttpCoreModule#client_body_in_single_buffer|client_body_in_single_buffer]] settings are followed partially. See [[#Known Issues|Know Issues]].


WARNING: This module is in its very early phase in development and is considered ''highly experimental''. But you're encouraged to play and test it in your non-production environment and report any quirks that you find. WARNING: This module is in its very early phase in development and is considered ''experimental''. But you're encouraged to play and test it in your environment and report any quirks that you find.

This module is not supposed to be merged into the Nginx core because I've used [http://www.complang.org/ragel/ Ragel] to generate the chunked encoding parser for joy :)

== How it works ==

Nginx explicitly checks chunked <code>Transfer-Encoding</code> headers and absent content length header in its very
early phase. Well, as early as the <code>ngx_http_process_request_header</code>
function. So this module takes a rather tricky approach. That is, use an output filter to intercept the <code>411
Length Required</code> error page response issued by ngx_http_process_request_header,
fix things and finally issue an internal redirect to the current location,
thus starting from those phases we all know and love, this time
bypassing the horrible <code>ngx_http_process_request_header</code> function.

In the <code>rewrite</code> phase of the newly created request, this module eagerly reads in the chunked request body in a way similar to that of the standard <code>ngx_http_read_client_request_body</code> function, but using its own chunked parser generated by Ragel. The decoded request body will be put into <code>r->request_body->bufs</code> and a corresponding <code>Content-Length</code> header will be inserted into <code>r->headers_in</code>.


Efforts have been made to reduce data copying and dynamic memory allocation, thus unfortunately raising the risk of potential buffer handling bugs caused by premature optimizations :P Those modules using the standard <code>ngx_http_read_client_request_body</code> function to read the request body will just work out of box because <code>ngx_http_read_client_request_body</code> returns immediately when it sees <code>r->request_body->bufs</code> already exists.


This module is not supposed to be merged into the Nginx core because I've used Ragel to generate the chunked encoding parser for joy :) Special efforts have been made to reduce data copying and dynamic memory allocation, thus unfortunately raising the risk of potential buffer handling bugs caused by premature optimizations :P


= Directives = = Directives =


Expand All @@ -53,15 +73,52 @@ This module is not supposed to be merged into the Nginx core because I've used R


Enables or disables this module's hooks. Enables or disables this module's hooks.


== chunkin_keepalive ==
'''syntax:''' ''chunkin_keepalive on|off''

'''default:''' ''off''

'''context:''' ''http, server, location, if''

Turns on or turns off HTTP 1.1 keep-alive and HTTP 1.1 pipelining support.

Keep-alive without pipelining should be quite stable but pipelining support is very preliminary, limited, and almost untested.

This directive was first introduced in the [[#v0.07|v0.07 release]].

'''Technical note on the HTTP 1.1 pipeling support'''

The basic idea is to copy the bytes left by my chunked parser in
<code>r->request_body->buf</code> over into <code>r->header_in</code> so that nginx's
<code>ngx_http_set_keepalive</code> and <code>ngx_http_init_request</code> functions will pick
it up for the subsequent pipelined requests. When the request body is
small enough to be completely preread into the <code>r->header_in</code> buffer,
then no data copy is needed here -- just setting <code>r->header_in->pos</code>
correctly will suffice.

The only issue that remains is how to enlarge <code>r->header_in</code> when the
data left in <code>r->request_body->buf</code> is just too large to be hold in the
remaining room between <code>r->header_in->pos</code> and <code>r->header_in->end</code>. For
now, this module will just give up and simply turn off <code>r->keepalive</code>.

I know we can always use exactly the remaining room in <code>r->header_in</code> as
the buffer size when reading data from <code>c->recv</code>, but's suboptimal when
the remaining room in <code>r->header_in</code> happens to be very small while
<code>r->request_body->buf</code> is quite large.

I haven't fully grokked all the details among <code>r->header_in</code>, <code>c->buffer</code>,
busy/free lists and those so-called "large header buffers". Is there a
clean and safe way to reallocate or extend the <code>r->header_in</code> buffer?

= Installation = = Installation =


Grab the nginx source code from [http://nginx.net/ nginx.net], for example, Grab the nginx source code from [http://nginx.net/ nginx.net], for example,
the version 0.8.28 (see [[#Compatibility|nginx compatibility]]), and then build the source with this module: the version 0.8.29 (see [[#Compatibility|nginx compatibility]]), and then build the source with this module:


<geshi lang="bash"> <geshi lang="bash">
$ wget 'http://sysoev.ru/nginx/nginx-0.8.28.tar.gz' $ wget 'http://sysoev.ru/nginx/nginx-0.8.29.tar.gz'
$ tar -xzvf nginx-0.8.28.tar.gz $ tar -xzvf nginx-0.8.29.tar.gz
$ cd nginx-0.8.28/ $ cd nginx-0.8.29/


# Here we assume you would install you nginx under /opt/nginx/. # Here we assume you would install you nginx under /opt/nginx/.
$ ./configure --prefix=/opt/nginx \ $ ./configure --prefix=/opt/nginx \
Expand All @@ -87,7 +144,7 @@ the following command from the root of the chunkin module's source tree:


The following versions of Nginx should work with this module: The following versions of Nginx should work with this module:


* '''0.8.x''' (last tested version is 0.8.28) * '''0.8.x''' (last tested version is 0.8.29)
* '''0.7.x >= 0.7.21''' (last tested version is 0.7.64) * '''0.7.x >= 0.7.21''' (last tested version is 0.7.64)
Earlier versions of Nginx like 0.6.x and 0.5.x will ''not'' work. Earlier versions of Nginx like 0.6.x and 0.5.x will ''not'' work.
Expand All @@ -107,6 +164,28 @@ Available on github at [http://github.com/agentzh/chunkin-nginx-module agentzh/c


= ChangeLog = = ChangeLog =


== v0.07 ==

* marked the disgarded 411 error page's output chain bufs as consumed by setting <code>buf->pos = buf->last</code>. (See [http://nginx.org/pipermail/nginx-devel/2009-December/000025.html this nginx-devel thread] for more details.)
* added the [[#chunkin_keepalive|chunkin_keepalive]] directive which can enable HTTP 1.1 keep-alive and HTTP 1.1 pipelining, and defaults to <code>off</code>.
* fixed the <code>alphtype</code> bug in the Ragel parser spec; which caused rejection of non-ascii octets in the chunked data.
* added <code>Test::Nginx::Socket</code> to test our nginx module on the socket level. Thanks J for his bug report.
* rewrote the bufs recycling part and preread-buf-to-rb-buf transition part, also refactored the Ragel parser spec, thus eliminating lots of serious bugs.
* provided better diagnostics in the error log message for "bad chunked body" parsefails in the chunked parser. For example:
<geshi lang="log">
2009/12/02 17:35:52 [error] 32244#0: *1 bad chunked body (offset 7, near "4^M
hell <-- HERE o^M
0^M
^M
", marked by " <-- HERE ").
, client: 127.0.0.1, server: localhost, request: "POST /main
HTTP/1.1", host: "localhost"
</geshi>

* added some code to let the chunked parser handle special 0-size chunks that are not the last chunk.
* fixed a connection leak bug regarding incorrect <code>r->main->count</code> reference counter handling for nginx 0.8.11+ (well, the <code>ngx_http_read_client_request_body</code> function in the nginx core also has this issue, I'll report it later.)
== v0.06 == == v0.06 ==
* minor optimization: we won't traverse the output chain link if the chain count is not large enough. * minor optimization: we won't traverse the output chain link if the chain count is not large enough.
Expand All @@ -132,16 +211,15 @@ Some parts of the test suite requires modules [[NginxHttpProxyModule|proxy]] and


= Known Issues = = Known Issues =


* May not work properly if keep-alive is enabled in Nginx. Use it with keep-alive at your own risk.
* May not work with certain 3rd party modules like the [http://www.grid.net.ru/nginx/upload.en.html upload module] because it implements its own request body reading mechanism. * May not work with certain 3rd party modules like the [http://www.grid.net.ru/nginx/upload.en.html upload module] because it implements its own request body reading mechanism.
* "client_body_in_single_buffer on" may *not* be obeyed for short contents and fast network. * "client_body_in_single_buffer on" may *not* be obeyed for short contents and fast network.
* "client_body_in_file_only on" may *not* be obeyed for short contents and fast network. * "client_body_in_file_only on" may *not* be obeyed for short contents and fast network.
* HTTP 1.1 pipelining may not fully work yet.
= TODO = = TODO =


* Fix the [[#Known Issues|known issues]]. * Fix the [[#Known Issues|known issues]].
* Add support for trailers and chunked extensions as specified in the [http://tools.ietf.org/html/rfc2616#section-3.6.1 RFC] * Add support for trailers and chunked extensions as specified in the [http://tools.ietf.org/html/rfc2616#section-3.6.1 RFC]
* Add tests for invalid chunked input by implementing Test::Nginx::Socket in the test scaffold.
= Getting involved = = Getting involved =


Expand Down Expand Up @@ -188,6 +266,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
* The original thread on the Nginx mailing list that inspires this module's development: [http://forum.nginx.org/read.php?2,4453,20543 "'Content-Length' header for POSTs"]. * The original thread on the Nginx mailing list that inspires this module's development: [http://forum.nginx.org/read.php?2,4453,20543 "'Content-Length' header for POSTs"].
* The orginal announcement thread on the Nginx mailing list: [http://forum.nginx.org/read.php?2,22967 "The chunkin module: Experimental chunked input support for Nginx"]. * The orginal announcement thread on the Nginx mailing list: [http://forum.nginx.org/read.php?2,22967 "The chunkin module: Experimental chunked input support for Nginx"].
* The original [http://agentzh.spaces.live.com/blog/cns!FF3A735632E41548!481.entry blog post] about this module's initial development. * The original [http://agentzh.spaces.live.com/blog/cns!FF3A735632E41548!481.entry blog post] about this module's initial development.
* The thread discussing chunked input support on the nginx-devel mailing list: [http://nginx.org/pipermail/nginx-devel/2009-December/000021.html "Chunked request body and HTTP header parser"].
* The [[NginxHttpEchoModule|echo module]] for Nginx module's automated testing. * The [[NginxHttpEchoModule|echo module]] for Nginx module's automated testing.
* [http://tools.ietf.org/html/rfc2616#section-3.6.1 RFC 2616 - Chunked Transfer Coding]. * [http://tools.ietf.org/html/rfc2616#section-3.6.1 RFC 2616 - Chunked Transfer Coding].

0 comments on commit fad343d

Please sign in to comment.