Permalink
Browse files

updated the docs and done other releng work for the v0.07 release.

  • Loading branch information...
1 parent 1149cd5 commit fad343da8b1c0639570f4ea5fcf90ec4ba6d6afc @agentzh committed Dec 4, 2009
Showing with 218 additions and 31 deletions.
  1. +128 −20 README
  2. +90 −11 doc/readme.wiki
View
148 README
@@ -6,15 +6,21 @@ Name
installation instructions.
Version
- This document describes chunkin-nginx-module v0.06
+ This document describes chunkin-nginx-module v0.07
(<http://github.com/agentzh/chunkin-nginx-module/downloads>) released on
- Nov 16, 2009.
+ Dec 4, 2009.
Synopsis
chunkin on;
location { ... }
...
+ chunkin on;
+ location {
+ chunkin_keepalive on;
+ ...
+ }
+
Description
This module adds HTTP 1.1 chunked
(<http://tools.ietf.org/html/rfc2616#section-3.6.1>) input support for
@@ -44,17 +50,40 @@ Description
are followed partially. See Know Issues.
WARNING: This module is in its very early phase in development and is
- considered *highly experimental*. But you're encouraged to play and test
- it in your non-production environment and report any quirks that you
- find.
+ considered *experimental*. But you're encouraged to play and test it in
+ your environment and report any quirks that you find.
- Efforts have been made to reduce data copying and dynamic memory
+ This module is not supposed to be merged into the Nginx core because
+ I've used Ragel (<http://www.complang.org/ragel/>) to generate the
+ chunked encoding parser for joy :)
+
+ How it works
+ Nginx explicitly checks chunked "Transfer-Encoding" headers and absent
+ content length header in its very early phase. Well, as early as the
+ "ngx_http_process_request_header" function. So this module takes a
+ rather tricky approach. That is, use an output filter to intercept the
+ <code>411 Length Required</code> error page response issued by
+ ngx_http_process_request_header, fix things and finally issue an
+ internal redirect to the current location, thus starting from those
+ phases we all know and love, this time bypassing the horrible
+ "ngx_http_process_request_header" function.
+
+ In the "rewrite" phase of the newly created request, this module eagerly
+ reads in the chunked request body in a way similar to that of the
+ standard "ngx_http_read_client_request_body" function, but using its own
+ chunked parser generated by Ragel. The decoded request body will be put
+ into "r->request_body->bufs" and a corresponding "Content-Length" header
+ will be inserted into "r->headers_in".
+
+ Those modules using the standard "ngx_http_read_client_request_body"
+ function to read the request body will just work out of box because
+ "ngx_http_read_client_request_body" returns immediately when it sees
+ "r->request_body->bufs" already exists.
+
+ Special efforts have been made to reduce data copying and dynamic memory
allocation, thus unfortunately raising the risk of potential buffer
handling bugs caused by premature optimizations :P
- This module is not supposed to be merged into the Nginx core because
- I've used Ragel to generate the chunked encoding parser for joy :)
-
Directives
chunkin
syntax: *chunkin on|off*
@@ -65,14 +94,54 @@ Directives
Enables or disables this module's hooks.
+ chunkin_keepalive
+ syntax: *chunkin_keepalive on|off*
+
+ default: *off*
+
+ context: *http, server, location, if*
+
+ Turns on or turns off HTTP 1.1 keep-alive and HTTP 1.1 pipelining
+ support.
+
+ Keep-alive without pipelining should be quite stable but pipelining
+ support is very preliminary, limited, and almost untested.
+
+ This directive was first introduced in the v0.07 release.
+
+ Technical note on the HTTP 1.1 pipeling support
+
+ The basic idea is to copy the bytes left by my chunked parser in
+ "r->request_body->buf" over into "r->header_in" so that nginx's
+ "ngx_http_set_keepalive" and "ngx_http_init_request" functions will pick
+ it up for the subsequent pipelined requests. When the request body is
+ small enough to be completely preread into the "r->header_in" buffer,
+ then no data copy is needed here -- just setting "r->header_in->pos"
+ correctly will suffice.
+
+ The only issue that remains is how to enlarge "r->header_in" when the
+ data left in "r->request_body->buf" is just too large to be hold in the
+ remaining room between "r->header_in->pos" and "r->header_in->end". For
+ now, this module will just give up and simply turn off "r->keepalive".
+
+ I know we can always use exactly the remaining room in "r->header_in" as
+ the buffer size when reading data from "c->recv", but's suboptimal when
+ the remaining room in "r->header_in" happens to be very small while
+ "r->request_body->buf" is quite large.
+
+ I haven't fully grokked all the details among "r->header_in",
+ "c->buffer", busy/free lists and those so-called "large header buffers".
+ Is there a clean and safe way to reallocate or extend the "r->header_in"
+ buffer?
+
Installation
Grab the nginx source code from nginx.net (<http://nginx.net/>), for
- example, the version 0.8.28 (see nginx compatibility), and then build
+ example, the version 0.8.29 (see nginx compatibility), and then build
the source with this module:
- $ wget 'http://sysoev.ru/nginx/nginx-0.8.28.tar.gz'
- $ tar -xzvf nginx-0.8.28.tar.gz
- $ cd nginx-0.8.28/
+ $ wget 'http://sysoev.ru/nginx/nginx-0.8.29.tar.gz'
+ $ tar -xzvf nginx-0.8.29.tar.gz
+ $ cd nginx-0.8.29/
# Here we assume you would install you nginx under /opt/nginx/.
$ ./configure --prefix=/opt/nginx \
@@ -98,7 +167,7 @@ Installation
Compatibility
The following versions of Nginx should work with this module:
- * 0.8.x (last tested version is 0.8.28)
+ * 0.8.x (last tested version is 0.8.29)
* 0.7.x >= 0.7.21 (last tested version is 0.7.64)
@@ -123,6 +192,44 @@ Source Repository
(<http://github.com/agentzh/chunkin-nginx-module>).
ChangeLog
+ v0.07
+ * marked the disgarded 411 error page's output chain bufs as consumed
+ by setting "buf->pos = buf->last". (See this nginx-devel thread
+ (<http://nginx.org/pipermail/nginx-devel/2009-December/000025.html>)
+ for more details.)
+
+ * added the chunkin_keepalive directive which can enable HTTP 1.1
+ keep-alive and HTTP 1.1 pipelining, and defaults to "off".
+
+ * fixed the "alphtype" bug in the Ragel parser spec; which caused
+ rejection of non-ascii octets in the chunked data.
+
+ * added "Test::Nginx::Socket" to test our nginx module on the socket
+ level. Thanks J for his bug report.
+
+ * rewrote the bufs recycling part and preread-buf-to-rb-buf transition
+ part, also refactored the Ragel parser spec, thus eliminating lots
+ of serious bugs.
+
+ * provided better diagnostics in the error log message for "bad
+ chunked body" parsefails in the chunked parser. For example:
+
+ 2009/12/02 17:35:52 [error] 32244#0: *1 bad chunked body (offset 7, near "4^M
+ hell <-- HERE o^M
+ 0^M
+ ^M
+ ", marked by " <-- HERE ").
+ , client: 127.0.0.1, server: localhost, request: "POST /main
+ HTTP/1.1", host: "localhost"
+
+ * added some code to let the chunked parser handle special 0-size
+ chunks that are not the last chunk.
+
+ * fixed a connection leak bug regarding incorrect "r->main->count"
+ reference counter handling for nginx 0.8.11+ (well, the
+ "ngx_http_read_client_request_body" function in the nginx core also
+ has this issue, I'll report it later.)
+
v0.06
* minor optimization: we won't traverse the output chain link if the
chain count is not large enough.
@@ -158,9 +265,6 @@ Test Suite
enabled as well when building Nginx.
Known Issues
- * May not work properly if keep-alive is enabled in Nginx. Use it with
- keep-alive at your own risk.
-
* May not work with certain 3rd party modules like the upload module
(<http://www.grid.net.ru/nginx/upload.en.html>) because it
implements its own request body reading mechanism.
@@ -171,15 +275,14 @@ Known Issues
* "client_body_in_file_only on" may *not* be obeyed for short contents
and fast network.
+ * HTTP 1.1 pipelining may not fully work yet.
+
TODO
* Fix the known issues.
* Add support for trailers and chunked extensions as specified in the
RFC (<http://tools.ietf.org/html/rfc2616#section-3.6.1>)
- * Add tests for invalid chunked input by implementing
- Test::Nginx::Socket in the test scaffold.
-
Getting involved
You'll be very welcomed to submit patches to the author or just ask for
a commit bit to the source repository on GitHub.
@@ -243,6 +346,11 @@ See Also
(<http://agentzh.spaces.live.com/blog/cns!FF3A735632E41548!481.entry
>) about this module's initial development.
+ * The thread discussing chunked input support on the nginx-devel
+ mailing list: "Chunked request body and HTTP header parser"
+ (<http://nginx.org/pipermail/nginx-devel/2009-December/000021.html>)
+ .
+
* The echo module for Nginx module's automated testing.
* RFC 2616 - Chunked Transfer Coding
View
@@ -6,14 +6,20 @@
= Version =
-This document describes chunkin-nginx-module [http://github.com/agentzh/chunkin-nginx-module/downloads v0.06] released on Nov 16, 2009.
+This document describes chunkin-nginx-module [http://github.com/agentzh/chunkin-nginx-module/downloads v0.07] released on Dec 4, 2009.
= Synopsis =
<geshi lang="nginx">
chunkin on;
location { ... }
...
+
+ chunkin on;
+ location {
+ chunkin_keepalive on;
+ ...
+ }
</geshi>
= Description =
@@ -36,11 +42,25 @@ The core module's [[NginxHttpCoreModule#client_body_buffer_size|client_body_buff
The [[NginxHttpCoreModule#client_body_in_file_only|client_body_in_file_only]] and [[NginxHttpCoreModule#client_body_in_single_buffer|client_body_in_single_buffer]] settings are followed partially. See [[#Known Issues|Know Issues]].
-WARNING: This module is in its very early phase in development and is considered ''highly experimental''. But you're encouraged to play and test it in your non-production environment and report any quirks that you find.
+WARNING: This module is in its very early phase in development and is considered ''experimental''. But you're encouraged to play and test it in your environment and report any quirks that you find.
+
+This module is not supposed to be merged into the Nginx core because I've used [http://www.complang.org/ragel/ Ragel] to generate the chunked encoding parser for joy :)
+
+== How it works ==
+
+Nginx explicitly checks chunked <code>Transfer-Encoding</code> headers and absent content length header in its very
+early phase. Well, as early as the <code>ngx_http_process_request_header</code>
+function. So this module takes a rather tricky approach. That is, use an output filter to intercept the <code>411
+Length Required</code> error page response issued by ngx_http_process_request_header,
+fix things and finally issue an internal redirect to the current location,
+thus starting from those phases we all know and love, this time
+bypassing the horrible <code>ngx_http_process_request_header</code> function.
+
+In the <code>rewrite</code> phase of the newly created request, this module eagerly reads in the chunked request body in a way similar to that of the standard <code>ngx_http_read_client_request_body</code> function, but using its own chunked parser generated by Ragel. The decoded request body will be put into <code>r->request_body->bufs</code> and a corresponding <code>Content-Length</code> header will be inserted into <code>r->headers_in</code>.
-Efforts have been made to reduce data copying and dynamic memory allocation, thus unfortunately raising the risk of potential buffer handling bugs caused by premature optimizations :P
+Those modules using the standard <code>ngx_http_read_client_request_body</code> function to read the request body will just work out of box because <code>ngx_http_read_client_request_body</code> returns immediately when it sees <code>r->request_body->bufs</code> already exists.
-This module is not supposed to be merged into the Nginx core because I've used Ragel to generate the chunked encoding parser for joy :)
+Special efforts have been made to reduce data copying and dynamic memory allocation, thus unfortunately raising the risk of potential buffer handling bugs caused by premature optimizations :P
= Directives =
@@ -53,15 +73,52 @@ This module is not supposed to be merged into the Nginx core because I've used R
Enables or disables this module's hooks.
+== chunkin_keepalive ==
+'''syntax:''' ''chunkin_keepalive on|off''
+
+'''default:''' ''off''
+
+'''context:''' ''http, server, location, if''
+
+Turns on or turns off HTTP 1.1 keep-alive and HTTP 1.1 pipelining support.
+
+Keep-alive without pipelining should be quite stable but pipelining support is very preliminary, limited, and almost untested.
+
+This directive was first introduced in the [[#v0.07|v0.07 release]].
+
+'''Technical note on the HTTP 1.1 pipeling support'''
+
+The basic idea is to copy the bytes left by my chunked parser in
+<code>r->request_body->buf</code> over into <code>r->header_in</code> so that nginx's
+<code>ngx_http_set_keepalive</code> and <code>ngx_http_init_request</code> functions will pick
+it up for the subsequent pipelined requests. When the request body is
+small enough to be completely preread into the <code>r->header_in</code> buffer,
+then no data copy is needed here -- just setting <code>r->header_in->pos</code>
+correctly will suffice.
+
+The only issue that remains is how to enlarge <code>r->header_in</code> when the
+data left in <code>r->request_body->buf</code> is just too large to be hold in the
+remaining room between <code>r->header_in->pos</code> and <code>r->header_in->end</code>. For
+now, this module will just give up and simply turn off <code>r->keepalive</code>.
+
+I know we can always use exactly the remaining room in <code>r->header_in</code> as
+the buffer size when reading data from <code>c->recv</code>, but's suboptimal when
+the remaining room in <code>r->header_in</code> happens to be very small while
+<code>r->request_body->buf</code> is quite large.
+
+I haven't fully grokked all the details among <code>r->header_in</code>, <code>c->buffer</code>,
+busy/free lists and those so-called "large header buffers". Is there a
+clean and safe way to reallocate or extend the <code>r->header_in</code> buffer?
+
= Installation =
Grab the nginx source code from [http://nginx.net/ nginx.net], for example,
-the version 0.8.28 (see [[#Compatibility|nginx compatibility]]), and then build the source with this module:
+the version 0.8.29 (see [[#Compatibility|nginx compatibility]]), and then build the source with this module:
<geshi lang="bash">
- $ wget 'http://sysoev.ru/nginx/nginx-0.8.28.tar.gz'
- $ tar -xzvf nginx-0.8.28.tar.gz
- $ cd nginx-0.8.28/
+ $ wget 'http://sysoev.ru/nginx/nginx-0.8.29.tar.gz'
+ $ tar -xzvf nginx-0.8.29.tar.gz
+ $ cd nginx-0.8.29/
# Here we assume you would install you nginx under /opt/nginx/.
$ ./configure --prefix=/opt/nginx \
@@ -87,7 +144,7 @@ the following command from the root of the chunkin module's source tree:
The following versions of Nginx should work with this module:
-* '''0.8.x''' (last tested version is 0.8.28)
+* '''0.8.x''' (last tested version is 0.8.29)
* '''0.7.x >= 0.7.21''' (last tested version is 0.7.64)
Earlier versions of Nginx like 0.6.x and 0.5.x will ''not'' work.
@@ -107,6 +164,28 @@ Available on github at [http://github.com/agentzh/chunkin-nginx-module agentzh/c
= ChangeLog =
+== v0.07 ==
+
+* marked the disgarded 411 error page's output chain bufs as consumed by setting <code>buf->pos = buf->last</code>. (See [http://nginx.org/pipermail/nginx-devel/2009-December/000025.html this nginx-devel thread] for more details.)
+* added the [[#chunkin_keepalive|chunkin_keepalive]] directive which can enable HTTP 1.1 keep-alive and HTTP 1.1 pipelining, and defaults to <code>off</code>.
+* fixed the <code>alphtype</code> bug in the Ragel parser spec; which caused rejection of non-ascii octets in the chunked data.
+* added <code>Test::Nginx::Socket</code> to test our nginx module on the socket level. Thanks J for his bug report.
+* rewrote the bufs recycling part and preread-buf-to-rb-buf transition part, also refactored the Ragel parser spec, thus eliminating lots of serious bugs.
+* provided better diagnostics in the error log message for "bad chunked body" parsefails in the chunked parser. For example:
+
+<geshi lang="log">
+ 2009/12/02 17:35:52 [error] 32244#0: *1 bad chunked body (offset 7, near "4^M
+ hell <-- HERE o^M
+ 0^M
+ ^M
+ ", marked by " <-- HERE ").
+ , client: 127.0.0.1, server: localhost, request: "POST /main
+HTTP/1.1", host: "localhost"
+</geshi>
+
+* added some code to let the chunked parser handle special 0-size chunks that are not the last chunk.
+* fixed a connection leak bug regarding incorrect <code>r->main->count</code> reference counter handling for nginx 0.8.11+ (well, the <code>ngx_http_read_client_request_body</code> function in the nginx core also has this issue, I'll report it later.)
+
== v0.06 ==
* minor optimization: we won't traverse the output chain link if the chain count is not large enough.
@@ -132,16 +211,15 @@ Some parts of the test suite requires modules [[NginxHttpProxyModule|proxy]] and
= Known Issues =
-* May not work properly if keep-alive is enabled in Nginx. Use it with keep-alive at your own risk.
* May not work with certain 3rd party modules like the [http://www.grid.net.ru/nginx/upload.en.html upload module] because it implements its own request body reading mechanism.
* "client_body_in_single_buffer on" may *not* be obeyed for short contents and fast network.
* "client_body_in_file_only on" may *not* be obeyed for short contents and fast network.
+* HTTP 1.1 pipelining may not fully work yet.
= TODO =
* Fix the [[#Known Issues|known issues]].
* Add support for trailers and chunked extensions as specified in the [http://tools.ietf.org/html/rfc2616#section-3.6.1 RFC]
-* Add tests for invalid chunked input by implementing Test::Nginx::Socket in the test scaffold.
= Getting involved =
@@ -188,6 +266,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
* The original thread on the Nginx mailing list that inspires this module's development: [http://forum.nginx.org/read.php?2,4453,20543 "'Content-Length' header for POSTs"].
* The orginal announcement thread on the Nginx mailing list: [http://forum.nginx.org/read.php?2,22967 "The chunkin module: Experimental chunked input support for Nginx"].
* The original [http://agentzh.spaces.live.com/blog/cns!FF3A735632E41548!481.entry blog post] about this module's initial development.
+* The thread discussing chunked input support on the nginx-devel mailing list: [http://nginx.org/pipermail/nginx-devel/2009-December/000021.html "Chunked request body and HTTP header parser"].
* The [[NginxHttpEchoModule|echo module]] for Nginx module's automated testing.
* [http://tools.ietf.org/html/rfc2616#section-3.6.1 RFC 2616 - Chunked Transfer Coding].

0 comments on commit fad343d

Please sign in to comment.