/
README
374 lines (281 loc) · 13.6 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
Name
drizzle-nginx-module - an upstream module that talks to
mysql, drizzle, and sqlite3 by libdrizzle
Status
This module is at its very early phase of development and
considered highly experimental. But you're encouraged to
test it out on your side and report any quirks that you
experience.
We need your help! If you find this module useful and/or
interesting, please consider joining the development!
Commit bit can be freely delivered at your request ;)
Synopsis
http {
...
upstream cluster {
# simple round-robin
drizzle_server 127.0.0.1:3306 dbname=test
password=some_pass user=monty protocol=mysql;
drizzle_server 127.0.0.1:1234 dbname=test2
password=pass user=bob protocol=drizzle;
}
upstream backend {
drizzle_server 127.0.0.1:3306 dbname=test
password=some_pass user=monty protocol=mysql;
}
server {
location /mysql {
set $my_sql 'select * from cats';
drizzle_query $my_sql;
drizzle_pass backend;
}
...
}
}
Description
This is an nginx upstream module integrating libdrizzle
(https://launchpad.net/libdrizzle) into nginx in an
non-blocking and streamming way.
Essentially it provides a very efficient and flexible way
for nginx internals to access mysql, drizzle, sqlite3,
as well as other RDBMS's that support the drizzle protocol
or mysql protocol. Also it can serve as a direct REST
interface to those RDBMS backends.
It also has a builtin per-worker connection pool
mechanism.
Here's a sample configuration:
upstream backend {
drizzle_server 127.0.0.1:3306 dbname=test
password=some_pass user=monty protocol=mysql;
drizzle_keepalive max=100 mode=single overflow=reject;
}
The drizzle_keepalive directive has the following options:
* max=<num>
Specify the capacity of the connection pool for
the current upstream block. The <num> value MUST
be non-zero. If set to 0, it effectively disables
the connection pool. And this is the default
if no "drizzle_keepalive" directive is specified.
* mode=<mode>
This supports two values, "single" and "multi".
"single" mode means the pool does not distinguish
various drizzle servers in the current ustream
block while "multi" means the pool will merely
reuse connections which have identical server
host names and ports. Note that it will igore
differences between dbnames or users.
* overflow=<action>
This option specifies what to do when the
connection pool is already full while new
database connection is required. Either "reject"
(without quotes) or "ignore" can be specified.
In case of "reject", it will reject the current
request, and returns "503 Service Unavaliable"
error page. For "ignore", this module will
go on creating a new DB connection.
For now, the connection pool uses a simple LIFO algorithm
to assign idle connections in the pool. That is,
most recently (successfully) used connections will be
reused first the next time. And new idle connections
will always replace the oldest idle connections in the
pool even if the pool is already full.
Caveats
* Other usptream modules like "upstream_hash" and
"upstream_keepalive" MUST NOT be used with this module
in a single upstream block.
* Directives like "server" MUST NOT be mixed with
"drizzle_server" either.
* Upstream backends that don't use "drizzle_server" to
define server entries MUST NOT be used in the
"drizzle_pass" directive.
Output
This module generates binary query results in a format
that will be shared among the various nginx database
driver modules like ngx_postgresql and ngx_oracle.
This data format is named "Resty DBD Stream" (RDS).
If you're a web app developer, you may be more interested
in using a source filter module like ngx_rds_json module
( http://github.com/agentzh/rds-json-nginx-module ) to
obtain JSON output.
For the HTTP response header part, the 200 OK status
code should always be returned.
The Content-Type header MUST be set to
"application/x-resty-dbd-stream" (without quotes).
And the driver generating this response is also set a
X-Resty-DBD header. For instance, this
module adds the following output header:
X-Resty-DBD-Module: drizzle 0.0.1
where 0.0.1 is this module's own version number. This
X-Resty-DBD-Module header is optional though.
Below is the HTTP response body format (version 0.0.2):
Header part
uint8_t endian type (1 means big-endian and little
endian otherwise)
uint32_t format version
(v1.2.3 is represented as 1002003 in
decimal)
uint8_t result type
(0 means normal SQL result type,
fixed for now)
uint16_t standard error code
uint16_t driver-specific error code
uint16_t driver-specific error string length
u_char* driver-specific error string data
uint64_t database rows affected
uint64_t insert id (if none, 0)
uint16_t column count
Body part
when the "column count" field in the Header part
is zero, then the whole body part is omitted.
0*Column (number of columns is determined by "column count")
uint16_t non-zero value for standard column type
code and for the column list
terminatoandr otherwise.
uint16_t driver-specific column type code
uint16_t column name length
u_char* column name data
0*Row (terminated by 8-bit zero)
uint8_t valid row (1 means valid, and 0 means
the row list terminator)
0*Field (count is predetermined by column number)
uint32_t field length
uint8_t* field data (in textual representation)
On the nginx output chain link level, the following
components should be put into a single ngx_buf_t struct:
* the header
* each column and the column list terminator
* each row's valid flag byte and row list terminator
* each field in each row (if any) but the field data
can span multiple bufs
Installation
You should first install the latest libdrizzle which can be
obtained from here (at least libdrizzle 0.6 is required):
https://launchpad.net/libdrizzle
Grab the nginx source code from nginx.net (<http://nginx.net/>), for
example, the version 0.8.30 (see nginx compatibility), and then build
the source with this module:
$ wget 'http://sysoev.ru/nginx/nginx-0.8.30.tar.gz'
$ tar -xzvf nginx-0.8.30.tar.gz
$ cd nginx-0.8.30/
# Here we assume you would install you nginx under /opt/nginx/.
$ ./configure --prefix=/opt/nginx \
--add-module=/path/to/drizzle-nginx-module
$ make -j2
$ make install
Download the latest version of the release tarball of this module from
drizzle-nginx-module file list
(<http://github.com/agentzh/drizzle-nginx-module/downloads>).
Compatibility
The following versions of Nginx should work with this module:
* 0.8.x (last tested version is 0.8.32)
* 0.7.x >= 0.7.44 (last tested version is 0.7.64)
Earlier versions of Nginx like 0.6.x and 0.5.x will *not* work.
If you find that any particular version of Nginx above 0.7.44 does not
work with this module, please consider reporting a bug.
TODO
* implement the "drizzle_max_output_size" directive.
When the RDS data is larger then the size specified,
the module will try to terminate the output as quickly
as possible but will still ensure the resulting
response body is still in valid RDS format.
* implement the "drizzle_upstream_next" mechanism for
failover support.
* add support for multiple "drizzle_query" directives
in a single location.
* provide a hash-by-key algorithm for the upstream
server clusters.
as an alternative approach to the simple round-robin.
* implement weighted round-robin algorithm for the
upstream server clusters.
* add the "max_idle_time" option to the "drizzle_server"
directive, so that the connection pool will automatically
release idle connections for the timeout you specify.
* add the "min" option to the "drizzle_server" directive
so that the connection pool will automatically create
that number of connections and put them into the pool.
* implement the "drizzle_quote" directive so that one
can do variable value escaping and quoting as well
as simple type checking. For instance:
drizzle_quote $model table;
drizzle_quote $col column;
drizzle_quote $val integer;
drizzle_query "select * from $model where $col > $val";
* add "subrequest in memory" support so that this module
can be used with the ngx_eval module. For instance:
eval $res {
drizzle_quote $arg_user;
drizzle_quote $arg_pass;
drizzle_quote $arg_order keyword; # asc or desc
drizzle_query "select count(1) from users
where user=$arg_user and password=$arg_pass
order by id $arg_order";
drizzle_pass backend;
};
if ($res ~ '...') {
return 403;
}
....
* Completed the "drizzle_dbname" directive's implementation
such that we can switch dbname on a per-request basis.
* Unix domain socket support in the "drizzle_server"
directive.
* Introduce a "drizzle_type" diretive to allow the user to
define custom types to be used in the "drizzle_quote"
directive. For example:
drizzle_type email
'^[-_A-Za-z0-9.]+\@[-_A-Za-z0-9.]+$' # regex constraint
inherit=string; # parent type
drizzle_type $arg_email email;
drizzle_query "select * from users where email = $arg_email";
Of course, regex constraints will require the pcre support in
nginx.
* Make the "drizzle_query" directive reject variables that
have not been processed by a drizzle_process directive.
This will pretect us from SQL injections. There will
also be an option ("strict=no") to disable such checks.
Known Issues
* When mysql's packets are splitted to 1 ~ 4, 6 and etc.
bytes long, the state machine in libdrizzle may go
wrong. This is very likely a bug in libdrizzle. When
this happens, ngx_drizzle will throw out the 502 error
page immediately and put an error message in your error.log.
See Also
* the ngx_rds_json module
( http://github.com/agentzh/rds-json-nginx-module )
Authors
chaoslawful (王晓哲) <chaoslawful at gmail dot com>,
agentzh (章亦春) <agentzh at gmail dot com>.
Copyright & License
The implementation of the builtin connection pool has borrowed
a lot of code from Maxim Dounin's upstream_keepalive module.
This part of code is copyrighted by Maxim Dounin.
I also borrowed a lot of code from ngx_http_upstream.c and
ngx_http_memcached_module.c in the nginx 0.8.30 core. This
part of code is copyrighted by Igor Sysoev.
This module is licenced under the BSD license.
Copyright (c) 2009, Taobao Inc., Alibaba Group ( http://www.taobao.com ).
Copyright (c) 2009, chaoslawful <chaoslawful@gmail.com>.
Copyright (C) 2009 by agentzh <agentzh@gmail.com>.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of the Taobao Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.