Skip to content
Newer
Older
100644 267 lines (171 sloc) 9.2 KB
2e4e99b @pgte initial commit
pgte authored Mar 27, 2012
1 # Creating Your Streams
2
d219c73 @pgte some corrections
pgte authored Mar 27, 2012
3 Node has this wonderful abstraction named "Streams". Streams come in two flavours: Readable Streams and Writable Streams. Node has some implementations of these stream interfaces that you have probably already used, like a TCP Connection, an HTTP Client Request or even a File read stream. Besides using them, it is sometimes useful to create your own streaming objects.
2e4e99b @pgte initial commit
pgte authored Mar 27, 2012
4
d219c73 @pgte some corrections
pgte authored Mar 27, 2012
5 Any object can be a Readable or a Writable Stream if it implements the right set of methods and events. Before we learn about them, let's catchup on some underlying conventions in Node regarding the interface these objects should have:
2e4e99b @pgte initial commit
pgte authored Mar 27, 2012
6
7 ## Readable Streams
8
9 An example of a Readable Stream is a video file stream. It will emit that file data stream and it will eventually end. You can pause it and resume it.
10
11 ## Writable Streams
12
d219c73 @pgte some corrections
pgte authored Mar 27, 2012
13 A Writable Stream is a stream that you can write to. An example of such a stream is a log file you can append to.
14
15 On top of letting you write data to it, a Writable Stream should also inform you if the write you did was able to flush the buffers into the destination stream, or if the destination stream was full. This simple mechanism and the fact that you can pause and resume Readable Streams allows Node to do some basic flow control.
2e4e99b @pgte initial commit
pgte authored Mar 27, 2012
16
17 ## Duplex Streams
18
19 A Duplex Stream is a stream that implements both the Readable and the Writable Stream interfaces. Generally, these streams perform some kind of transformation. An example of such a stream is the Node `zlib.Gzip` pseudo-class, which lets you write data to it and that outputs a gzip stream.
20
21 ## Piping
22
23 You can pipe any Readable Stream into another writable stream by using:
24
25 ```javascript
26 var sourceReadStream = ...
27 var targetWriteStream = ...
28
29 readStream.pipe(writeStream);
30 ```
31
32 On top of writing to the destination every data that it gets on the source stream, pipe also does some basic flow control: if the data written to the destination stream is not flushed, pipe pauses the source stream. Pipe resumes the source stream once the data on the target stream is flushed.
33
34 ## Implementing a Writable Stream
35
36 Any object can be a Writable Stream: it just has to implements a handful of methods and emit some events.
37
38 ### .writable
39
d219c73 @pgte some corrections
pgte authored Mar 27, 2012
40 First of all, a Writable Stream object must have a property named "writable" set to the boolean value "true". This makes this object viable for piping data to when using the `Stream.prototype.pipe` method.
2e4e99b @pgte initial commit
pgte authored Mar 27, 2012
41
42 ### .write(buffer, [encoding])
43
44 Your Writable Stream should obviously implement the `write` method. The first argument may be a string or a buffer. If it's a string, the second argument may contain the encoding. If it doesn't you should assume the default "utf8" encoding.
45
d219c73 @pgte some corrections
pgte authored Mar 27, 2012
46 If you managed to flush this new buffer out of the Node process, you should return `true`. If not, return `false` to enable the propper flow control mechanisms when piping.
2e4e99b @pgte initial commit
pgte authored Mar 27, 2012
47
d219c73 @pgte some corrections
pgte authored Mar 27, 2012
48 If you returned `false`, you should emit the "drain" once the buffer is flushed out. If you don't, `Stream.prototype.pipe` will assume you are still buffering and will wait indefinitely.
2e4e99b @pgte initial commit
pgte authored Mar 27, 2012
49
50 ### .end([buffer, [encoding]])
51
52 Your stream should be able to end. While ending, a final buffer or string may be passed in to be written, in which case the two arguments should have the same semantics as in the `write` method.
53
d219c73 @pgte some corrections
pgte authored Mar 27, 2012
54 This method should be interpreted as a command to end in the near future and may not happen immediately. For instance, some data may still be buffered and may have to be flushed out before actually closing the underlying resources (if any).
2e4e99b @pgte initial commit
pgte authored Mar 27, 2012
55
56 Typically this command is implemented as just a change in object state since the actual cleaning-up can be done in the `.destroy()` method.
57
58 ### .destroy()
59
60 This method will be called by the `Stream.prototype.pipe()` method once the source has emitted the "close" event.
61 If you need to do some cleanup (like closing a file or a socket), this is where you should do it, but only after all the pending buffers (if any) have been flushed.
62
63 ### Emit the "close" event
64
65 Once the underlying stream (a file or socket, for instance) has been closed you should emit this event.
66
67 ### Emit the "error" event
68
69 Should anything go wrong while writing, transforming or flushing the buffers, you should emit an "error" event like this:
70
71 ```javascript
72 this.emit('error', new Error('Something went terribly wrong'));
73 ```
74
d219c73 @pgte some corrections
pgte authored Mar 27, 2012
75 This will enable the stream programatic clients to catch and handle errors gracefully. If there is noone listening to the "error" events, Node will throw an uncaught exception.
2e4e99b @pgte initial commit
pgte authored Mar 27, 2012
76
1900257 @pgte correcting example
pgte authored Mar 29, 2012
77
2e4e99b @pgte initial commit
pgte authored Mar 27, 2012
78 ## Implementing a Readable Stream
79
80 A Readable Stream is mainly an object that emits "data" events.
81
82 The best way to start implementing a Readable Stream is to inherit from the Node "stream" module. This module exports the `Stream` constructor so you can derive new pseudo-classes from like this:
83
84 ```javascript
85 var inherits = require('util').inherits;
86 var Stream = require('stream');
87
88 function MyReadableStream(options) {
89 Stream.call(this);
90 this.readable = true;
91 }
92
93 inherits(MyReadableStream, Stream);
94
95 MyReadableStream.protoytype.pause = function() {
96 // ...
97 }
98
99 // ...
100 ```
101
102 This will get your prototype chain setup correctly and inherit the correct pipe behavior.
103
104 Now you will have to implement a handful of methods and properties.
105
106
107 ### .readable
108
109 First of all, a Readable Stream object must have a property named `readable` that has the boolean value "true". This tells `Stream.prototype.pipe()` that this stream implements the Readable Stream interface.
110
111
112 ### .setEncoding(encoding)
113
114 By default you should emit Buffer objects, but if the encoding gets set, you should emit encoded strings. You should support the official Node encodings: "utf8", "ascii" and "base64".
115
116 ### Emitting "data" events
117
118 What a readable stream mainly does is emit "data" events. You can emit a data events like this:
119
120 ```javascript
121 this.emit('data', new Buffer('some string'));
122 ```
123
124 Notice that if the object client sets the encoding, you shouuld emit strings, not buffers. A more correct situation for emitting a wrapped string would be this one:
125
126 ```javascript
127
128 MyReadableStream.prototype.setEncoding = function(encoding) {
129 this.encoding = encoding;
130 };
131
132 MyReadableStream.prototype.encodeString = function(str) {
133 var encoded = new Buffer(str);
134 if (this.encoding) {
135 encoded = encoded.toString(this.encoding);
136 }
137 return encoded;
138 }
139
140 MyReadableStream.prototype.emitDataString = function(str) {
141 this.emit('data', this.encodeString(str));
142 };
143
144 // ...
145 this.emitDataString('my UTF-8 string');
146 ```
147
148 This code will make sure you have the right string encoding when emitting "data" events. You can further optimize this code for the case where the encoding is already 'utf8':
149
150 ```javascript
151 MyReadableStream.prototype.encodeString = function(str) {
152
153 if (this.encoding === 'utf8') { return str; }
154
155 var encoded = new Buffer(str);
156 if (this.encoding) {
157 encoded = encoded.toString(this.encoding);
158 }
159 return encoded;
160 }
161
162 ```
163
164 On the other hand, if you originally have your data in a Buffer format, you can generally do something like this before emitting:
165
166 ```javascript
167 var buf = // some buffer I got
168 if (this.encoding) { buf = buf.toString(this.encoding); }
169 this.emit('data', buf);
1900257 @pgte correcting example
pgte authored Mar 29, 2012
170 ```
2e4e99b @pgte initial commit
pgte authored Mar 27, 2012
171
172 ### .pause()
173
174 This should make the readable stream not emit any further "data" event before the next call to `.resume()`.
175
176 ### .resume()
177
178 This should unpause your stream, enabling future "data" events to be emitted.
179
180 ## A Readable Stream Example
181
182 Here is an example of a Readable Stream module that emits random buffers at fixed intervals:
183
184 ```javascript
185 var Stream = require('stream');
186 var inherits = require('util').inherits;
187
188 function RandomStream(options) {
189
190 Stream.call(this);
191
192 this.readable = true;
193
194 if(! options) { options = {}};
195 if (! options.interval) options.interval = 1000; // Defaults to 1 sec
196 if (! options.size) options.size = 10; // Defaults to 10 bytes
197
198 this.options = options;
199
200 this.resume();
201 }
202
203 inherits(RandomStream, Stream);
204
205 RandomStream.prototype.setEncoding = function(encoding) {
206 this.encoding = encoding;
207 };
208
209 RandomStream.prototype.encode = function(buffer) {
210 if (this.encoding) {
211 buffer = buffer.toString(this.encoding);
212 }
213 return buffer;
1900257 @pgte correcting example
pgte authored Mar 29, 2012
214 };
2e4e99b @pgte initial commit
pgte authored Mar 27, 2012
215
216 RandomStream.prototype.emitRandom = function() {
217 var buffer = new Buffer(this.options.size);
218 for(var i = 0; i < buffer.length; i ++) {
219 buffer[i] = Math.floor(Math.random() * 256);
220 }
221 this.emit('data', this.encode(buffer));
1900257 @pgte correcting example
pgte authored Mar 29, 2012
222 };
2e4e99b @pgte initial commit
pgte authored Mar 27, 2012
223
224
225 RandomStream.prototype.pause = function() {
226 if (this._interval) {
227 clearInterval(this._interval);
228 delete this._interval;
229 }
1900257 @pgte correcting example
pgte authored Mar 29, 2012
230 };
2e4e99b @pgte initial commit
pgte authored Mar 27, 2012
231
232 RandomStream.prototype.resume = function() {
233 var self = this;
234
1900257 @pgte correcting example
pgte authored Mar 29, 2012
235 if (this.ended) { throw new Error('Stream has ended'); }
236
2e4e99b @pgte initial commit
pgte authored Mar 27, 2012
237 if (! this._interval) {
1900257 @pgte correcting example
pgte authored Mar 29, 2012
238 this._interval =
239 setInterval(function() {
240 self.emitRandom();
241 }, this.options.interval);
2e4e99b @pgte initial commit
pgte authored Mar 27, 2012
242 }
1900257 @pgte correcting example
pgte authored Mar 29, 2012
243 };
244
245 RandomStream.prototype.end = function(buf) {
246 this.ended = true;
247 if (buf) { this.write(buf); }
248 this.pause();
249 };
250
251 RandomStream.prototype.destroy = function() {
252 // do nothing
2e4e99b @pgte initial commit
pgte authored Mar 27, 2012
253 }
254
255 module.exports = RandomStream;
256 ```
257
258 You can save this module in a file in the local directory named "random_stream.js" and use it like this:
259
260 ```javascript
261 var RandomStream = require('./random_stream');
262
263 var randomStream = new RandomStream({interval: 500, size: 20});
264 randomStream.on('data', console.log);
265 ```
266
267 This should start printing out random 20 byte buffers every half a second.
Something went wrong with that request. Please try again.