-
Notifications
You must be signed in to change notification settings - Fork 106
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Respecting encoding? #14
Comments
that's a good question actually... try |
So I gave that a try, still got a buffer: var fs = require('fs');
var through2 = require('through2');
fs.createReadStream('./file.json', { encoding: 'utf8' })
.pipe(through2({ encoding: 'utf8' }, function(chunk, enc, cb) {
console.log('chunk', chunk, 'enc', enc);
this.push(chunk);
cb();
})) $ node stream_test.js
chunk <Buffer 7b 0a 20 20 22 6e 61 6d 65 22 3a 20 22 61 70 70 2d 74 72 61 69 6e 69 6e 67 2d 30 31 22 2c 0a 20 20 22 76 65 72 73 69 6f 6e 22 3a 20 22 30 2e 30 2e 30 22 ...> enc buffer |
@Raynos help! I don't have time to try and understand this at the moment, any chance you understand this off the top of your head? |
You might try setting edit: curiousity got me -- yes, that works. |
@stu-salsbury Thanks! I suppose this works this way so that a transform can be configured by its creator to know what type of data to transform. Still a little weird though that |
np -- I've been spending more time than I ever intended reading and rereading that page lately! |
Note to myself: should try to figure out how to fix this. |
Ok so I think I've got to the bottom of it: https://github.com/isaacs/readable-stream/blob/master/lib/_stream_writable.js#L233-L240 The thing is that there is an option called |
Seems like a bug. |
we shouldnt encode a buffer as utf8 to just decode it to a buffer. |
Why should through2 not respect the decoding defaults of node? I think it's working as it should be designed... That is to say that I don't think through2 should make assumptions about encoding or decoding, and the node If the thinking is that node's |
@stu-salsbury agreed. this sounds like a node core bug. |
quoting from the docs -- "encoding String If specified, then buffers will be decoded to strings using the specified encoding." So this says that if a buffer comes in, do you treat it as if it is encoded a certain way.... if an incoming stream is already a string, as is the case in the example above, then this option will have no effect... so the point here is that you aren't actually ever decoding a buffer in through2 if you get an incoming string... and that decodeStrings = false is telling the transform stream not to decode back to a buffer. I agree that this is a strange choice though perhaps meant to encourage buffer usage and hence drive performance? I dunno. Somebody went to the trouble to document that decodeStrings defaults to true so it seems intentional. EDIT: I wonder if this is a backward compatibility issue for node, more than a choice that anyone would support in a greenfield design. |
If this is still an open issue, what would be needed to resolve it? If the start of stream is being defined somewhere else (in another function, another class, etc.), it may not be clear what encoding the person using through2 should expect. Example: //this is really annoying
someObject.getStream().pipe(through2(function (chunk, enc, cb) {
switch(enc) {
case 'utf8':
case 'base64':
etc...
}
})) Therefore, there is a big advantage to being able to set the encoding as an option of through2, and there is also a big advantage in when you don't want to. I think that having the setting optional, as well as being close to the function using the data, is very useful. So is this really an issue? |
Hi, my English is bad, so I will express myself in code.
So if you want your
The relations are so complex,I think |
I have a problem of trying to get to the bottom of things too often, and I found that this behavior actually boils down to following:
Considering all above, observed behavior is not a problem (at least not with through2 or transform streams). What OP tried to do (or at least ended up with) is object-input stream (don't know about output), so he should have used The real problem is that I got all of above from experimenting and reading library code, not from documentation. |
@evg-zhabotinsky Knowing that maybe something like this should be done. var t2 = new DestroyableTransform(options)
//Right after construction.
//Maybe a more precise condition would be better.
//if(!!options.decodeStrings && options.encoding !== 'buffer'){
if(options.encoding !== 'buffer'){
t2.once('pipe', function(src){
//This is emitted synchronously in the src.pipe method
if(src._readableState.encoding !== 'buffer'){
t2._writableState.objectMode = true;
}
});
} But that might cause problems for people who expect buffers all the time because of how the current streams work. |
Shouldn't
enc
in the below example be 'utf8'? In node 0.10.26 it'sbuffer
:Is it expected that transforms are doing string conversions? Am I doing something wrong?
The text was updated successfully, but these errors were encountered: