UTF-8 encoding

[Tink-]

I am using your event-stream module in combination with json-csv to create a CSV file from an array of objects. Some elements contain text with utf-8 characters. When using readArray, these characters are converted. I am using the following code to save the file:

var out = fs.createWriteStream(csvFile, {encoding: 'utf8'});
var readable = event-stream.readArray(results);
readable.pipe(jsoncsv.csv(options)).pipe(out);

[dominictarr]

your issue doesn't state what the problem is...

[Tink-]

Sorry, I apparently lost a part of my text.

The problem is that after converting the array to a stream, the utf8 characters are converted.

[dominictarr]

converted to what?

[dominictarr]

can you provide code I can run to reproduce the problem?

[Tink-]

Converted to non-UTF8 characters, like the ones listed here:
www.i18nqa.com/debug/utf8-debug.html .

On Mon, May 4, 2015 at 9:26 AM, Dominic Tarr notifications@github.com
wrote:

converted to what?

—
Reply to this email directly or view it on GitHub
#85 (comment)
.

[dominictarr]

Read array is dead simple and doesn't concatenate any values so it should not do anything to alter the strings. My guess is that the problem is in jsoncsv though I don't really know because you haven't given me any runnable code.

You are basically expecting me to figure out what you are trying to do and then solve your problem too, for free...

This is not how open source works. How open source works is you help me find problems with my code and I help you with your problem - it's reciprocal. Its a skill and ita okay I'd you are still learning it

[dominictarr]

Oops.

You need to put more effort into how you ask for help otherwise you make it hard to help you

[Tink-]

Sorry this took so long. This is the code I used to test it, I attached the
htmlFile that is used.

var util = require('util');
var path = require('path');
var fs = require('fs');
var jsdom = require('jsdom');
var es = require('event-stream');
var jsoncsv = require('json-csv');

var options = {fields: [
{
name : 'id',
label : 'BookID'
},
{
name: 'paragraph',
label: 'Paragraph'
}
]};
var htmlFile = 'testfile.htm';

var results = [];

fs.readFile(htmlFile, 'utf8', function (error, htmlData) {
if (error) {
console.log('Encountered an error while loading file');
console.log(error);
throw error;
}
else {
jsdom.env(htmlData, [], function (errors, window) {

                    var $ = require('jquery')(window);
                    var body = $(window.document.body);
                    var paragraphs = [];

                    //Add all p-nodes to array
                    var children = body.find("p").removeClass("toc");
                    children.each(function () {
                            paragraphs.push($(this).text());
                    });


                    for (var i = 0; i < paragraphs.length ; i++){
                            results.push({id: i, paragraph: paragraphs[i]});
                    }

                    console.log("Generating CSV file");
                    var csvFile = path.resolve(__dirname, 'testset.csv');
                    var out = fs.createWriteStream(csvFile,

{encoding: 'utf8'});
var readable = es.readArray(results);
readable.pipe(jsoncsv.csv(options)).pipe(out);
});
}
});

Thanks in advance for all the help,
And again, sorry, for not sending the runnable code right away, I had to
alter my code to get a minimal working example, and that proved to be more
work then I expected.

On Tue, May 5, 2015 at 2:48 AM, Dominic Tarr notifications@github.com
wrote:

Oops.

You need to put more effort into how you ask for help otherwise you make
it hard to help you

—
Reply to this email directly or view it on GitHub
#85 (comment)
.

[dominictarr]

can you put this in a gist with package.json and the html file?
Also, can you reproduce this problem without using jsdom?
Try to simplify the code to until you have the simplest thing that causes the problem

[right9ctrl]

Closing for lack of reponse