UTF8 Converter #108

nhlpl · 2015-02-25T09:42:11Z

UTF8 Database Converter have to be used before using NodeBB Import Tool.

At this moment nodebb-plugin-import breaks non-latin charakcters.
ą > ±
ć > æ
ę > ê
ł > ³
ń > ñ
ś > ¶
ż > ¿

SQL System Variables for IP.Board 2.3.x

collation_connection: latin1_swedish_ci
collation_database: latin1_swedish_ci
collation_server: latin1_swedish_ci

character_set_client: latin1
character_set_connection: latin1
character_set_database: latin1

character_set_results: latin1
character_set_server: latin1
character_set_system: utf8

akhoury · 2015-02-25T14:44:02Z

can you suggest a module that does that? I'll gladly add it.

nhlpl · 2015-02-25T18:36:31Z

Do you think about npm modules like this one?
https://www.npmjs.com/package/gulp-convert-encoding

akhoury · 2015-02-25T19:17:20Z

more like https://github.com/ashtuchkin/iconv-lite which is used by gulp-convert-encoding, thanks for the lead

nhlpl · 2015-02-26T10:21:20Z

Do you need any additional support to make it work? I'll be happy to test it out when it gets ready.

akhoury · 2015-02-26T15:13:04Z

I would love to have to a sample DB dump to test with my self.

sohrabafard · 2015-12-11T19:31:40Z

Hello,
about "vbulletin" to "nodebb" migration using nodebb import plugin, the contents were transferred as follows:

Ù„Ø·Ù�Ø§ Ù�Ù‚Ø· Ú©ØªØ¨ Ùˆ Ø¬Ø²ÙˆØ§ØªÛŒ Ø±Ø§ Ù‚Ø±Ø§Ø± Ø¯Ù‡ÛŒØ¯ Ú©Ù‡ Ù†ÙˆÛŒØ³Ù†Ø¯Ù‡ ÛŒ Ø¢Ù† Ù‡Ø³ØªÛŒØ¯ Ùˆ ÛŒØ§ Ø§Ø² Ù†ÙˆÛŒØ³Ù†Ø¯Ù‡ ÛŒ Ø¢Ù† Ø§Ø¬Ø§Ø²Ù‡ Ø¯Ø§Ø±ÛŒØ¯.

correct form is :

لطفا فقط کتب و جزواتی را قرار دهید که نویسنده ی آن هستید و یا از نویسنده ی آن اجازه دارید.

Who can help me?

akhoury · 2015-12-11T22:23:55Z

let me see what I can do. Im gonna close this one here akhoury/nodebb-plugin-import-vbulletin#4, as they are duplicates

sohrabafard · 2015-12-11T22:36:08Z

thanks,
When I copy any corrupted text in notepad++ and then convert it from ansi to utf8 ,
text will be ok...

akhoury · 2015-12-11T22:42:31Z

@sohrabafard can you provide me a with database dump to test with? feel free to send it privately to email bentael@gmail.com, and to obfuscate emails, password and other data unless you want to trust me with it :)

sohrabafard · 2015-12-11T23:26:45Z

I will send two tables vb_thread and vb_post that have bunch of utf8 data ( about 1.5K records )
with foratmail@gmail.com.
.
each Table:
Type: InnoDB
Collation: utf8_general_ci

Thanks a lot!

sohrabafard · 2015-12-11T23:40:29Z

an idea :
make a plugin that iterate on nodebb Posts, Category, ... and change encoding of text(String) from ANSI to "UTF-8" :

convert fron Ansi to Utf-8 dosen't change any English text.

https://community.nodebb.org/topic/7213/iterate-on-nodebb-posts

sohrabafard · 2015-12-12T11:50:07Z

let me know, if you get my mail!

sohrabafard · 2015-12-13T00:13:31Z

After install encoding packages:

    npm install encoding & npm install iconv

As you suggest:

encoding.convert(str,  'windows-1252','utf8');

work fine to recover text, so I entered below JavaSacript code in pre-post input of nodebb-plugin-import:
content = encoding.convert(content, 'windows-1252','utf8');
get this error :

13/12 03:26 [8180] - error: ReferenceError: encoding is not defined
    at eval (eval at <anonymous> (/home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:339:18), <anonymous>:3:11)
    at Object.Controller.convert (/home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:368:34)
    at /home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:1266:87
    at /home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:567:21
    at /home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:246:17
    at /home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:122:13
    at _each (/home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:46:13)
    at async.each (/home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:121:9)
    at _asyncMap (/home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:245:13)
    at Object.map (/home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:216:23)
ReferenceError: encoding is not defined
    at eval (eval at <anonymous> (/home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:339:18), <anonymous>:3:11)
    at Object.Controller.convert (/home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:368:34)
    at /home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:1266:87
    at /home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:567:21
    at /home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:246:17
    at /home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:122:13
    at _each (/home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:46:13)
    at async.each (/home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:121:9)
    at _asyncMap (/home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:245:13)
    at Object.map (/home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:216:23)

I checked server/controller.js you define "encoding" I don't know why we have this error!

akhoury · 2015-12-13T00:47:17Z

i know why, hang on will fix it in an hour.

akhoury · 2015-12-13T00:50:41Z

actually just be fixed now

sohrabafard · 2015-12-13T01:33:32Z

in Topic Title:

    13/12 05:00 [9592] - error: TypeError: undefined is not a function
        at /home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/bbcode-to-markdown/bbcode-to-markdown.js:24:31
        at Object.Controller.convert (/home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:368:24)
        at Data.eachGroup.async (/home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:1225:25)
        at replenish (/home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:191:21)
        at /home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:203:33
        at nxt (/home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:1220:12)
        at Data.eachGroup.async (/home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:1229:12)
        at replenish (/home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:191:21)
        at /home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:203:33
        at nxt (/home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:1220:12)
    TypeError: undefined is not a function
        at /home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/bbcode-to-markdown/bbcode-to-markdown.js:24:31
        at Object.Controller.convert (/home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:368:24)
        at Data.eachGroup.async (/home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:1225:25)
        at replenish (/home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:191:21)
        at /home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:203:33
        at nxt (/home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:1220:12)
        at Data.eachGroup.async (/home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:1229:12)
        at replenish (/home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:191:21)
        at /home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/async/lib/async.js:203:33
        at nxt (/home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:1220:12)

sohrabafard · 2015-12-13T01:35:50Z

and this error:

    13/12 05:03 [9720] - error: TypeError: undefined is not a function
        at Object.module.exports.utils.slugify (/home/nodeBB/forum/public/src/utils.js:81:14)
        at /home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:1310:88
        at /home/nodeBB/forum/src/database/mongo/hash.js:15:4
        at handleCallback (/home/nodeBB/forum/node_modules/mongodb/lib/utils.js:96:12)
        at /home/nodeBB/forum/node_modules/mongodb/lib/collection.js:974:5
        at /home/nodeBB/forum/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/server.js:778:13
        at Callbacks.emit (/home/nodeBB/forum/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/server.js:95:3)
        at null.messageHandler (/home/nodeBB/forum/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/server.js:249:23)
        at Socket.<anonymous> (/home/nodeBB/forum/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:297:20)
        at Socket.emit (events.js:107:17)
    TypeError: undefined is not a function
        at Object.module.exports.utils.slugify (/home/nodeBB/forum/public/src/utils.js:81:14)
        at /home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:1310:88
        at /home/nodeBB/forum/src/database/mongo/hash.js:15:4
        at handleCallback (/home/nodeBB/forum/node_modules/mongodb/lib/utils.js:96:12)
        at /home/nodeBB/forum/node_modules/mongodb/lib/collection.js:974:5
        at /home/nodeBB/forum/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/server.js:778:13
        at Callbacks.emit (/home/nodeBB/forum/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/server.js:95:3)
        at null.messageHandler (/home/nodeBB/forum/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/server.js:249:23)
        at Socket.<anonymous> (/home/nodeBB/forum/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:297:20)
        at Socket.emit (events.js:107:17)

sohrabafard · 2015-12-13T01:51:36Z

so we have just two error:

first on:

 Topics titles and content

that is:

13/12 05:03 [9720] - error: TypeError: undefined is not a function
    at Object.module.exports.utils.slugify (/home/nodeBB/forum/public/src/utils.js:81:14)
    at /home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:1310:88
...

second on:

BB-Code to MarkDown

that is:

    13/12 05:00 [9592] - error: TypeError: undefined is not a function
    at /home/nodeBB/forum/node_modules/nodebb-plugin-import/node_modules/bbcode-to-markdown/bbcode-to-markdown.js:24:31
    at Object.Controller.convert (/home/nodeBB/forum/node_modules/nodebb-plugin-import/server/controller.js:368:24)
...

and user,Category name and description,Private Message, user signature have no problem.

sohrabafard · 2015-12-13T01:59:25Z

secound problem solved buy running this command:

npm install bbcode-to-markdown

So I recommend you to add bbcode-to-markdown to dependecy.

akhoury · 2015-12-13T02:13:02Z

it is added as dependency. https://github.com/akhoury/nodebb-plugin-import/blob/master/package.json#L44

something is wrong with your installation.

sohrabafard · 2015-12-13T02:17:08Z

i install plugin using npm,
what about first error?

akhoury · 2015-12-13T02:28:21Z

can you show me your parseBefore function syntax ? you're not passing a string through.

sohrabafard · 2015-12-13T02:33:54Z

 content = encoding.convert(content,"windows-1252","utf8");

this works for category and post.

akhoury · 2015-12-13T03:09:06Z

console.log(typeof encoding.convert("test","windows-1252","utf8")); // prints object

use this instead, basically just add toString() to it.

 content = encoding.convert(content,"windows-1252","utf8").toString();

sohrabafard · 2015-12-13T03:25:27Z

Yes, the problem was solved!

please close issue!

Thanks a lotو

akhoury added a commit that referenced this issue Dec 13, 2015

related #108

d5eb5fc

akhoury closed this as completed Dec 13, 2015

Neurovert mentioned this issue Oct 11, 2018

Some special characters not supported? Polish MyBB import #224

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UTF8 Converter #108

UTF8 Converter #108

nhlpl commented Feb 25, 2015

akhoury commented Feb 25, 2015

nhlpl commented Feb 25, 2015

akhoury commented Feb 25, 2015

nhlpl commented Feb 26, 2015

akhoury commented Feb 26, 2015

sohrabafard commented Dec 11, 2015

akhoury commented Dec 11, 2015

sohrabafard commented Dec 11, 2015

akhoury commented Dec 11, 2015

sohrabafard commented Dec 11, 2015

sohrabafard commented Dec 11, 2015

sohrabafard commented Dec 12, 2015

sohrabafard commented Dec 13, 2015

akhoury commented Dec 13, 2015

akhoury commented Dec 13, 2015

sohrabafard commented Dec 13, 2015

sohrabafard commented Dec 13, 2015

sohrabafard commented Dec 13, 2015

sohrabafard commented Dec 13, 2015

akhoury commented Dec 13, 2015

sohrabafard commented Dec 13, 2015

akhoury commented Dec 13, 2015

sohrabafard commented Dec 13, 2015

akhoury commented Dec 13, 2015

sohrabafard commented Dec 13, 2015

UTF8 Converter #108

UTF8 Converter #108

Comments

nhlpl commented Feb 25, 2015

akhoury commented Feb 25, 2015

nhlpl commented Feb 25, 2015

akhoury commented Feb 25, 2015

nhlpl commented Feb 26, 2015

akhoury commented Feb 26, 2015

sohrabafard commented Dec 11, 2015

akhoury commented Dec 11, 2015

sohrabafard commented Dec 11, 2015

akhoury commented Dec 11, 2015

sohrabafard commented Dec 11, 2015

sohrabafard commented Dec 11, 2015

sohrabafard commented Dec 12, 2015

sohrabafard commented Dec 13, 2015

akhoury commented Dec 13, 2015

akhoury commented Dec 13, 2015

sohrabafard commented Dec 13, 2015

sohrabafard commented Dec 13, 2015

sohrabafard commented Dec 13, 2015

sohrabafard commented Dec 13, 2015

akhoury commented Dec 13, 2015

sohrabafard commented Dec 13, 2015

akhoury commented Dec 13, 2015

sohrabafard commented Dec 13, 2015

akhoury commented Dec 13, 2015

sohrabafard commented Dec 13, 2015