.textlintrc should be encoded as UTF-8

[azu]

Currently, textlint throw error when .textlintrc file is encoded as Shift-JIS.
We should check that the .textlintrc is UTF-8.

Proposal

If .textlintrc is not encoded by UTF-8, show error mesage in console.

Modules

• utf-8-validate
  • It is native module
  • Previoulsly, it is introduced in Validate config utf8 #397, but it is reverted in fix(textlint): remove utf-8-validate #509

Edit: Re-open by #509

[azu]

We can use utf-8-validate.
Buffer.from need Node. 5

• utf-8-validate
• utf-8-validate/test.js at master · websockets/utf-8-validate
• Buffer | Node.js v8.2.1 Documentation

[azu]

alt https://github.com/hcodes/isutf8

[azu]

https://github.com/shinnn/is-file-utf8

[Leko]

I think utf-8-validate would be better.
I try it on Node.js v4.8.7, it works expected.

> node -e "console.log(require('utf-8-validate')(require('fs').readFileSync('utf8.txt')))"
true
> node -e "console.log(require('utf-8-validate')(require('fs').readFileSync('sjis.txt')))"
false

[Leko]

Currently, textlint throw error when .textlintrc file is encoded as Shift-JIS.

@azu How do I reproduce it ?

I try to reproduce on my blog( https://github.com/Leko/WEB-EGG ).

1. clone this repo
2. Run npm i
3. Convert .textlintrc to Shift-JIS
4. Run npm run lint:md

It does not throw an error.
But max-kanji-continuous-len.allow not working.
If .textlintrc encoded as UTF8, it works fine.

My .textlintrc is here: https://github.com/Leko/WEB-EGG/blob/master/.textlintrc#L17

[azu]

This issue is based on bug report on twitter.

• https://twitter.com/okinaka3/status/867298831263715328
• https://twitter.com/azu_re/status/867662138831130624

textlint use rc-loader in days past.
But, I've replaced it to rc-config-loader #39 #262

Maybe, "throwing error" is caused by rc-loader?

[Leko]

Hmm...
It doesn't throw error even in rc-loader@1.0.0.

We can close this issue ?

[azu]

https://github.com/textlint-ja/textlint-rule-preset-JTF-style/blob/master/src/index.js
JTF preset use multibyte string in the key.
Is it a reason?

It does not throw an error.
But max-kanji-continuous-len.allow not working.

It is better that assert this. Silent failing make user confuse.
Bacause, It is not expeceted bahavior for user.

[azu]

I love loud error rather than silent error.

[Leko]

max-kanji-continuous-len.allow not working.

In this case, if assert .textlintrc encoded as UTF8, markdown should be UTF8.
textlint can support text files encoded as UTF8 only.
It's OK ?

BTW, rc-loader throws error in https://github.com/textlint-ja/textlint-rule-preset-JTF-style/blob/master/src/index.js .

2017/12/16 1:16
I tested this JSON. src/index.js is javascript so I update as JSON.
https://gist.github.com/Leko/c937dbc2d9cdf451fc3595531e99b2da

[Leko]

I love loud error rather than silent error.

I think so too.

[azu]

textlint can support text files encoded as UTF8 only.

Yes.
it looks good that implement the assertion in config-loader.

textlint/packages/textlint/src/config/config-loader.ts

Line 43 in 41cae97

filePath: result.filePath

[Leko]

I got it.
I'll try to fix it :)

[azu]

micnic/uv: Ultrafast UTF-8 data validation looks like good.
require Node.js 6+.

But, we can drop Node.js 4 support in near future #443

[azu]

hcodes/isutf8: Quick check if a Node.js Buffer is UTF-8

[azu]

Next Node.js will introduce utf8 checker
nodejs/node#45947

[azu]

buffer.isUtf8(input) require Node.js v18.14.0.

textlint 14 treat Node.js 18.14 as the minimal version.

• v14 roadmap #1200

[JayC561]

Hey @azu , is this issue closed or are we still looking for solutions?

[azu]

#298 (comment)
is not implemented yet.