Japanese characters are not grabbed with this regex?

[mikeda37]

wtf_wikipedia/src/list/index.js

Line 6 in 476c7c1

const has_word = /[a-z_0-9\]}]/i

Does this regex pattern grab Japanese characters?

[spencermountain]

oh, good catch. no it doesn't.
we should probably start using the \p{Letter} syntax - both here and elsewhere!
prs welcomed.
thanks

[mikeda37]

@spencermountain
ok, i understand using Unicode property escapes but i just wonder if this filtering is necessary.

wtf_wikipedia/src/list/index.js

Line 42 in 476c7c1

sub = sub.filter((a) => a && has_word.test(a))

can you please tell me what this is for? i tried reading code but it's hard for me!
thanks

[spencermountain]

hey mikeda37, good question. I don't remember.
If it's killing-off japanese language lists, feel free to remove or change it. My policy is that if the tests pass, it's good!

The number of list formats is completely nuts, if you're digging around in that code, there's a lot of random template bits.

Lastly, if you're making a pr, I've been converting dev branch to use es-modules. I was gonna ship it today, but you're not supposed to do releases on fridays. So i may do that on monday. it may make sense to wait until then.
cheers

[mikeda37]

ok, you don't remember! 😄 i understand your policy!
i tested omitting the line but some unit tests fail here.

wtf_wikipedia/tests/unit/Document.test.js

Line 680 in a3abfaa

test('lists - get - return all lists on the page', (t) => {

wtf_wikipedia/tests/unit/Document.test.js

Line 699 in a3abfaa

test('lists - get - if the clue is any other type then return all lists', (t) => {

you don't remember these tests too? do you think i can remove these tests?

if you want to keep those tests, i would think of using Unicode property for filtering and then i need a few weeks until pr.
you can work for es-modules converting soon!
but in case you approve to remove those two tests, i can make a pr in a few days!
thanks

[spencermountain]

ok, 10.0.0 is released, you'll need to update before making a PR, let me know if I can help.
cheers

[mikeda37]

ok, i got it!
how do you think about deleting those two tests? I did't understand those tests contents..
do you think we should keep them? or you think I have to analyze and judge?
thanks

[spencermountain]

hey - in the anarchism page, there are a bunch of lists with citations, with render no text, so they are empty lists.
the tests are correct - we need to filter them out. Feel free to come up with whatever solution you'd like
cheers

[mikeda37]

@spencermountain, thanks for your explanation!
I see, those filtering is to remove empty lists, right?
So I won't delete the filtering and instead, I will fix this regex pattern /[a-z_0-9\]}]/i as removing empty lines but not to kill the unicode characters. is it fine?

by the way, the test fails with the latest source!
cheers

[spencermountain]

yep, that sounds good.
thanks

[spencermountain]

hi @mikeda37 i've added unicode support to the list regex you found in 10.0.2 - please let me know if you find any others
cheers

[mikeda37]

hi, @spencermountain sorry for leaving the task and thanks for the fix!
I will try the fixed code when I got time.
thanks