NormalizerNFKC*: Add tests for mixing katakana options #1539
Conversation
| 6, | ||
| 0, | ||
| 0, | ||
| 32616, |
There was a problem hiding this comment.
This is odd, this should be 3 but 32616.
The value 32626 is not stable, changes each time I run it.
I will check it.
There was a problem hiding this comment.
「ズァ」と「ヅァ」のときだけ発生していて、「ヴァ」などでは起きないので、昨日の須藤さんの修正とは関係なさそうな感じもします。（調べています。）
There was a problem hiding this comment.
最新のmasterにrebaseしたところ直ったので、関係あったようです。
There was a problem hiding this comment.
いえ、ご対応ありがとうございます。
HashidaTKS
force-pushed
the
add-multiple-normalizer-option-test
branch
from
0533db3
to
4ea7c23
Compare
|
Would you review this when you have time? |
lib/normalizer.c
Outdated
| grn_nfkc_normalize_unify_katakana_trailing_o, | ||
| &need_trailing_check, | ||
| "[unify][katakana-trailing-o]"); | ||
| grn_nfkc_normalize_unify_stateless(ctx, data, &unify, false); |
There was a problem hiding this comment.
I don't want to change the current following order:
grn_nfkc_normalize_unify_stateless(before)grn_nfkc_normalize_unify_statefull()sgrn_nfkc_normalize_unify_stateless(after)
For keeping unifying order simple.
How about adding ョ and small letters to katakana_trailing_o targets.
There was a problem hiding this comment.
Ok, I will fix so as to normalize オ after small letters.
Agree. |
|
Thank you, I have addressed your comments. |
| @@ -0,0 +1,14 @@ | |||
| normalize \ | |||
| 'NormalizerNFKC100("unify_katakana_v_sounds", true, \ | |||
| "unify_katakana_gu_small_sounds", true, \ | |||
There was a problem hiding this comment.
Broken indent but I'll merge this as-is. I can't review all tests...
kou
changed the title
Added tests for mixing katakana options.
And fixed
unify_katakana_trailing_oin order to work fine for the following cases.ョー -> ヨー -> ヨオ -> ヨウョーwas normalized toヨオ(should beヨウ) becauseunify_katakana_trailing_owas applied beforeunify_kana_caseandunify_katakana_trailing_ois not normalizedオafter small letters.I have added
オafterォorョas a target ofunify_katakana_trailing_o.ヺー -> ヺオ -> ヺウヺーwas normalized toヺオbecauseunify_katakana_trailing_odidn't targetヺ.I have added
ヺas a target ofunify_katakana_trailing_o.