NormalizerNFKC: add unify_katakana_trailing_o option

[HashidaTKS]

When unify_katakana_trailing_o is specified, NormalizerNFKC* normalize characters as below.

オオ -> オウ
コオ -> コウ
ソオ -> ソウ
トオ -> トウ
ノオ -> ノウ
ホオ -> ホウ
モオ -> モウ
ヨオ -> ヨウ
ロオ -> ロウ
ゴオ -> ゴウ
ゾオ -> ゾウ
ドオ -> ドウ
ボオ -> ボウ
ポオ -> ポウ

Usage:

normalize \
  'NormalizerNFKC130("unify_katakana_trailing_o", true, \
                     "report_source_offset", true)' \
  "オオコオソオトオノオホオモオヨオロオゴオゾオドオボオポオ" \
  WITH_CHECKS|WITH_TYPES

[HashidaTKS]

@kou

Would you review this when you have time?

[kou]

Could you fix indent?

[HashidaTKS]

Fixed.

[HashidaTKS]

I think you told me what checks are before, but now I'm not confident with that these checks results are correct or not...
-1 corresponds to オウ's ウ, for example, right?
And does -1 mean that the character is the second or subsequent character normalized by one definition that normalizes multiple characters at once?

[kou]

The results are incorrect.
#1506 (comment) will fix them.

-1 corresponds to オウ's ウ, for example, right?

Right.

And does -1 mean that the character is the second or subsequent character normalized by one definition that normalizes multiple characters at once?

No. -1 means that the character (ウ) doesn't have corresponding character in the source (オオ). But the second オ is the corresponding character of ウ.

[HashidaTKS]

Hmm, the results applied #1506 (comment) seem wrong...
Even the normalized result ("normalized": "オウオソウオノウオモウオロウオゾウオボウオ",) is wrong.
I will re-check it.

[HashidaTKS]

*n_used_bytes and *n_used_characters are already added first character's value, so they should not have added the first character's value again at that line.

[HashidaTKS]

But checks result still have -1...

[HashidaTKS]

Current implementation normalize two bytes at once, that means when normalizing コオ -> コウ, we normalize not only オ but コオ it self. Is that reason why the check result of オ is -1 ...?

[HashidaTKS]

I think in order to normalize only オof コオ, we need to know the previous character in grn_nfkc_normalize_unify_katakana_trailing_o.
In order to know the previous character in grn_nfkc_normalize_unify_katakana_trailing_o, we need to pass data about the previous character as the user_data argument like

groonga/lib/normalizer.c

Line 1888 in 8f04e6d

grn_nfkc_normalize_strip(grn_ctx *ctx,

Is it a good idea?

[kou]

Let's try.

[HashidaTKS]

Modified.

[HashidaTKS]

@kou

Thank you for your comments.
I have addressed you comments.
Would you re-review this when you have time?