Move UAX#14 defines to line.toml

[makotokato]

• Generate UAX#14 table using toml file like UAX#29.
• Remove python tool to generate UAX#14 machine state table

[aethanyc]

I'm excited to see python script gets converted to TOML! I assume this doesn't change the behavior?

Suggestion:

1. Remove rule_table.rs since it's longer used.
2. How about renaming line_breaker.rs to line.rs to match other breaker implementation like word.rs, etc?

[aethanyc]

Suggestion: Is this # LB31?

[aethanyc]

This seems like a reasonable change to reflect LB21: BB ×.

[jira-pull-request-webhook]

Notice: the branch changed across the force-push!

• experimental/segmenter/src/line.rs is different
• experimental/segmenter/src/provider.rs is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

[makotokato]

I assume this doesn't change the behavior?

Yes, tests are passed although some unused states are different, but it is unused.

[makotokato]

And I also add multi thread version of data generation. But I will improve more after this.

[aethanyc]

And I also add multi thread version of data generation. But I will improve more after this.

It's nice to speed up bulid.rs as the interim developer efficiency. However, I'd feel we probably shouldn't spend more time on the performance of bulid.rs because we really should migrate the segmenter data generation into icu4x data generation tool, i.e. https://github.com/unicode-org/icu4x/tree/main/tools/datagen. Ideally, the segmenter data generation can be speed up on top of the work in #1600.

[jira-pull-request-webhook]

Notice: the branch changed across the force-push!

• experimental/segmenter/README.md is different
• experimental/segmenter/src/lib.rs is different
• experimental/segmenter/src/line.rs is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot