Character map duplicated replacement issues and rule order problem

[ronaldtse]

Pattern replacements probably shouldn't be applied more than once. For example,

  59) Interscript maps/un-jpn-Hrkt-Latn-hepburn.yaml system test for {"source"=>"とうきょう", "expected"=>"tôkyô"}
      Failure/Error: expect(result).to eq(test["expected"])
      
        expected: "tôkyô"
             got: "tôukyoょu"
      
        (compared using ==)
      # ./spec/interscript_spec.rb:16:in `block (5 levels) in <top (required)>'

This is a problem. とうきょう is only supposed to match these rules:

    "とう": "tô"
    "きょう": "kyô"

But the result shows it matching (we should have a debugging method to show which rules are matching):

    "とう": "tô"
    "う": "u"
    "きょ": "kyo"
# No rule for "ょ"
    "う": "u"

We must remove those replaced tokens after they were swapped, and we must ensure that longer patterns are matched first.

[ronaldtse]

Moreover, some characters do not have a direct replacement. For example, ょ has no character replacement because it does not exist on its own, it can only depend on other characters.

[andrew2net]

@ronaldtse I can't find maps/un-jpn-Hrkt-Latn-hepburn.yaml system. Have you pushed it?

[ronaldtse]

I haven't pushed it. But the same issue applies to mext-jpn-Hrkt-Latn-hepburn as you see in #19 .

[andrew2net]

@ronaldtse please try it again using the latest master

[ronaldtse]

@andrew2net thanks, but the issues in #19 persist, e.g.

  13) Interscript maps/mext-jpn-Hrkt-Latn-kunrei.yaml system test for {"source"=>"ぽっぽっや", "expected"=>"poppoyya"}
      Failure/Error: expect(result).to eq(test["expected"])
      
        expected: "poppoyya"
             got: "poppoっya"
      
        (compared using ==)
      # ./spec/interscript_spec.rb:13:in `block (5 levels) in <top (required)>'

[andrew2net]

@ronaldtse I fixed #19

[chaaklau]

This issue is still around.

In this simple map below, after applying the first rule, ABCD becomes y, yielding the output string yE, which should have removed the context for the second rule. Currently it returns Z instead, showing that both rules have been applied, and the offset is not handled correctly.

tests:
  - source: 'ABCDE'
    expected: "yE"

map:
  rules:
  - pattern: 'ABCD'
    result: "y"
  - pattern: 'DE'
    result: "z"

(The commit b9154f5 removes this logic and fixes the Russian map which relies on the problematic part.)

[ronaldtse]

@andrew2net do you have time to fix this issue? Thanks!

[andrew2net]

Rules aways apply to an original string, not to a previous rule result. It's done for a purpose.
For example, we have a string её which means her in Russian. When we apply a rule which changes first е to ye we got yeё. Then we should apply the rule "ё after е should be converted to ye", but we already replace the first Russian е with Latinised ye.

[chaaklau]

Rules aways apply to an original string, not to a previous rule result. It's done for a purpose.

This can be problematic (see my comments above).

For example, we have a string её which means her in Russian. When we apply a rule which changes first е to ye we got yeё. Then we should apply the rule "ё after е should be converted to ye", but we already replace the first Russian е with Latinised ye.

Instead of replacing Cyrillic е to Latin ye, which will remove the context for subsequent rule, one can instead insert a Latin y before Cyrillic е. Then Cyrillic ё will still be following Cyrillic е, which allows the Latin y insertion rule to click in.

Can you please take a look at b9154f5 and check if looks good to you?

[andrew2net]

Instead of replacing Cyrillic е to Latin ye, which will remove the context for subsequent rule, one can instead insert a Latin y before Cyrillic е. Then Cyrillic ё will still be following Cyrillic е, which allows the Latin y insertion rule to click in.

I've tried to do that way, but if there are many rules like in maps/bgnpcgn-rus-Cyrl-Latn-1947.yaml it becomes too hard to predict all possible changes in previous rules and patterns in the rules become huge and hard to debug.

Can you please take a look at b9154f5 and check if looks good to you?

It is similar to my first implementation. For example, the third rule won't work if previous changes E or e.

I'll think about what we can do with this issue