[indic] DIsallow matras after explicit half-form

[behdad]

It was reported to me that we accept matras after a explicit half form (C,H,ZWJ) whereas Uniscribe doesn't. Indeed the Indic specs do not allow that in the grammar. I tracked this down to f0b8ed1:

commit f0b8ed1b6dd9f1d2b9084c101a6fc5dee0cc22a8
Author: Behdad Esfahbod <behdad@behdad.org>
Date:   Wed Sep 5 17:32:57 2012 -0400

    [Indic] Allow "H,ZWJ,M"
    
    Uniscribe accepts a Halant,ZWJ before matras.  Allow that.
    
    BENGALI down from 295 to 291
    DEVANAGARI down from 69 to 57
    GUJARATI down from 19 to 17
    KANNADA down from 871 to 867
    MALAYALAM down from 340 to 337
    TELUGU down from 20 to 16

Given how little the gains were (~4 per script), that does look quite suspicious to me now. We should re-evaluate.

[behdad]

I tried this. If I revert this change, I see 2 extra test failures in Gujarati, and 12 in Devanagari.

In Gujarati, it looks like Uniscribe is allowing C,H,ZWJ,left-matra, for left-matra being U+0ABF. Ie. Uniscribe accepts this sequence: <U+0AA7,U+0ACD,U+200D,U+0ABF,U+0AA7>.

In Devanagari, looks like Uniscribe is allowing the following three matras after C,H,ZWJ: U+093E, U+0947, U+094B.

@PeterCon, can you clarify the above by any chance? Currently we do not distinguish between different types of matras in our Indic shaper grammar.

[PeterCon]

In Devanagari, looks like Uniscribe is allowing the following three matras after C,H,ZWJ: U+093E, U+0947, U+094B.

I'm not sure how you are inferring that. We will insert a dotted-circle glyph after ZWJ and before the vowel marks to reflect that the vowel marks don't have a valid base.

Same for <U+0AA7,U+0ACD,U+200D,U+0ABF,U+0AA7>: we'll display 0ABF placed on a dotted-circle glyph.

[behdad]

I'm not sure how you are inferring that. We will insert a dotted-circle glyph after ZWJ and before the vowel marks to reflect that the vowel marks don't have a valid base.

My testing has been based on Windows 7 Uniscribe. I'll see if I can test with latest Windows and adapt. Thanks.

[behdad]

@jfkthame should I just revert the mentioned commit? WDYT? Any chance you can test these on latest Windows and report? Thanks.

[behdad]

@anthrotype if you have Win 10 machine, can you please test these for me? Thanks.

[anthrotype]

sure I can do that. How should I test it?

[anthrotype]

I tried reverting commit f0b8ed1, then I recompiled the ragel files, and run ninja test, and I see "100% tests passed, 0 tests failed out of 101".
Running on latest Windows 10. I hope I did it right...

[anthrotype]

I had to un-comment the ragel tasks in the top-level CMakeLists.txt in order to make it generate the respective *.hh files in the ${PROJECT_BINARY_DIR}/src, and also I had to pass the full path to ragel.exe to the cmake command line with -DRAGEL=C:\msys64\mingw64\bin\ragel.exe.

Is there another, better way? cc @ebraminio

[ebraminio]

Behdad will kill me :)) (as he didn't like the addition of alternative build tool which can make confusion, like this case) I guess he refers to https://github.com/harfbuzz/harfbuzz/tree/master/test/shaping/data/text-rendering-tests which is not CMake related and should be tested with mingw/msys2.

[anthrotype]

oh, ok.. so I need to rebuild with mingw-w64 and run the tests with that. I'll try tomorrow, unless you do it before me.

[anthrotype]

dumb question.. is this the same issue as the one @behdad asked me to test here?
9940504#commitcomment-29580127

[ebraminio]

I'll try tomorrow, unless you do it before me.

The problem is I don't have a stable Windows machine with the build tools installed and updated. Every time I go to my Windows machine everything should be updated and that is a bit annoying with my net.

dumb question.. is this the same issue as the one @behdad asked me to test here?

That one can be done with cmake and doesn't make much difference. Feel free to contact me on some IM of your choice (Telegram for example, with the same id) so I can help you quicker if needed :)

[behdad]

My testing has been based on Windows 7 Uniscribe. I'll see if I can test with latest Windows and adapt. Thanks.

I cannot reproduce.

I think we should revert this commit.