Cmn voice not correctly translated

[hgneng]

@jaacoppi

This is a following issue of #1044.

I apologize for my not careful test. I find issues after @cary-rowen tell me that there are still problems with Chinese Mandarin voice.

Here is the current voice output:

$ espeak-ng -x -vcmn '此电脑 means this computer'
c'i3_| di11'a4n_| na11'o3_| (en)mi:11nz_| DI11s_| k@11mpju:11t311(cmn)_|

And following output from older version is much better than current situation:

tsh'i[21_| t'iE51n_| n'Au21_| (en)mi:44nz_| DI11s_| k@11mpju:11t311(cmn)_|

Maybe this is why you had comment: #1044 (comment)

Here is the revert code:
cmn_revert.patch.txt

In fact, it's a temporary solution of #1163 .

I will try to compare the rules of Cantonese and Mandarin to find whether there are clues to make both Mandarin and English read correctly.

[hgneng]

I think the problem is that eSpeak not correctly handle diphthongs in cmn. It inserts a low level tone 11 between diphthongs in cmn.

Current situaltion is:
$ espeak-ng -x -vcmn '此电脑'
c'i3_| di11'a4n_| na11'o3_|

It should be translated as:
c'i3_| d'ia4n_| n'ao3_|

[hgneng]

"此电脑" in the cmn_list is translated as "ci3 dian4 nao3" in Pinyin.

As I have mentioned above, the pronunciation of "此电脑" in the old version (i.e. 1.50) is correct: tsh'i[21_| t'iE51n_| n'Au21_|

The latest pronunciation of "此电脑" is not correct: c'i3_| di11'a4n_| na11'o3_|

So I try to make some changes to phsource/ph_cmn:

1. add phoneme ia like iE

phoneme iE
  vwl  starttype #i  endtype #e
  length 250 
  FMT(vwl_zh/ie)
endphoneme

phoneme ia
  vwl  starttype #i  endtype #a
  length 250 
  FMT(vwl_zh/ie)
endphoneme

2. add phoneme ao like Au

wl starttype #a endtype #u
  length 250
  FMT(vwl_zh/aau)
endphoneme

phoneme ao
  vwl starttype #a endtype #o
  length 250
  FMT(vwl_zh/aau)
endphoneme

3. replace phoneme i with i[

//phoneme i
//  vwl  starttype #i  endtype #i
//  length 250
//  IF nextPh(N) THEN
//    FMT(vowel/ii_2)
//  ENDIF
//  IF nextPh(n) THEN
//    FMT(vowel/ii_5)
//  ENDIF
//  FMT(vowel/i)
//endphoneme

phoneme i
  vwl  starttype #i  endtype #i
  length 250
  FMT(vowel/i#_7)
endphoneme

phoneme i[ //after ts tsh s 
  vwl  starttype #i  endtype #i
  length 250
  FMT(vowel/i#_7)
endphoneme

phoneme i. //after ts. ts.h s. z.
  vwl  starttype #i  endtype #i
  length 250
  FMT(vowel/i#_6)
endphoneme

After applying the changes and rebuild, it sounds fine:

$ espeak-ng -x -vcmn '此电脑'
c'i3_| d'ia4n_| n'ao3_|

@jaacoppi
I need you help. Change 1 and 2 above seems fine. Shall I make changes to other cmn phonemes?

Change 3 should not work. I just don't understand how the rules work. How does eSpeak know when to use phoneme "i[" or "i." instead of "i"? I suppose this is not defined by the comment "//after ts tsh s" or "//after ts. ts.h s. z.". How do I write a rule for "i after c"?

Thank you very much!

[hgneng]

I see. In yue_rules, the rules are quite simple:

.group i
?1        i        _^_EN
?2        i        i
?2        iu       iu

In cmn_rules, the rules are complicated. Selecting "i", "i[" or "i." is defined in cmn_rules. After defining dictrules 1 in lang/sit/cmn, rules without prefix $1 will be ignore. So cmn phoneme translation break.

.group i
?1     i        _^_EN
       i        i //i in ing
    z) i        i[
    c) i        i[
    s) i        i[
    h) i        i. //after zh ch sh
    r) i        i.
       ia       iA
       ia (DnK  iE
       ia (DngK iA
       iao      jAu
       ia1o     jAu55
       ia2o     jAu35
       ia3o     jAu214
       ia4o     jAu51
       ie       iE
       io (DngK y
    q) io (DngK u
       iu       iou

@jaacoppi
What do you think we should do to "ci3"? How do we write a rule for "i" after "c" for Pinyin?

[jaacoppi]

@hgneng I speak very little chinese. If you think it sounds better then you should do a PR. Just remember to test with both combinations: chinese characters with english text and chinese characters with pinyin text.

Syntax for rules is explained in docs/dictionary.md. For "i" aftter "c", use something like:
c) i

[hgneng]

@jaacoppi Thank you for replying.

Since the current cmn_rules is complicate, I think adding English rules basing on Mandarin rules is easier than the reverse one. I will try to revert cmn dictrules to 2 and fix English reading. The following test case seems works. I will write more test cases and fix other rules.

$ espeak-ng -x -vcmn '此电脑 means this computer 你好吗 ma is a horse'
tsh'i[21_| t'iE51n_| n'Au21_| (en)mi:44nz_| DI11s_| k@11mpju:11t311(cmn)_| n'i35_| X'Au21_| mA44_| (en)mA:11_| I11z_| a#11_| hO@11s(cmn)_|

[hgneng]

I have rewritten most of the rules of cmn. Vowel will be spoken as Mandarin only when it's with a tone number. Otherwise, it will be regarded as English. This will make English words translated more correctly.

$ espeak-ng -x -vcmn '人之初，性本善。性相近，习相远。Men at their birth are naturally good. Their natures are much the same; their habits become widely different.'
z.'@35n_| ts.'i.55_| ts.h'u55_|
S;'i51N_| p'@21n_| s.'a51n_|
S;'i51N_| S;'iA55N_| tS;'i51n_|
S;'i35_| S;'iA55N_| 'y&214n_|
(en)m'E55n_| a55t_| De@55_| b'3:55T_| A@55_| n'a55tS@55r@L55i55_| g'U55d(cmn)_|
(en)De@55_| n'eI55tS355z_| A@55_| m'V55tS_| D@55_| s'eI55m(cmn)_|
(en)De@55_| h'a55bI55ts_| bI55k,V55m_| w'aI55dli55_| d'I55fr@55nt(cmn)_|

However, there is one more thing that can be improved. All English vowel is added with a tone 55. Cantonese also has the same problem. I don't know how to fix it. It should be more natural without the tone for English.

[jaacoppi]

Excellent work! The tone marker 55 is probably somewhere in the code, not in the rules.

[hgneng]

@ssb22
Dear Silas,
Have you been using espeak-ng these years? I find that the Mandarin extra dictionary doesn't work any more.

For example, there is an item in cmn_listx:
(天 地) tian1di4

But it is still matching the default item in cmn_list:

$ espeak-ng -X -vcmn '天地'
Replace: 天   tian1 
Translate 'tian1'
  1	t        [th]

  1	i        [_^_]
 57	ia (nL02 [iE]

  1	n        [n]

 22	1        [55]

Replace: 地   de5 
Translate 'de5'
  1	d        [t]

 22	d) e     [@]
  1	e        [_^_]
 21	e (L02   [o-]

 22	5        [11]

th'iE55n_| t@22_|

I pick up another word (长 大) zhang3da4 randomly and have the same issue.

The Cantonese extra dictionary yue_listx doesn't have this issue. espeak-ng can translate Cantonese words correctly.

This issue seems exists for a long time at least since 1.50. Could you help to confirm it? And do you have any idea why cmn_listx doesn't work?

I also find that there are meaningless items in cmn_listx. For example there is an item (天 主) tian1zhu3 but either 天 or 主 has unique Pinyin. There is no need to add an item for 天主. I don't know when and why this happen.

[hgneng]

Single character in cmn_listx is correctly loaded. For example, 㐀 doesn't exists in cmn_list but only in cmn_listx. espeak-ng can translate 㐀 correctly.

$ espeak-ng -x -vcmn '㐀'
tS;h'iou55_|

[hgneng]

I test with eSpeak 1.47.11 and get correct result:

$ ./espeak -X -vzh '天地'
Replace: 天 地   tian1di4 
Translate 'tian1di4'
  1	t        [th]

  1	i        [i]
 22	ia       [iA]
 65	ia (DnK  [iE]

  1	n        [n]

 22	1        [55]

  1	d        [t]

  1	i        [i]

 22	4        [51]

 thiE55nt'i51_|

eSpeak 1.47 with Chinese dictionaries can be downloaded from eSpeak-Chinese: https://sourceforge.net/projects/e-guidedog/files/eSpeak-Chinese/1.47.11/

(I no longer maintained eSpeak-Chinese since 2013)

It seems that the problem lies in "Replace" stage.

[hgneng]

Cantonese goes another route "Found" different from Mandarin's "Replace".

$ espeak-ng -X -vyue '长江'
Found: '长' [coeng4]   $text
Found: '江' [gong1]   $text
c'oeng4_| g'ong1_|
$ espeak-ng -X -vyue '长大'
Found: '长 大
' [zoeng2daai6]   $text
zoeng2d'aai6_|

[hgneng]

Following code can fix the Mandarin dictionary word matching issue:

diff --git a/src/libespeak-ng/tr_languages.c b/src/libespeak-ng/tr_languages.c
index 610676bc..a2050b8a 100644
--- a/src/libespeak-ng/tr_languages.c
+++ b/src/libespeak-ng/tr_languages.c
@@ -1590,8 +1590,8 @@ Translator *SelectTranslator(const char *name)
                tr->langopts.our_alphabet = 0x3100;
                tr->langopts.word_gap = 0x21; // length of a final vowel is less dependent on the next consonant, don't merge consonant with next word
                tr->langopts.textmode = true;
+               tr->langopts.listx = 1; // compile *_listx after *_list
                if (name2 == L3('y', 'u', 'e')) {
-                       tr->langopts.listx = 1; // compile zh_listx after zh_list
                        tr->langopts.numbers = NUM_DEFAULT;
                        tr->langopts.numbers2 = NUM2_ZERO_TENS;
                        tr->langopts.break_numbers = BREAK_INDIVIDUAL;

However, the second character of the word is read twice. It seems that the string pointer doesn't advance correctly for Mandarin word. I haven't found the cause yet.

$ espeak-ng -X -vcmn '天地'
Replace: 天 地   tian1di4 
Translate 'tian1di4'
  1	t        [th]

  1	i        [_^_]
 57	ia (nL02 [iE]

  1	n        [n]

 22	1        [55]

  1	d        [t]

  1	i        [_^_]
 21	i (L02   [i]

 22	4        [51]

Replace: 地   de5 
Translate 'de5'
  1	d        [t]

 22	d) e     [@]
  1	e        [_^_]
 21	e (L02   [o-]

 22	5        [11]

thiE55nt'i51_| t@11_|

[jaacoppi]

Might be a multibyte character issue. Those have known to cause problems in the past. Some index or offset is wrong.. somewhere.

[hgneng]

@jaacoppi
After a whole days' trying, here is my fix for the Mandarin word issue. Do you think it's reasonable? Is there any test script in espeak-ng that I can run to confirm other functions not breaking by this patch?

$ git diff src/libespeak-ng/translate.c
diff --git a/src/libespeak-ng/translate.c b/src/libespeak-ng/translate.c
index 8759b48c..ca60cc84 100644
--- a/src/libespeak-ng/translate.c
+++ b/src/libespeak-ng/translate.c
@@ -164,6 +164,10 @@ int TranslateWord(Translator *tr, char *word_start, WORD_TAB *wtab, char *word_o
                                wtab->flags &= ~FLAG_FIRST_UPPER;
                        }
 
+                       // dictionary_skipwords is a global variable and TranslateWord3 will reset it to 0 at the beginning.
+                       // However, dictionary_skipwords value is still needed outside this scope.
+                       // So we backup and restore it at the end of this scope.
+                       int skipwords = dictionary_skipwords;
                        TranslateWord3(tr, word_out, wtab, NULL, &any_stressed_words, current_alphabet, word_phonemes, sizeof(word_phonemes));
 
                        int n;
@@ -182,6 +186,7 @@ int TranslateWord(Translator *tr, char *word_start, WORD_TAB *wtab, char *word_o
                                while (!isspace(*word_out)) ++word_out;
                                while (isspace(*word_out))  ++word_out;
                        }
+                       dictionary_skipwords = skipwords;
                }
 
                // If the list file contains a text replacement to another

[jaacoppi]

Tests are in tests/ and are run with `make check`. Your patch looks interesting. Let's see what happens. It might be that the current tests are faulty and this should have been the correct behavior all along.