-
Notifications
You must be signed in to change notification settings - Fork 0
Background and current status
The number and quality of machine translation systems depend on the availability of linguistic resources such as parallel corpora.This availability correlates with the official status and the number of speakers of each language.The Romance languages in the Iberian Peninsula (Spanish, Portuguese, Catalan, Galician, Asturian, Aragonese, Mirandese and Aranese) have different official statuses and numbers of speakers. Regarding the official status, we could distinguish three levels: state official status (in an entire state of the Iberian Peninsula), autonomic or regional official status (in an autonomous region or a part of it), and international official status (in international institutions such as the European Union or the United Nations). Table 1 shows schematically the level of official status and the approximate number of speakers in the Iberian Peninsula for these languages.
| Language | ISO | State | Region | International | Speakers |
|---|---|---|---|---|---|
| Spanish | spa | X | X | X | 46.000.000 |
| Portuguese | por | X | X | X | 11.000.000 |
| Catalan | cat | X | X | 10.000.000 | |
| Galician | gal | X | 2.500.000 | ||
| Asturian | ast | 110.000 | |||
| Aragonese | arg | 30.000 | |||
| Mirandese | mwl | X | 15.000 | ||
| Aranese | oc-aran | X | 4.500 |
Table 1. Official status and number of speakers in the Iberian Peninsula.
Parallel corpora is the main resource for training machine translation systems. In table 2 we can see the number of segments available in the parallel corpus included in the Opus Corpus collection (https://opus.nlpl.eu/) between Spanish and the rest of the above-mentioned languages. Opus Corpus is the main repository of freely available parallel corpora.
| Portuguese | 249.7 M |
|---|---|
| Catalan | 105.7 M |
| Galician | 37.3 M |
| Asturian | 7.4 M |
| Mirandese | 9.7 K |
| Aranese | 1.5 M |
Table 2. Number of segments of the parallel corpus available in the Opus Corpus collection between Spanish and the rest of languages. As no Aranese corpora are available, we state the figures for Occitan.
Monolingual corpora are also interesting resources for training machine translation engines. They can be used to improve the quality of a system through backtranslation (Sennrich, Haddow and Birch 2015). This technique consists in using a monolingual corpus of the target language (B) to create a parallel corpus, where the sentences in the source language (A) are obtained using a machine translation system for the B-A pair. This new synthetic parallel corpus is added to the actual available A-B parallel corpus, and both models are used to train the new A-B machine translation system. It is important to bear in mind that the only synthetic part of the synthetic parallel corpus obtained through backtranslation is the part corresponding to the source language (A), since the part corresponding to the target language (B) has been obtained from real texts of language B. Furthermore, as we will explain in the next sections, there are also unsupervised techniques to train machine translation systems from monolingual corpora. For Spanish, Portuguese, Catalan and Galician, large amounts of text can be easily collected from Common Crawl, which periodically downloads all the content from the web and makes the downloaded data available. A language detection algorithm is applied to this download that allows you to request the data for a specific language. Unfortunately, for the rest of the languages under study (Asturian. Aragonese, Aranese and Mirandese), no data is available, since the language detector does not detect these languages. Another possible source of monolingual corpus is Wikipedia, which has versions for all the languages of this project (except for Aranese, which could use the Occitan data experimentally). Table 3 shows the number of Wikipedia articles for each of the languages of this proposal.
| Language | Wikipedia articles |
|---|---|
| Spanish | 402.430 |
| Portuguese | 429.730 |
| Catalan | 133.214 |
| Galician | 39.627 |
| Asturian | 11.734 |
| Aragonese | 10.552 |
| Aranese | 14.584 |
| Mirandese | 2.000 |
Regarding the available automatic translation systems from Spanish to the rest of the languages, we are going to analyse three specific systems: Apertium (https://www.apertium.org/), which is a superficial syntactic transfer system that is distributed with a free licence; Google Translate (https://translate.google.com), a very popular neural machine translation system that provides numerous language pairs; and DeepL (https://www.deepl.com), a commercial neural system that is well-known for the quality of the translation it delivers. In table 4 we can observe the systems from Spanish to the rest of the languages of this study.
As can be seen from Table 4, only three languages (Portuguese, Catalan and Galician) have a neural machine translation system with Spanish as the source language. Currently, the predominant machine translation methodology and the one that achieves the best quality levels is neural machine translation (Castilho et al., 2017). Thus, most of the Romance languages under study do not have access to this technology.
| Apertium | Google Translate | DeepL | |
|---|---|---|---|
| Portuguese | X | X | X |
| Catalan | X | X | |
| Galician | X | X | |
| Asturian | X | ||
| Aragonese | X | ||
| Mirandese | |||
| Aranese | X |
Table 4. Availability of the Spanish pair to the rest of the languages for three widely used machine translation systems.
As we can see in table 4, Mirandese has no machine translation system available. As an important outcome of TAN-IBE will be the comparison of the newly trained neural machine translation systems with the existing ones, we decided to keep Mirandese out of the scope of this project.
Neural machine translation systems are trained using large, good-quality parallel corpora. The data in Table 2 are not encouraging for languages that do not have neural machine translation systems, since there are no corpora of sufficient size and quality for these languages. There is, therefore, a need for these languages to have larger parallel corpora.
In recent years, considerable interest has been aroused in the development of methodologies for training neural machine translation systems for language pairs that have very few resources. Four large groups of strategies can be highlighted: neural machine translation based on transfer learning; multilingual machine translation; self-supervised machine translation and unsupervised machine translation.
Let's say we want to have a machine translation system from language A to language C, but this language pair has very few parallel segments in the available parallel corpora. But there is a language B that is closely related to language C (for example, they are close languages within the same family, as the working languages of TAN-IBE) and we have a large parallel corpus for the A-B language pair. With the so-called transfer learning technique, we start by training a neural system from language A to language B and once the training is finished, we continue training it using a corpus of the B-C language pair. Zoph et al. (2016) apply transfer learning to train a French-Uzbek system (A-C) and start by training a French-English system (A-B) and once trained they continue training with a parallel corpus from English to Uzbek (B-C). Nguyen and Chians (2017) introduce a modification to this methodology using the vocabulary overlap between these languages. To increase the vocabulary overlap, they divide the words into subwords using BPE (Byte Pair Encoding) (Rico, Sennrich and Birch, 2016). They then train the A-B system and transfer the parameters including word embeddings of the source language to another model and continue to train the B-C system. In TAN-IBE, a Spanish-Aranese system could be trained, first, by training a Spanish-Catalan system with a large corpus and, once trained, by training the Catalan-Aranese system.
Multilingual machine translation systems (Firat et al. 2016) allow us to train a single multilingual neural system that shares a unique attention mechanism. Let's consider the languages A, B, C and D. If we have a parallel corpus for any of these combinations (for example: A-B, A-C, A-D, B-C and B-D) we can train a machine translation system that can translate between all pairs, regardless that for some of the pairs we do not have a parallel corpus (such as the C-D pair in our example). This is possible because the resulting system is capable of utilizing the similarities between languages. This configuration can be useful to train systems for language pairs with few resources, while training pairs of languages with more resources at the same time. In our proposal, the Spanish-Portuguese, Spanish-Catalan and Spanish-Galician pairs would be the pairs with many resources; while Spanish-Asturian, Spanish-Aragonese and Spanish-Aranese would be the pairs with few resources. This same configuration could produce translation systems for pairs without any parallel corpus, such as the Asturian-Aranese one. This capability is called zero-shot translation. Johnson et al. (2017) show that the quality of these zero-shot translations can be notably improved by having a few parallel segments of the C-D pair (Asturian-Aranese, in the previous example). Fan et al. (2021) emphasize that most multilingual systems have English as the central language, since they are trained only with parallel corpora made up of texts that have been translated from English or into English. In their work, they demonstrate that up to 10 BLEU points of improvement can be achieved using non-English-focused models in the translation of non-English language pairs. This work is important for this project proposal, since the central language will not be English.
Another aspect that has occupied the attention of researchers is the influence of typological differences between the languages that intervene in a multilingual system. In some studies (Zhang et al., 2020), backtranslation is used in multilingual systems to improve the translation quality for language pairs with no parallel corpus available.
Ruiter el al. (2019) present a method to train machine translation systems using comparable corpus in a self-supervised way. While training the system, two tasks are performed simultaneously: parallel segments are selected from the comparable corpus and the parameters of the neural system are learnt. They use Wikipedia, since it is a very rich source for creating comparable corpora. The system starts by calculating the monolingual word embeddings for each language A and B. Then the same Wikipedia article is taken in the two languages. Let us remember that Wikipedia articles in different languages are not translations of the same text, except in a few cases, but rather different texts that talk about the same topic. The system, article by article for each language, represents the sentences using the word embeddings for each language and, if the representations of two sentences in the two languages are similar enough, it considers the sentences as parallel segments and uses them to train the neural translation system. In other words, for each article in language A and its equivalent article in language B, all the sentences of language A are compared, using the vector representation, with all the sentences of the equivalent article in language B and if they are similar enough they are considered parallel segments. In this way, a parallel corpus is created to train the neural machine translation system. This self-supervised technique could be used for all the languages under study in this proposal, since they have Wikipedia versions. An exception would be Aranese, but tentative experiments could be done using the Occitan Wikipedia.
Self-supervised techniques have been successfully used in neural machine translation for resource-poor language pairs, but they usually do not achieve notable improvements for high-resource language pairs. Cheng (2021) presents a strategy to take advantage of self-monitoring techniques for high-resource language pairs, using a new process called encoder-decoder crossover to train with examples that come from both the parallel corpus and the monolingual corpus.
Unsupervised machine translation techniques use only monolingual texts in the source language and the target language. These texts do not have to be translations of each other. Generally, these techniques are based on word embeddings and on language models calculated separately for each of the languages. Word embeddings are representations of words using multidimensional vectors (with 200 or more dimensions). Each word of the language occupies a point in this multidimensional space. Similar words from the semantic point of view tend to occupy nearby positions within this space. The vector spaces for the two languages are different, but they maintain this relationship between similar words. Certain transformations and rotations of the spaces can be calculated that allow the two vector spaces to be aligned and overlapped. These transformations and rotations can be calculated from bilingual dictionaries or from words that are the same in both languages (numbers and proper names can be also used). Once the vector spaces are aligned and overlapped, the equivalent words in both languages will tend to be in very close positions. Following this principle, the system will be able to determine a first word-by-word translation model, although the model will contain some noise. Using this word-for-word model and with the help of monolingual language models, a synthetic parallel corpus can be created which, although it will again contain a lot of noise, can be used to train a machine translation system. This first system can be used to retranslate monolingual data and create a new synthetic parallel corpus, which in turn can be used to improve the translation system. This process can be repeated iteratively until no more improvements are achieved.
Artetxe et al. (2018) have developed Monoses, a complete set of tools for training unsupervised machine translation systems. In Artexte et al. (2019) a series of improvements to the system are introduced: on the one hand, using information from subwords; and on the other, using an improved statistical machine translation system to initialize a neural machine translation system that is optimized using backtranslation. Caswell, Chelba and Grangier (2019) propose to label the segments in the source language with a special mark to indicate whether they come from a real parallel corpus or from a synthetic corpus through backtranslation to mitigate possible adverse effects.
Lample et al. (2018) present two unsupervised systems: one statistical and one neural. The authors argue that unsupervised machine translation can be achieved by combining three components: (a) proper initialization of translation models; (b) language models and (c) iterative backtranslation. For (a) the authors suggest initializing the machine translation system using induced bilingual dictionaries; for (b), create language models with a sequence-to-sequence system and for (c), go from an unsupervised paradigm to a supervised one through the automatic generation of segment pairs through backtranslation.
Ruiter et al. (2021) demonstrate that supervised and unsupervised machine translation techniques can be combined to improve the results of supervised techniques alone.
The research interest on machine translation for language pairs with few resources is demonstrated by the organization of Shared Tasks and Workshops in the last years in various scientific conferences. In these events a common task for all participants is proposed. Guidelines and resources to perform the task are provided. To cite some of the most recent:
LoResMT 2021 MT Summit (https://sites.google.com/view/loresmt/), where a shared task was proposed for translation between the following languages: Taiwanese Sign Language - Traditional Chinese; English-Irish, English-Marathi. Shared Task Multilingual Low-Resource Translation for Indo-European Languages at EMNLP 2021 (http://statmt.org/wmt21/multilingualHeritage-translation-task.html), which included the following languages: to and from Icelandic, Norwegian Bokmäl and Swedish; Catalan-Occitan, Catalan-Romanian, Catalan-Italian. Shared Task Unsupervised MT and very low resource supervised MT at EMNLP 2021, which included three levels: unsupervised machine translation: Low Sorbian - German and German - Low Sorbian. supervised machine translation with very few resources: Low Sorbian - German and German - Low Sorbian. supervised machine translation with few resources: Russian-Chuvash and Chuvash-Russian. In the 2020 LoResMT workshop (https://aclanthology.org/2020.loresmt-1.0/), publications were published that included works for the following languages: Ashaninka, Assamese, Bambara, Bengali, Bhojpuri, Bicolano, Cebuano, English, Esperanto, French, Greek, Hiligaynon, Hindi, Ilocano, Kurdish, Manipuri, Pangasinense, Russian, Tagalog, Tamil and Vietnamese.
The LoResMT workshop is an annual event starting in 2018.
Given the interest in these topics, this project aims to organize a shared task and a workshop on machine translation for languages with few resources: Asturian, Aragonese and Aranese. This workshop will be allocated in LoResMT or in some other conference on Machine Translation. More details about this are given in section 5. Scientific -technic impact.
Artetxe, Mikel; Labaka, Gorka; Agirre, Eneko (2018) Unsupervised Statistical Machine Translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 3632- 3642
Artetxe, Mikel; Labaka, Gorka; Agirre, Eneko (2019) An Effective Approach to Unsupervised Machine Translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 194-203
Castilho, Sheila , Moorkens, Joss, Gaspari, Federico, Sennrich, Rico, Sosoni, Vilelmini, Georgakopoulou, Panayota, Lohar, Pintu, Way, Andy , Miceli Barone, Antonio Valerio and Gialama, Maria (2017) A comparative quality evaluation of PBSMT and NMT using professional translators. In: MT Summit XVI, 18-22 Sept 2017, Nagoya, Japan.
Caswell, Isaac; Chelba, Ciprian; Grangier, David. (2019) Tagged Back-Translation. In: Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers).. p. 53-63.
Cheng Yong; Wang, Wei, Jiang, Lu; Macherey, Wolfgang (2021). Self-supervised and Supervised Joint Training for Resource-rich Machine Translation. arXiv preprint arXiv:2106.04060.
European Observatory for Multilingualism (2020) Minority languages spoken or signed and inclusive spaces. https://observatoireplurilinguisme.eu/en-us/pole-of-research/call-for-papers-2 /177778537-appels-%C3%A0-communication-ou-projet-2020/13369-minority-languages-spoken-or-signed-and-inclusive-spaces (retrieved 12/12/2021)
Fan, Angela; Bhosale, Shruti; Schwenk, Holger; Ma, Zhiyi; El-Kishky, Ahmed; Goyal, Siddharth; Baines. Mandeep; Celebi, Onur; Wenzek, Guillaume; Chaudhary, Vishrav; Armand, Goyal, Naman; Birch, Tom; Liptchinsky, Vitaliy; Edunov, Sergey; Grave, Edouard Michael; Joulin, Auli. (2021). Beyond English-centric multilingual machine translation. Journal of Machine Learning Research, 22(107), 1-48.
Fan, Angela ∗ , Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, Armand Joulin. (2021). Beyond English-centric multilingual machine translation. Journal of Machine Learning Research, 22(107), 1-48.
Firat, Orhan, Kyunghyun Cho; Bengio Yoshua (2016) Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism. In Proceedings of the 5th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technology.
Johnson, Melvin; Schuster, Mike; Le, Quoc V.; Krikun, Maxim; Wu, Yonghui; Chen, Zhifeng; Thorat, Nikhil; Viégas, Fernanda; Wattenberg, Martin; Corrado, Greg; Hughes, Macduff; Dean, Jeffrey (2017). Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5, 339- 351.
Lample, Guillaume; Ott, Myle; Conneau, Alexis, Denoyer, Ludovic; Ranzato, Marc’Aurelio (2018). Phrase-Based & Neural Unsupervised Machine Translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 5039-5049).
Nguyen, Toan Q.; Chiang, Daniel (2017). Transfer learning across low-resource, related languages for neural machine translation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (pp. 296-301).
Rico Sennrich, Barry Haddow and Alexandra Birch (2016): Neural Machine Translation of Rare Words with Subword Units Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016). Berlin, Germany.
Ruiter, Dana; España-Bonet, Cristina; van Genabith, Josef (2019). Self-supervised neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 1828-1834)
Ruiter, Dana; Klakow ,Dietrich; van Genabith, Josef; España-Bonet, Cristina (2021). Integrating Unsupervised Data Generation into Self-Supervised Neural Machine Translation for Low-Resource Languages. arXiv preprint arXiv:2107.08772.
Sennrich, Rico, Haddow, Barry; Birch, Alexandra (2015). Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 86-96).
Zhang, Biao, Williams, Philip, Titov, Ivan, and Sennrich, Rico (2020). Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 1628-1639).
Zoph, Barret; Yuret, Deniz; May, Jonathan; Knight, Kevin (2016) Transfer learning for low-resource neural machine translation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin. Texas. pp. 1568-1575.