End-to-end machine translation for low-resource languages spoken by the indigenous peoples of Taiwan.
This project was created by Hunter Scheppat, Dylan Leddy, Junhao Yan, Zelig Riyanto, and Christopher DiScala at Boston College.
- Google Drive account
- Colab compute hours
- Optional DeepL API key
BeautifulSoup
Selenium
Sacremoses
Sacrebleu
Sentencepiece
Transformers v.4.33
Datasets
Ipynb
Import-ipynb
Installations are handled through the notebooks.
- Download the files in src and upload them to your Google Drive. Modify code directly in the Colab notebooks.
- We include our raw data (data/raw) - feel free to inspect them. Though, the only data you need for working with this project are the CSV files in data/unprocessed and data/processed.
- If you plan on using pivot scripts, you'll need your own DeepL API key. Scripts used for pivoting are src/Pivoter and src/scripts/ScrapeDataSources.
- Translator.ipynb is unfinished. Instead, use NLLB-fine-tuning.ipynb in src/scripts. src/scripts/ScrapeDataSources.ipynb contains more comprehensive code for extracting data than Scraper.ipynb.
- Amis to English | hunterschep/amis-english
- English to Amis | dylan-leddy/nllb-eng-ami-reverse
- Paiwan to English | hunterschep/paiwan-english
- English to Paiwan | hunterschep/english-paiwan