Learning Chinese has never been easier. With PinYinTir, you can scan an image of Chinese text and understand it immediately!
This is the backend repo. Frontend repo can be found here.
Table of Contents
Mandarin Chinese is one of the most widely spoken languages in the world, with over 900 million native speakers and close to 200 million second language speakers, as per the 2022 Edition of Ethnologue.
The problem with languages like Chinese is that it is not a phonetic language. Hanyu Pinyin, often abbreviated to pinyin, is the official romanization system for Standard Mandarin Chinese. It is often used to teach Mandarin Chinese, which is written using the standard Chinese characters and the pronounciations of these characters are demonstrated through their according pinyin romanization.
A problem then arises: how does one go about learning or pronouncing a new character that one encounters without pinyin?
Traditional means like dictionaries are not a viable option as they require one to know either the pinyin or the radicals (i.e. the different strokes of the character) to be able to look up the character.
Newer means like Google Translate provide some help in this regard but are lacking also, as the pinyin and pronounciation are not the main focus. Rather, the focus is on the translation and it is really only helpful if you want to know just the meaning of the words in another language.
We present PinYinTir. A mobile app that provides the pinyin, translation, and pronounciation of any Chinese text scanned via the camera through the use of optical character recognition. These are overlaid on the original image for easy reference and users can save phrases and/or images for future reference. Look no further for your solution to learning Chinese easily!
1. Once the app loads, you should be brought to the Scan page and be greeted with a camera view. Scan an image that includes Chinese text. The pinyin of any detected Chinese text will be shown as an overlay on the image. You can tap on each individual phrase to see the English translation and text-to-speech functionality.
2. Although this application can be used without any sign up needed, signing up for an account will unlock the other features - mainly to save phrases and images. Click on Log In on the nav bar to log in or sign up.
3. Upon log in, you will be brought back to the Scan page. This time, the nav bar should be updated with additional pages - Images, Phrases, and Settings.
4. Now, when any image is scanned and the results shown, there is an option to save this image. Also, when viewing individual phrases, there is also an option to save just this phrase alone.
5. Go to the Images page to view all your past saved images.
6. Go to the Phrases page to view all your past saved phrases. Here, you can create categories for these phrases and sort them accordingly.
7. Go to the Settings page to adjust the pitch and speed of the text-to-speech voice. You can also log out here.
Frontend
Backend
Functionality
To get a local copy up and running follow these simple example steps.
- npm
npm install npm@latest -g
- Install Expo Go on your mobile device from either App Store (Apple) or Play Store (Android
Backend
The backend is hosted on Heroku already and no setup is necessary. Please contact the developers if you wish to get a local copy of the backend repo running.
Frontend
- Clone the repo
- Create .env file
REACT_APP_BACKEND=https://chinese-ar-backend.herokuapp.com- Install NPM packages
npm i
- Run frontend
npm start
- Scan generated QR code with camera (iOS) or Expo Go app (Android)
Jia Hao: GitHub - lim.jiahaoo@gmail.com
Dominique: GitHub - dominiqueyeo@hotmail.com
Tristan: GitHub - teo.jia.jun.29@gmail.com
To start the backend:
"npm run devStart"
To seed the db:
Run "npm run seed"
- Mongoose hosted on MongoDB Atlas
- Helper functions to upload and download images from S3 are stored in s3.js
- Controller functions for upload/download are stored in userCtrl.js
- To upload images (frontend > express > AWS S3 bucket):
const result = await axios.post(
`localhost:3008/api/images`,
formData,
{
headers: { "Content-Type": "multipart/form-data" },
}
);
- To download and read image data (AWS S3 bucket > express > frontend):
<img src={`${REACT_APP_BACKEND}/images/05e735782f00bba93f01df9335ecde0a`} ></img>