MeetDot is a videoconferencing system with live translation captions overlaid onscreen. The system aims to facilitate conversation between people who speak different languages, reducing communication barriers between multilingual participants.
Currently, our system supports speech and captions in four languages (English, Chinese, Spanish, Portuguese) and combines automatic speech recognition (ASR) and machine translation (MT) in a cascade.
- Smooth scrolling captions to reduce flicker and therefore cognitive load
- Modular architecture to integrate different ASR and MT services
- Integrated evaluation suite to optimize metrics such as accuracy, latency and erasure
- Word-guessing game for an extrinsic measure of end-to-end performance
- conda 64-bit with Python3.7+
- Tested on Chrome and Firefox
Create a conda environment that is Python 3.7 or 3.8 (newer versions will probably also work)
conda create -n streaming_translation python=3.7.9conda activate streaming_translation
cd backendpip install -r requirements.txt, from your conda environment.pip install -e .- Under the repository's root directory, execute
python env_setup.py --copy-keys. Sets environment variables neccessary to run APIs and copies credentials. It also installs git commit hooks. If not running on our shared server, seepython env_setup.py --help
In MeetDot, we provide ASR/MT integrations with Google Cloud so it can be run without access to our servers or internal APIs. If not running on our shared server, do the following:
- Go to Google Cloud and get a custom google api key link and copy it into
backend/resources/credentials.json - We use API from daily.co to optimize real-time video call quality. Create an account with daily.co, and add your URL and key to
.env.
In MeetDot, we uses dotenv in the frontend and backend.
Variables can be set by editing the .env file or on the command line BACKEND_PORT=5888 python src/app.py.
See .env.default for a list of possible keys.
python src/app.py(run with the--debugflag to restart on code changes, useful for development)
cd frontendyarn installyarn run serve --port=8080(You can use--mode=productionto serve a minified bundle)- Open localhost:8080 in browser.
cd backend/pytest
We use pre-commit for managing git hooks.
Run: pre-commit install to set it up.
This should run the formatter (Prettier for javascript) and syntax checks whenever you commit.
It should be set up to update itself and library installation, but see .pre-commit-config.yaml for more information.
- Download test data
cd backend/python src/services/evaluation/evaluate.py
In the first time running MeetDot system, the browser needs microphone permission to run the demo. Modern browsers prevent unsecure websites from accessing the microphone.
Local environments (localhost) are considered secure. For remote deployment, you will need to do one of the following:
- Serve the website over HTTPS. Set paths to your key and cert files in
backend/resources/keys.json. The filenames are defaults generated by LetsEncrypt. - Use a self-signed certificate and have your users click through the browser warnings. (Not recommended)
- Have users enable experimental flags to bypass browser security setttings. (Also not recommended). In Chrome, this is
chrome://flags#unsafely-treat-insecure-origin-as-secure. In Firefox, this ismedia.devices.insecure.enabledinabout:config.
Our deployment script copies files into $DEPLOY_ROOT (e.g. /srv/example.com, set in gitlab variables).
Static frontend files are served from $DEPLOY_ROOT/dist with nginx.
Backend files are saved in $DEPLOY_ROOT/backend and served with a bare flask-socketio server.
These processes (nginx, the backend) are managed by systemd scripts.