Assist 🗣

A tool to create & manage audio descriptions.

• 🚀 Experimenting with using react everywhere.
• 💻 The main focus is on the desktop / electron app.

Background 🐣

Audio description, also referred to as a video description, described video, or more precisely called a visual description, is a form of narration used to provide information surrounding key visual elements in a media work (such as a film or television program, or theatrical performance) for the benefit of blind and visually impaired consumers. [source: wikipedia]

When talking to a friend who writes audio descriptions here in Berlin, I often heard mention of the lack of tools (at least inexpensive / open source ones) that could help with their work.

Currently, everything is done manually:

• Write dialogs from the speech in the video with matching timestamps (subtitles?).
• Find "gaps" between the speech, where descriptions could be inserted, measure the duration.
• Write description matching these gaps' scene and durartion, annotate whether the speaker mush speak it at a normal, fast, or super-fast pace etc.
• Rewind / Forward videos manually each time to listen to / watch a specific part of the transcript.

There seem quite a few opportunities here to solve problems using technology! Hence this project was born.

Planned features 🦄

• Playback video with controls and keyboard shortcuts to match various requirements of the writers.
• Automate speech recognition and speaker identification via technologies such as AWS Transcribe
• Display transcription in editable and interactive UI, where each word in the transcription is linked to the position in the video and vice versa. Example: Check the Transcript tab of this wwdc video - clicking the text jumps to the related video position.

Setup 🤷

Development environment 👨‍💻👩‍💻

You will need to install the following. Versions don't need to match exactly, using higher version should be fine.

• node.js - Needed to setup and run JavaScript

Verify:

> node -v
v10.20.1

• ffmpeg: A powerful library for editing videos from the command line!

Verify:

> ffmpeg -version
ffmpeg version 4.2.3

• yarn package manager for node (installed globally)

Install via node:

npm install -g yarn

Verify:

> yarn -v
1.22.4

The project is currently being developed on MacOS and there might be issues running it in other environments. Feel free to add an Issue or create a Pull Request! 😀

Download & Install Project 🛠

• Clone this repo

git clone git@github.com:mouselangelo/assist.git

• Install project dependencies

yarn install

Run 🎉

Currently only the electron app works - the others build and run but functionality maybe stubbed.

# desktop / electron app
yarn expo-electron start

#### others ( WIP, not all features implemented)

# web app
yarn web

# ios
yarn ios

# android (connect device or start emulator first)
yarn android

# interactive interface:
yarn start

Code 🎯

• typescript - is the JavaScript "flavour" of choice 🍦
• eslint / prettier - let's keep things pretty 💄
• testing (jest?) - currently non-existent, hopefully coming soon! 💣
• redux - also coming soon 🎛

React (Native) Everywhere 🤹‍♂️

We are building an electron desktop app using react-native (via react-native-web)! 🙊

Yeah, it's kind of experimental - so expect things to not work at times. However, there is awesome work already done by the amazing React and JS community! 🙏

So, it's been a fun adventure, and so far - it works! 🏄‍♂️

Here are some of the awesome frameworks / libraries this is built upon:

Core:

• expo / react-native
• react-native-web 🖤
• electron
• react-native-paper

Also:

• react-navigation
• react-native-vector-icons

Contributing

I'm still figuring out where this project is going, but if you are interested in contributing, just do it! Add an Issue or make a Pull Request!