Read My Lips

Lip reading is hard.

Directory structure

The pre_processing.py script is setup to handle a directory structure of:

data/align/<speaker>     # Align files for <speaker> go here.
data/videos/<speaker>    # Video files for <speaker> go here.
data/features/<speaker>  # Facial feature data for <speaker> is saved here. 

Saving facial feature data

First create the data/features/<speaker> directory for the speaker you're working with. Ensure you also have the align and video files for that speaker, as outlined above.

Now, all you need to so is run python pre_processing.py <speaker> to start generating and saving speaker data. You can stop at any time by pressing 'q' (it may take a few seconds to exit). You can pick up where you left off because the program is smart enough not to reprocess videos it already has data for.

Results

The paper describing our process and results can be found here.