Update readme, add demo

hassanhub · Nov 6, 2017 · b4b510e · b4b510e
1 parent 351e610
commit b4b510e
Show file tree

Hide file tree

Showing 132 changed files with 26 additions and 1 deletion.
diff --git a/README.md b/README.md
@@ -1 +1,26 @@
-# LipReading
+# LipReading
+
+This is the keras implementation of *Lip2AudSpec: Speech reconstruction from silent lip movements video.* 
+
+![Main Network](figures/Network_main.png)
+
+### Abstract
+In this study, we propose a deep neural network for reconstructing intelligible speech from silent lip movement videos. We use auditory spectrogram and its corresponding sound generation method which preserves pitch information resulting in a more natural sounding reconstructed speech. Our proposed network consists of an autoencoder to extract bottleneck features from the auditory spectrogram which is then used as target to our main lip reading network comprising of CNN, LSTM and fully connected layers. Our experiments show that the autoencoder is able to reconstruct the original auditory spectrogram with a 98% correlation and also improves the quality of reconstructed speech from the main lip reading network.
+Our model, trained jointly on different speakers is able to extract individual speaker characteristics and gives promising results of reconstructing intelligible speech with superior word recognition accuracy.
+
+### Demo
+
+You can find all demo files [here](demo/).
+
+A few samples of the network output are given below:
+
+**Sample 1**
+
+[![Sample1](https://img.youtube.com/vi/Op7Z9KH5Fis/0.jpg)](https://youtu.be/Op7Z9KH5Fis "Sample1_s1")
+
+**Sample 2**
+
+[![Sample2](https://img.youtube.com/vi/O0Gfb-1lu2k/0.jpg)](https://youtu.be/O0Gfb-1lu2k "Sample2_s29")
+
+
+
diff --git a/demo/Icon b/demo/Icon
diff --git a/demo/autoencoder (targets to network)/Icon b/demo/autoencoder (targets to network)/Icon
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Icon b/demo/autoencoder (targets to network)/coded_decoded/Icon
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_1.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_1.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_10.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_10.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_11.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_11.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_12.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_12.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_13.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_13.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_14.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_14.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_15.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_15.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_16.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_16.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_17.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_17.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_18.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_18.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_19.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_19.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_2.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_2.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_20.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_20.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_3.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_3.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_4.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_4.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_5.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_5.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_6.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_6.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_7.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_7.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_8.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_8.wav
diff --git a/demo/autoencoder (targets to network)/coded_decoded/Sample_9.wav b/demo/autoencoder (targets to network)/coded_decoded/Sample_9.wav
diff --git a/demo/autoencoder (targets to network)/originals/Icon b/demo/autoencoder (targets to network)/originals/Icon
diff --git a/demo/autoencoder (targets to network)/originals/Sample_1.wav b/demo/autoencoder (targets to network)/originals/Sample_1.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_10.wav b/demo/autoencoder (targets to network)/originals/Sample_10.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_11.wav b/demo/autoencoder (targets to network)/originals/Sample_11.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_12.wav b/demo/autoencoder (targets to network)/originals/Sample_12.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_13.wav b/demo/autoencoder (targets to network)/originals/Sample_13.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_14.wav b/demo/autoencoder (targets to network)/originals/Sample_14.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_15.wav b/demo/autoencoder (targets to network)/originals/Sample_15.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_16.wav b/demo/autoencoder (targets to network)/originals/Sample_16.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_17.wav b/demo/autoencoder (targets to network)/originals/Sample_17.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_18.wav b/demo/autoencoder (targets to network)/originals/Sample_18.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_19.wav b/demo/autoencoder (targets to network)/originals/Sample_19.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_2.wav b/demo/autoencoder (targets to network)/originals/Sample_2.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_20.wav b/demo/autoencoder (targets to network)/originals/Sample_20.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_3.wav b/demo/autoencoder (targets to network)/originals/Sample_3.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_4.wav b/demo/autoencoder (targets to network)/originals/Sample_4.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_5.wav b/demo/autoencoder (targets to network)/originals/Sample_5.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_6.wav b/demo/autoencoder (targets to network)/originals/Sample_6.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_7.wav b/demo/autoencoder (targets to network)/originals/Sample_7.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_8.wav b/demo/autoencoder (targets to network)/originals/Sample_8.wav
diff --git a/demo/autoencoder (targets to network)/originals/Sample_9.wav b/demo/autoencoder (targets to network)/originals/Sample_9.wav
diff --git a/demo/lip2audspec/Icon b/demo/lip2audspec/Icon
diff --git a/demo/lip2audspec/original/Icon b/demo/lip2audspec/original/Icon
diff --git a/demo/lip2audspec/original/Sample_1.wav b/demo/lip2audspec/original/Sample_1.wav
diff --git a/demo/lip2audspec/original/Sample_10.wav b/demo/lip2audspec/original/Sample_10.wav
diff --git a/demo/lip2audspec/original/Sample_11.wav b/demo/lip2audspec/original/Sample_11.wav
diff --git a/demo/lip2audspec/original/Sample_12.wav b/demo/lip2audspec/original/Sample_12.wav
diff --git a/demo/lip2audspec/original/Sample_13.wav b/demo/lip2audspec/original/Sample_13.wav
diff --git a/demo/lip2audspec/original/Sample_14.wav b/demo/lip2audspec/original/Sample_14.wav
diff --git a/demo/lip2audspec/original/Sample_15.wav b/demo/lip2audspec/original/Sample_15.wav
diff --git a/demo/lip2audspec/original/Sample_16.wav b/demo/lip2audspec/original/Sample_16.wav
diff --git a/demo/lip2audspec/original/Sample_17.wav b/demo/lip2audspec/original/Sample_17.wav
diff --git a/demo/lip2audspec/original/Sample_18.wav b/demo/lip2audspec/original/Sample_18.wav
diff --git a/demo/lip2audspec/original/Sample_19.wav b/demo/lip2audspec/original/Sample_19.wav
diff --git a/demo/lip2audspec/original/Sample_2.wav b/demo/lip2audspec/original/Sample_2.wav
diff --git a/demo/lip2audspec/original/Sample_20.wav b/demo/lip2audspec/original/Sample_20.wav
diff --git a/demo/lip2audspec/original/Sample_3.wav b/demo/lip2audspec/original/Sample_3.wav
diff --git a/demo/lip2audspec/original/Sample_4.wav b/demo/lip2audspec/original/Sample_4.wav
diff --git a/demo/lip2audspec/original/Sample_5.wav b/demo/lip2audspec/original/Sample_5.wav
diff --git a/demo/lip2audspec/original/Sample_6.wav b/demo/lip2audspec/original/Sample_6.wav
diff --git a/demo/lip2audspec/original/Sample_7.wav b/demo/lip2audspec/original/Sample_7.wav
diff --git a/demo/lip2audspec/original/Sample_8.wav b/demo/lip2audspec/original/Sample_8.wav
diff --git a/demo/lip2audspec/original/Sample_9.wav b/demo/lip2audspec/original/Sample_9.wav
diff --git a/demo/lip2audspec/predicts/Icon b/demo/lip2audspec/predicts/Icon
diff --git a/demo/lip2audspec/predicts/Sample_1.wav b/demo/lip2audspec/predicts/Sample_1.wav
diff --git a/demo/lip2audspec/predicts/Sample_10.wav b/demo/lip2audspec/predicts/Sample_10.wav
diff --git a/demo/lip2audspec/predicts/Sample_11.wav b/demo/lip2audspec/predicts/Sample_11.wav
diff --git a/demo/lip2audspec/predicts/Sample_12.wav b/demo/lip2audspec/predicts/Sample_12.wav
diff --git a/demo/lip2audspec/predicts/Sample_13.wav b/demo/lip2audspec/predicts/Sample_13.wav
diff --git a/demo/lip2audspec/predicts/Sample_14.wav b/demo/lip2audspec/predicts/Sample_14.wav
diff --git a/demo/lip2audspec/predicts/Sample_15.wav b/demo/lip2audspec/predicts/Sample_15.wav
diff --git a/demo/lip2audspec/predicts/Sample_16.wav b/demo/lip2audspec/predicts/Sample_16.wav
diff --git a/demo/lip2audspec/predicts/Sample_17.wav b/demo/lip2audspec/predicts/Sample_17.wav
diff --git a/demo/lip2audspec/predicts/Sample_18.wav b/demo/lip2audspec/predicts/Sample_18.wav
diff --git a/demo/lip2audspec/predicts/Sample_19.wav b/demo/lip2audspec/predicts/Sample_19.wav
diff --git a/demo/lip2audspec/predicts/Sample_2.wav b/demo/lip2audspec/predicts/Sample_2.wav
diff --git a/demo/lip2audspec/predicts/Sample_20.wav b/demo/lip2audspec/predicts/Sample_20.wav
diff --git a/demo/lip2audspec/predicts/Sample_3.wav b/demo/lip2audspec/predicts/Sample_3.wav
diff --git a/demo/lip2audspec/predicts/Sample_4.wav b/demo/lip2audspec/predicts/Sample_4.wav
diff --git a/demo/lip2audspec/predicts/Sample_5.wav b/demo/lip2audspec/predicts/Sample_5.wav
diff --git a/demo/lip2audspec/predicts/Sample_6.wav b/demo/lip2audspec/predicts/Sample_6.wav
diff --git a/demo/lip2audspec/predicts/Sample_7.wav b/demo/lip2audspec/predicts/Sample_7.wav
diff --git a/demo/lip2audspec/predicts/Sample_8.wav b/demo/lip2audspec/predicts/Sample_8.wav
diff --git a/demo/lip2audspec/predicts/Sample_9.wav b/demo/lip2audspec/predicts/Sample_9.wav
diff --git a/demo/vid2speech/Icon b/demo/vid2speech/Icon
diff --git a/demo/vid2speech/originals_vid2speech/Icon b/demo/vid2speech/originals_vid2speech/Icon
diff --git a/demo/vid2speech/originals_vid2speech/s1_000459.wav b/demo/vid2speech/originals_vid2speech/s1_000459.wav
diff --git a/demo/vid2speech/originals_vid2speech/s1_000586.wav b/demo/vid2speech/originals_vid2speech/s1_000586.wav
diff --git a/demo/vid2speech/originals_vid2speech/s1_000660.wav b/demo/vid2speech/originals_vid2speech/s1_000660.wav
diff --git a/demo/vid2speech/originals_vid2speech/s1_000678.wav b/demo/vid2speech/originals_vid2speech/s1_000678.wav
diff --git a/demo/vid2speech/originals_vid2speech/s1_000700.wav b/demo/vid2speech/originals_vid2speech/s1_000700.wav
diff --git a/demo/vid2speech/originals_vid2speech/s29_000065.wav b/demo/vid2speech/originals_vid2speech/s29_000065.wav
diff --git a/demo/vid2speech/originals_vid2speech/s29_000112.wav b/demo/vid2speech/originals_vid2speech/s29_000112.wav
diff --git a/demo/vid2speech/originals_vid2speech/s29_000671.wav b/demo/vid2speech/originals_vid2speech/s29_000671.wav
diff --git a/demo/vid2speech/originals_vid2speech/s29_000806.wav b/demo/vid2speech/originals_vid2speech/s29_000806.wav
diff --git a/demo/vid2speech/originals_vid2speech/s29_000837.wav b/demo/vid2speech/originals_vid2speech/s29_000837.wav
diff --git a/demo/vid2speech/originals_vid2speech/s2_000252.wav b/demo/vid2speech/originals_vid2speech/s2_000252.wav
diff --git a/demo/vid2speech/originals_vid2speech/s2_000521.wav b/demo/vid2speech/originals_vid2speech/s2_000521.wav
diff --git a/demo/vid2speech/originals_vid2speech/s2_000591.wav b/demo/vid2speech/originals_vid2speech/s2_000591.wav
diff --git a/demo/vid2speech/originals_vid2speech/s2_000671.wav b/demo/vid2speech/originals_vid2speech/s2_000671.wav
diff --git a/demo/vid2speech/originals_vid2speech/s2_000796.wav b/demo/vid2speech/originals_vid2speech/s2_000796.wav
diff --git a/demo/vid2speech/originals_vid2speech/s4_000189.wav b/demo/vid2speech/originals_vid2speech/s4_000189.wav
diff --git a/demo/vid2speech/originals_vid2speech/s4_000438.wav b/demo/vid2speech/originals_vid2speech/s4_000438.wav
diff --git a/demo/vid2speech/originals_vid2speech/s4_000448.wav b/demo/vid2speech/originals_vid2speech/s4_000448.wav
diff --git a/demo/vid2speech/originals_vid2speech/s4_000600.wav b/demo/vid2speech/originals_vid2speech/s4_000600.wav
diff --git a/demo/vid2speech/originals_vid2speech/s4_000774.wav b/demo/vid2speech/originals_vid2speech/s4_000774.wav
diff --git a/demo/vid2speech/predicts_vid2speech/Icon b/demo/vid2speech/predicts_vid2speech/Icon
diff --git a/demo/vid2speech/predicts_vid2speech/s1_000459.wav b/demo/vid2speech/predicts_vid2speech/s1_000459.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s1_000586.wav b/demo/vid2speech/predicts_vid2speech/s1_000586.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s1_000660.wav b/demo/vid2speech/predicts_vid2speech/s1_000660.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s1_000678.wav b/demo/vid2speech/predicts_vid2speech/s1_000678.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s1_000700.wav b/demo/vid2speech/predicts_vid2speech/s1_000700.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s29_000065.wav b/demo/vid2speech/predicts_vid2speech/s29_000065.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s29_000112.wav b/demo/vid2speech/predicts_vid2speech/s29_000112.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s29_000671.wav b/demo/vid2speech/predicts_vid2speech/s29_000671.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s29_000806.wav b/demo/vid2speech/predicts_vid2speech/s29_000806.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s29_000837.wav b/demo/vid2speech/predicts_vid2speech/s29_000837.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s2_000252.wav b/demo/vid2speech/predicts_vid2speech/s2_000252.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s2_000521.wav b/demo/vid2speech/predicts_vid2speech/s2_000521.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s2_000591.wav b/demo/vid2speech/predicts_vid2speech/s2_000591.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s2_000671.wav b/demo/vid2speech/predicts_vid2speech/s2_000671.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s2_000796.wav b/demo/vid2speech/predicts_vid2speech/s2_000796.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s4_000189.wav b/demo/vid2speech/predicts_vid2speech/s4_000189.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s4_000438.wav b/demo/vid2speech/predicts_vid2speech/s4_000438.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s4_000448.wav b/demo/vid2speech/predicts_vid2speech/s4_000448.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s4_000600.wav b/demo/vid2speech/predicts_vid2speech/s4_000600.wav
diff --git a/demo/vid2speech/predicts_vid2speech/s4_000774.wav b/demo/vid2speech/predicts_vid2speech/s4_000774.wav
diff --git a/figures/Network_main.png b/figures/Network_main.png