Speakly won the Best use of Google Cloud Platform in LA Hacks - March 2019.
Speakly is a program in python that uses Google's Speech-To-Text, Natural Language, and Vision API to track the speaker’s facial movements and emotions. It then utilizes this information to grade and highlight specific areas the speaker needs to improve.
Inspiration
Public-speaking is a common fear in society. Because of this, we decided to create Speakly to not only help ourselves address and hopefully overcome this fear, but assist others as well.
What it does
Speakly uses Google's Speech-To-Text, Natural Language, and Vision API to track the speaker’s facial movements and emotions. Our program then utilizes this information to grade and highlight specific areas the speaker needs to improve.
How WE built it
We used multiple laptops running python code to implement a local host UI in order to display the results of our back-end algorithms given a video (.mp4) file.
Challenges WE ran into
The initial planning of this project was fairly easy, but we ran into a lot of trouble attempting to implement Google’s cloud APIs. Google’s Speech to Text APIs gave us a slew of issues, but once we were able to figure out how to utilize the APIs, we only needed to implement our own code to convert this information into usable data. Unfortunately, we also ran into run-time errors, but as we increased the effectiveness of our algorithms, we were able to gradually decrease these errors.
Accomplishments that WE’RE proud of
We are proud of the fact our program was able to successfully grade the speaker and give a fairly accurate score. Additionally, our program was able to give the user targeted advice, which would increase user return to our program.
What WE learned
Back end team: We learned how to implement Google’s Vision, Speech to Text, and Sentiment Analysis APIs, and how to write and execute scripts to prepare the data for our main algorithm. We also gained experience in Python. Front end team: We had no prior experience with HTML/CSS, so we had the chance to learn how to develop the front end of an application. We also were able to learn about UI/UX, implementing the website with php to create the final product.
What's next for Speakly
We are proud of what we accomplished, but there are more improvements that can be made. We would like to increase the number of video frames that the program analyzes to produce more accurate results. At the same time, the grading algorithm can be tweaked and improved.