Interactive Reading App 📖

Overview

The Interactive Reading App is designed to enhance reading and comprehension skills through interactive tools, including text extraction, audio-to-text transcription, speech synthesis, and image/object recognition. Built with Streamlit and Google’s Gemini API, this app leverages machine learning to offer real-time feedback, enabling users to improve reading, pronunciation, and object identification abilities.

Real-World Use Case

This app provides tailored support for individuals with reading difficulties, such as dyslexia. By offering tools for text extraction, pronunciation guidance, and structured feedback, the app aims to create a user-friendly environment for building reading confidence and comprehension. It acts as an accessible reading coach, helping users improve their reading and verbal skills at their own pace.

Features

• Text Extraction from Images: Extracts text from images for reading comprehension exercises.
• Image and Object Identification: Recognizes objects and scenes within an image and allows users to practice describing them.
• Audio-to-Text Transcription: Transcribes user-provided audio to evaluate reading performance.
• Text-to-Speech: Converts evaluation feedback to audio, providing users with a spoken assessment.
• Evaluation & Feedback: Offers real-time evaluation on reading accuracy, pronunciation, and object identification with a concise and structured feedback report.

Workflow

1. Input: Users upload images or audio of text passages.
2. Processing:
   • Text is extracted from images for reading.
   • Audio is transcribed and analyzed for reading accuracy.
   • Objects in images are identified for practice in verbal description.
3. Output: Users receive a feedback report with visual and audio components, enhancing both reading and comprehension skills.

Installation

Prerequisites

• Python 3.7 or later
• Google Cloud SDK for managing API keys and credentials.

Required Packages

Install required packages using:

pip install streamlit google-cloud google-auth google-auth-oauthlib pillow numpy

Configuration

Set up Google Cloud credentials:

1. Create a Google Cloud project and enable the necessary APIs (Text-to-Speech, Gemini API).
2. Store your service account key JSON in Streamlit secrets.

Example configuration in secrets.toml:

[gcp_service_account]
type = "service_account"
project_id = "<your_project_id>"
private_key_id = "<your_private_key_id>"
private_key = "<your_private_key>"
client_email = "<your_client_email>"
client_id = "<your_client_id>"
auth_uri = "https://accounts.google.com/o/oauth2/auth"
token_uri = "https://oauth2.googleapis.com/token"
auth_provider_x509_cert_url = "https://www.googleapis.com/oauth2/v1/certs"
client_x509_cert_url = "<your_client_cert_url>"

GEMINI_API_KEY = "<your_gemini_api_key>"

Example configuration in config.toml:

[client]
showSidebarNavigation = false

Usage

1. Launch the app: Run the Streamlit app using streamlit run app.py.
2. Navigate through pages:
   • Start by uploading an image or taking a picture.
   • Proceed to the Reading and Comprehension section, where you can extract text and transcribe your reading.
   • Use Image and Object Identification to practice recognizing and describing images.
   • Q&A offers simple questions based on the text for reinforcement.
3. Receive Feedback: Review the reading and image identification evaluation reports and listen to audio feedback generated by the app.

Key Functions

• text_from_image(image_path): Extracts text content from images.
• image_recognition(image_path): Identifies and describes the content of an image.
• evaluate_image_recognition(response, observation): Evaluates accuracy of image recognition by comparing user response with system observation.
• text_from_audio(audio_path): Transcribes text from audio files.
• set_questions(text): Generates simple questions based on text for comprehension.
• text_to_wav(voice_name, text, output_filename): Converts feedback text into speech and saves as a WAV file.
• evaluate_passage_reading(passage, word): Provides reading evaluation with pronunciation guidance.

Live Demo & Video

• Live Demo: Try out the Interactive Reading App live.
• Video Demo: Watch a demo of the app in action.

Future Modifications & Enhanced Model Integration

To improve accuracy and user experience, future modifications will focus on integrating more advanced models and additional features. Current constraints with the Gemini Flash model limit the app's performance in nuanced reading assessment and object identification. Potential upgrades include:

• Model Enhancements: Integrating more advanced, state-of-the-art language models as they become accessible, such as the more powerful Gemini or similar LLMs. These models can provide deeper contextual understanding and better pronunciation analysis, allowing for more personalized feedback.

• Advanced Speech and Object Recognition: Utilizing more sophisticated audio and visual processing models to refine reading evaluations and object recognition capabilities, potentially offering feedback on tone, intonation, and specific object details.

• Improved Pronunciation and Fluency Scoring: Leveraging models with robust NLP capabilities to provide nuanced insights into fluency, pacing, and pronunciation, enhancing learning for users with reading challenges.

• User Customization: Adding options to personalize feedback types and formats based on individual user needs, supported by more flexible AI models.

Contributing

1. Fork the repository and create a new branch for your feature.
2. Make your changes, and submit a pull request with a detailed description.

License

This project is licensed under the MIT License.

Acknowledgments

• Google Cloud for providing the APIs