Voice Recording & Real-Time Translation Application

A full-stack application that records user voice input and translates each spoken sentence using AI API via a real-time WebSocket connection.

Project Structure

webapi-test/
├── frontend/          # React + TypeScript + Vite
│   ├── src/
│   │   ├── components/    # UI components
│   │   ├── context/        # Context API for state management
│   │   ├── hooks/          # Custom React hooks
│   │   ├── services/       # API/WebSocket services
│   │   └── types/          # TypeScript interfaces
│   ├── package.json
│   └── vite.config.ts
├── backend/           # Node.js + Express + Socket.io
│   ├── src/
│   │   ├── config/         # Configuration files
│   │   ├── services/       # Business logic (translation service)
│   │   ├── types/          # TypeScript interfaces
│   │   └── index.ts        # Server entry point
│   ├── package.json
│   └── tsconfig.json
└── README.md

Features

• Voice Recording: Uses MediaRecorder API to capture audio from user's microphone
• Real-time Communication: WebSocket connection via Socket.io for instant translation updates
• Sentence Detection: Pause-based detection to identify complete sentences
• AI Translation: Integration with OpenAI API (Whisper for transcription, GPT for translation)
• Mock Mode: Works without OpenAI API key using mock translations
• Error Handling: Comprehensive error handling for microphone permissions, connection issues, and API failures
• Modern UI: Clean and responsive interface with real-time status indicators

Prerequisites

• Node.js: Version 18 or higher (required for File API support in backend)
• npm: Package manager
• Microphone: For voice recording functionality
• OpenAI API Key (optional): For real transcription and translation. Without it, the app uses mock mode.

Setup Instructions

Backend Setup

1. Navigate to the backend directory:

cd backend

2. Install dependencies:

npm install

3. Create a .env file in the backend directory (optional):

PORT=3001
OPENAI_API_KEY=your_openai_api_key_here
CORS_ORIGIN=http://localhost:5173
TRANSLATION_TARGET_LANGUAGE=en

Note: If you don't provide an OPENAI_API_KEY, the application will run in mock mode with simulated translations.

4. Start the development server:

npm run dev

The backend server will start on http://localhost:3001 (or the port specified in your .env file).

Frontend Setup

1. Navigate to the frontend directory:

cd frontend

2. Install dependencies:

npm install

3. Start the development server:

npm run dev

The frontend application will start on http://localhost:5173 (default Vite port).

4. Open your browser and navigate to http://localhost:5173

How to Use

1. Start the Backend: Ensure the backend server is running on port 3001
2. Start the Frontend: Run the frontend development server
3. Allow Microphone Access: When prompted, allow the browser to access your microphone
4. Check Connection: Verify the connection status indicator shows "Connected" (green)
5. Start Recording: Click the "Start Recording" button
6. Speak: Speak clearly into your microphone. The app will detect sentence pauses and send audio for translation
7. View Translations: Translated sentences will appear in real-time below the recording controls
8. Stop Recording: Click "Stop Recording" when finished
9. Clear: Use the "Clear" button to remove all translations

Technical Decisions & Trade-offs

Audio Processing

• Decision: Use MediaRecorder API with pause-based sentence detection
• Trade-off: Simpler implementation but less accurate than advanced speech recognition. Sentence detection relies on silence thresholds which may not always align with natural speech patterns.

Real-time Communication

• Decision: Use Socket.io for WebSocket communication
• Trade-off: Socket.io adds overhead compared to native WebSockets but provides better cross-browser compatibility and automatic reconnection handling.

State Management

• Decision: Use React Context API for global state
• Trade-off: Context API is simpler for this use case but could become less performant with very large state. For a production app with more complexity, Zustand or Redux might be better.

Translation Service

• Decision: Support both OpenAI API and mock mode
• Trade-off: Mock mode allows development without API costs but doesn't provide real transcription. The mock mode generates random sentences rather than actual speech-to-text.

Sentence Detection

• Decision: Pause-based detection (1.5 seconds of silence)
• Trade-off: Simple and works for most cases, but may split sentences incorrectly or combine multiple sentences if pauses are inconsistent.

Audio Format

• Decision: Convert audio to base64 for transmission
• Trade-off: Base64 encoding increases payload size by ~33%, but it's simpler than streaming binary data. For production, consider using binary WebSocket frames or chunked uploads.

Assumptions

1. Input Language: The application assumes input speech is in English. The backend is configured to transcribe English audio. This can be changed in the translation service.

2. Browser Support: The application requires modern browsers with:

   • MediaRecorder API support
   • WebSocket support
   • Microphone access permissions

3. Network: Assumes frontend and backend are on the same network or localhost during development.

4. Node.js Version: Backend requires Node.js 18+ for File API support. For older versions, the app will fall back to mock mode for audio transcription.

5. Microphone Quality: Assumes reasonable microphone quality. Poor quality microphones may result in inaccurate transcriptions.

Error Handling

The application includes comprehensive error handling for:

• Microphone Permissions: Clear error messages when access is denied
• Connection Issues: Automatic reconnection attempts with status indicators
• API Failures: Graceful fallback to mock mode if OpenAI API fails
• Browser Compatibility: Checks for required APIs before attempting to use them

Development Scripts

Backend

• npm run dev: Start development server with hot reload
• npm run build: Build TypeScript to JavaScript
• npm start: Run production build

Frontend

• npm run dev: Start Vite development server
• npm run build: Build for production
• npm run preview: Preview production build

Future Improvements

1. Speech Recognition: Integrate Web Speech API for client-side transcription before sending to backend
2. Language Selection: Allow users to select source and target languages
3. Audio Visualization: Add waveform visualization during recording
4. History: Persist translation history in local storage or database
5. Export: Allow users to export translations as text files
6. Better Sentence Detection: Implement more sophisticated NLP-based sentence detection
7. Streaming: Use streaming audio for lower latency
8. Multiple Language Support: Support multiple source languages for transcription

License

ISC