LiltLab

AI pronunciation coaching with phoneme-level analysis, articulatory feedback, and personalized practice.

Description

LiltLab is an AI-powered pronunciation training platform for English learners, accent coaches, actors, remote professionals, public speakers, and creators who want more than generic right-or-wrong speech feedback. Users can paste or type text, record themselves reading it aloud, and receive detailed analysis at the word, phoneme, and prosody level, along with AI-generated coaching and optional reference audio.

The project solves a common gap in language and speech tools: most products can tell you whether something sounds wrong, but not what sound broke down, how severe the mismatch was, or what to practice next. LiltLab combines phoneme recognition, articulatory-distance scoring, visual feedback, and coaching workflows to make speech practice measurable and actionable.

Table of Contents

• Features
• Tech Stack
• Architecture Overview
• Installation
• Usage
• Configuration
• Screenshots / Demo
• API / CLI Reference
• Tests
• Roadmap
• Contributing
• License
• Contact / Support

Features

• Real-time pronunciation analysis for custom text passages and word practice.
• Phoneme-level alignment with articulatory-distance scoring and top-k predictions.
• Per-word scoring, difficult-word tracking, and saved practice history.
• Prosody visualization for pitch, stress, pauses, and speech rate.
• AI coaching powered by Gemini for next-step feedback and practice guidance.
• Reference audio generation and side-by-side comparison using ElevenLabs.
• Multiple persistence modes with local storage, Supabase, or Neo4j.
• Local development mode that works without production auth and database services.
• Repo-specific Tessl skills for developer context and repeatable workflows.

Tech Stack

• Frontend: Next.js 16, React 19, TypeScript, Tailwind CSS 4.
• Backend: Next.js App Router API routes and a FastAPI analysis service.
• Speech and ML: wav2vec2 phoneme recognition, articulatory-distance scoring, local deterministic analysis fallback.
• Data and Auth: Supabase, Neo4j, Clerk.
• AI and Media: Gemini via Vercel AI SDK, ElevenLabs text-to-speech.
• Developer Tooling: Node test runner, ESLint, Python unittest, Tessl.

Architecture Overview

flowchart LR
  User[User] --> Client[Next.js Web App]
  Client --> WebAPI[Next.js API Routes]
  WebAPI --> AnalysisAPI[FastAPI Analysis Service]
  AnalysisAPI --> SpeechModel[Phoneme Recognition + Articulatory Scoring]
  WebAPI --> DB[(Supabase or Neo4j)]
  WebAPI --> Auth[Clerk]
  WebAPI --> TTS[ElevenLabs]
  WebAPI --> Coach[Gemini]

Loading

The user interacts with a Next.js web application that handles recording, playback, and analysis views. The web layer calls a FastAPI-based analysis service for phoneme and prosody feedback, stores practice data in Supabase or Neo4j, and integrates with Clerk, ElevenLabs, and Gemini for authentication, reference audio, and coaching.

Installation

Prerequisites

• Node.js 18 or later
• npm 9 or later
• Python 3.11 or later
• A Supabase project if you want relational persistence
• A Neo4j instance if you want graph persistence
• A Clerk application for production auth
• ElevenLabs and Google AI API keys for reference audio and coaching
• A Modal account if you want to deploy the full production analysis backend

1. Clone the repository

git clone https://github.com/MasteraSnackin/LiltLab.git
cd LiltLab

2. Create a Python environment

python3 -m venv .venv
source .venv/bin/activate
pip install fastapi uvicorn modal

3. Install web dependencies

cd web
npm install
cp .env.example .env.local
cd ..

4. Choose a data backend

Set DATA_BACKEND in web/.env.local to one of the following:

• local for no external database
• supabase for relational persistence
• neo4j for graph persistence

5. Optional Neo4j setup

cd web
npm run neo4j:apply-schema
npm run neo4j:smoke
cd ..

6. Optional production backend deployment

source .venv/bin/activate
modal setup
modal deploy api/app.py

Usage

Run locally with the built-in development backend

1. Update web/.env.local for local development:

DATA_BACKEND=local
LOCAL_DEV_MODE=true
LOCAL_DEV_USER_ID=local-dev-user
MODAL_API_URL=http://127.0.0.1:8000
LILTLAB_API_KEY=local-dev-key

2. Start the local FastAPI analysis service:

source .venv/bin/activate
python3 -m uvicorn api.local_dev_server:app --host 127.0.0.1 --port 8000

3. Start the web app:

cd web
npm run dev

4. Open http://localhost:3000, paste a passage, record audio, and review the generated analysis.

Run with Neo4j persistence

DATA_BACKEND=neo4j
LOCAL_DEV_MODE=false
NEO4J_URI=neo4j+s://<your-instance>
NEO4J_USERNAME=<your-username>
NEO4J_PASSWORD=<your-password>
NEO4J_DATABASE=neo4j

cd web
npm run neo4j:apply-schema
npm run neo4j:smoke
npm run dev

Practical example

Use the local backend directly to verify the analysis service:

curl -X POST http://127.0.0.1:8000/api/analyze \
  -H "X-API-Key: local-dev-key" \
  -F "audio=@sample.webm" \
  -F "target=The weather is beautiful today."

Configuration

The main configuration file is web/.env.local. Start from web/.env.example and fill in the values that match your environment.

Variable	Required	Description
DATA_BACKEND	Yes	Selects local, supabase, or neo4j.
LOCAL_DEV_MODE	Recommended for local setup	Enables local auth and storage fallbacks.
LOCAL_DEV_USER_ID	Local only	Fake user identifier used in local mode.
MODAL_API_URL	Yes	URL of the pronunciation analysis backend.
LILTLAB_API_KEY	Yes	Shared API key between the web app and analysis backend.
NEXT_PUBLIC_SUPABASE_URL	Supabase mode	Supabase project URL.
NEXT_PUBLIC_SUPABASE_ANON_KEY	Supabase mode	Supabase public client key.
SUPABASE_SERVICE_ROLE_KEY	Supabase mode	Server-side Supabase key used by API routes.
NEO4J_URI	Neo4j mode	Neo4j connection string.
NEO4J_USERNAME	Neo4j mode	Neo4j username.
NEO4J_PASSWORD	Neo4j mode	Neo4j password.
NEO4J_DATABASE	Neo4j mode	Neo4j database name, usually neo4j.
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY	Production auth	Clerk client key.
CLERK_SECRET_KEY	Production auth	Clerk server key.
GOOGLE_GENERATIVE_AI_API_KEY	Coaching	Gemini API key.
ELEVENLABS_API_KEY	Reference audio	ElevenLabs API key.
ELEVENLABS_VOICE_ID	Optional	Default reference voice.

Screenshots / Demo

• Live demo: <ADD LIVE DEMO URL HERE>
• Product walkthrough video: <ADD DEMO VIDEO URL HERE>

API / CLI Reference

LiltLab does not expose a standalone CLI today, but it does include web and backend HTTP endpoints that power the application.

Web API routes

Method	Endpoint	Purpose
POST	/api/analyze	Uploads audio, runs pronunciation analysis, stores results, and returns user and reference analysis.
POST	/api/coach	Generates coaching guidance for a practice result.
GET	/api/history	Returns saved practice sessions for the active user.
GET	/api/passages/:id	Returns a stored passage and related take data.
GET	/api/voices	Returns supported voice options for reference audio generation.

Local analysis backend

The local FastAPI service mirrors the production analysis surface and is useful for fast development without external infrastructure.

curl -X POST http://127.0.0.1:8000/api/analyze \
  -H "X-API-Key: local-dev-key" \
  -F "audio=@sample.webm" \
  -F "target=Practice makes progress."

Tests

LiltLab uses the Node test runner for frontend and server-side TypeScript tests, plus Python unittest for the local analysis backend.

cd web
npm test
npm run lint
npm run build

python3 -m unittest tests/test_local_dev_server.py

Roadmap

• Add production-ready deployment guides for Modal, Vercel, Supabase, and Neo4j.
• Expand adaptive coaching and personalized review loops for repeated mistakes.
• Introduce richer reference libraries for speeches, scripts, and accent targets.
• Improve observability, CI automation, and release workflows.
• Add more shareable progress views and learner analytics.

Contributing

Contributions are welcome. If you want to improve LiltLab:

1. Fork the repository.
2. Create a feature branch from main.
3. Make your changes with tests where appropriate.
4. Run the local test and lint commands before opening a pull request.
5. Open an issue or pull request at https://github.com/MasteraSnackin/LiltLab.

Please use issues for bug reports, feature requests, and implementation discussions before large changes.

License

This project is licensed under the CC BY-SA 4.0 License. See LICENSE for details.

Contact / Support

• Maintainer: MasteraSnackin
• GitHub: https://github.com/MasteraSnackin
• Repository: https://github.com/MasteraSnackin/LiltLab
• Issues: https://github.com/MasteraSnackin/LiltLab/issues
• Email: <ADD MAINTAINER EMAIL HERE>
• Website: <ADD PROJECT WEBSITE HERE>