CraftConnect

Multimodal AI platform that empowers traditional Indian folk artisans to create professional marketplace listings — through voice, vision, and cultural intelligence.

An artisan photographs their craft, answers a few voice questions in Hindi or Kannada, and walks away with a fully structured marketplace listing, rich cultural context, and a downloadable PDF report — without typing a single word.

---

Key Highlights

• Custom ViT-B/16 classifier fine-tuned on Indian handicraft imagery — 89.51% validation accuracy across 7 artforms
• Multilingual voice interview pipeline supporting Hindi and Kannada via Groq Whisper STT
• Visual grounding + RAG — AWS Bedrock Nova Lite extracts visual attributes; ChromaDB retrieves cultural heritage context from 131 documents
• Production-grade LLM routing — cascading fallback chain across Nova 2 → Groq → Gemini → Ollama → Mock
• Zero typing required — the entire artisan-facing workflow is voice and image driven
• PDF report + marketplace listing generated end-to-end from a single upload

---

📹 Demo

Demo video coming soon.

---

Supported Artforms & Languages

Artforms: Gond · Kalighat · Kangra · Kerala Mural · Madhubani · Pichwai · Warli

Voice interview languages: Hindi (hi) · Kannada (kn)

Conversation assistant & TTS: English (en)

---

System Architecture

Image Upload
     │
     ▼
CraftClassifierAgent        ← Custom ViT-B/16 (ONNX) — 89.51% validation accuracy
     │
     ▼
VisualGroundingAgent         ← AWS Bedrock Nova Lite (multimodal visual extraction)
     │
     ▼
RAGRetrieverAgent            ← ChromaDB + sentence-transformers (131 heritage docs)
     │
     ▼
StoryGenerationAgent         ← LLMRouter (Nova 2 → Groq → Gemini → Ollama)
     │
     ▼
ListingContentAssistant      ← LLMRouter (Nova 2 → Groq → Gemini → Ollama)
     │
     ▼
MarketplaceReadinessAgent    ← LLMRouter (quality validation + compliance check)
     │
     ▼
PDF Report + Marketplace Listing

Voice Interview Pipeline:

Voice Input (Hindi / Kannada)
     │
Groq Whisper STT (whisper-large-v3)
     │
CustomizationInterviewAgent
     │
Listing Generation via LLMRouter

LLMRouter Fallback Chain:

Nova 2 Lite → Groq (llama-3.3-70b) → Gemini (gemini-1.5-flash) → Ollama (mistral:7b) → Mock

---

Tech Stack

Layer	Technology
Backend	FastAPI 0.115.0, Python 3.10+, Uvicorn
Frontend	React 19.2.0, TypeScript 5.4.0, Vite, Tailwind CSS 3.4.16
Classification	Custom ViT-B/16 → ONNX Runtime 1.19.2
Visual Grounding	AWS Bedrock Nova Lite (apac.amazon.nova-lite-v1:0)
Listing Generation	AWS Bedrock Nova 2 Lite via LLMRouter
STT	Groq Whisper (whisper-large-v3) + faster-whisper (local fallback)
RAG	ChromaDB 1.5.2 + sentence-transformers
Primary DB	Supabase (missions, listings)
Interview DB	SQLite
PDF Generation	ReportLab 4.0+
i18n	i18next 25.7.3

---

Key Results

Metric	Result
ViT classifier accuracy	89.51% validation accuracy (Epoch 15 of 15)
Artforms supported	7 (Gond, Kalighat, Kangra, Kerala Mural, Madhubani, Pichwai, Warli)
Voice interview languages	2 (Hindi, Kannada)

---

Project Structure

CraftConnect/
├── app/                        # FastAPI backend
│   ├── agents/                 # AI agent pipeline
│   ├── routers/                # API endpoints
│   ├── services/               # PDF generation, STT, TTS
│   ├── llm_backends/           # Nova, Gemini, Groq, Ollama adapters
│   ├── database/               # SQLite interview DB
│   └── knowledge/              # RAG document loader
├── craftconnect-frontend/      # React frontend (Vite)
├── craft_classifier_agent/     # ViT ONNX inference pipeline
├── models/                     # Model artifacts (gitignored)
├── migrations/                 # SQLite schema migrations
├── scripts/                    # Benchmarks, QA, utilities
├── docs/                       # API guide, technical documentation
├── audio/                      # Interview question audio files
└── fonts/                      # PDF font assets

---

Quick Start

Prerequisites

• Python 3.10+
• Node.js 18+
• AWS account with Bedrock access (Nova Lite + Nova 2 Lite)
• Supabase project
• Groq API key

1. Clone and set up environment

git clone https://github.com/RohanLC1263/CraftConnect.git
cd CraftConnect
python -m venv .venv
source .venv/bin/activate        # Linux/Mac
# .venv\Scripts\activate         # Windows
pip install -r requirements.txt

2. Configure environment variables

cp .env.example .env
# Fill in your keys in .env

3. Download model files

The ViT classifier is hosted on HuggingFace:

pip install huggingface_hub
python -c "
from huggingface_hub import hf_hub_download
import shutil, os
os.makedirs('models/vit_paintings_v1', exist_ok=True)
for f in ['vit_best.onnx', 'vit_best.onnx.data', 'class_index.json']:
    path = hf_hub_download(repo_id='RohanLC/craftconnect-vit', filename=f)
    shutil.copy(path, f'models/vit_paintings_v1/{f}')
print('Model files downloaded.')
"

4. Run database migrations

python scripts/run_interview_migrations.py

5. Start the backend

uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

6. Start the frontend

cd craftconnect-frontend
npm install
npm run dev

• Frontend: http://localhost:5173
• Backend API: http://localhost:8000

Docker (alternative)

docker build -t craftconnect .
docker run --env-file .env.production -p 8080:8080 craftconnect

---

Environment Variables

Variable	Description
AWS_ACCESS_KEY_ID	AWS IAM access key
AWS_SECRET_ACCESS_KEY	AWS IAM secret key
AWS_SESSION_TOKEN	AWS session token (if using temporary credentials)
BEDROCK_REGION	AWS region for Bedrock (e.g. us-east-1)
NOVA2_MODEL_ID	Nova 2 model ID
NOVA2_INFERENCE_PROFILE	Nova 2 cross-region inference profile ARN
GEMINI_API_KEY	Google Gemini API key
GROQ_API_KEY	Groq API key (Whisper STT + LLM fallback)
SUPABASE_URL	Supabase project URL
SUPABASE_KEY	Supabase service role key
CRAFTCONNECT_DATA_DIR	Local data directory (default: data)
WHISPER_MODEL_SIZE	Whisper model size for local STT (default: base)
DEMO_MODE	Enable demo mode without auth (default: false)

---

API Endpoints

Method	Endpoint	Description
GET	/api/health	Health check
POST	/api/mission	Create new mission (upload image)
GET	/api/missions	List all missions
GET	/api/mission/{id}/status	Poll mission status
GET	/api/mission/{id}/report	Get mission report JSON
GET	/api/mission/{id}/report/pdf	Download PDF report
POST	/api/interview/initialize	Start voice interview
POST	/api/interview/submit-answer	Submit voice answer
POST	/api/mission/{id}/listing/generate-from-interview	Generate listing from interview
POST	/api/conversation/understand	Multilingual conversation assistant
POST	/api/explain/tts	Text-to-speech explanation

Full API documentation: docs/API_TESTING_GUIDE.md

---

Model

The ViT-B/16 classifier is publicly available on HuggingFace: RohanLC/craftconnect-vit

Detail	Value
Architecture	ViT-B/16 fine-tuned on Indian handicraft dataset
Format	ONNX (exported from PyTorch vit_b_16, optimized for CPU inference)
Accuracy	89.51% validation accuracy across 7 artform classes
Training	Epoch 15 of 15, verified from training log

---

License

MIT License — see LICENSE for details.

---

Author

Built by Rohan L C