DLC8 - Vietnamese Translation & Localization System

Open-source Vietnamese translation and content localization system for social media marketing.

🎯 Purpose

DLC8 is a comprehensive NLP pipeline designed specifically for translating and localizing English marketing content into Vietnamese for social media platforms. It ensures culturally appropriate, grammatically correct Vietnamese text with automated quality checks.

Key Features:

• ✅ Glossary enforcement (preserve brand terms, ban inappropriate phrases)
• ✅ Tone analysis (formal, casual, promotional, informational)
• ✅ Quality scoring (grammar, word choice, readability)
• ✅ Vietnamese text normalization (abbreviations, number formats, punctuation)
• ✅ Diacritics validation (coverage analysis, missing diacritics detection, auto-correction)

🚀 Quick Start

Prerequisites

• Docker & Docker Compose
• OpenAI API key (or local Ollama setup)

Installation

1. Clone the repository:

git clone https://github.com/YOUR_USERNAME/dlc8.git
cd dlc8

2. Set up environment:

cp .env.example .env
# Edit .env and add your OPENAI_API_KEY

3. Start services:

docker compose up -d

4. Verify services:

# Check NLP service health
curl http://localhost:8000/health

# Check backend service
curl http://localhost:3000/health

Usage Example

Translate with all NLP features:

curl -X POST http://localhost:3000/translate \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Visit our bakery at Q.1, TP.HCM! Prices from 50,000 VND. Open T2-T7.",
    "glossary": {
      "preserve": ["Dulce"],
      "banned": ["best"]
    },
    "nlp_postprocess": true
  }'

Response:

{
  "translation": "Ghé tiệm bánh Dulce tại Quận 1, Thành phố Hồ Chí Minh! Giá từ 50.000 VND. Mở cửa Thứ hai-Thứ bảy.",
  "tone": {
    "detected_tone": "promotional",
    "confidence": 0.78,
    "model": "vinai/phobert-base"
  },
  "quality": {
    "score": 82,
    "grade": "B",
    "issues": []
  }
}

📚 Documentation

Complete Guides

• NLP Enhancement Guide - Complete overview of all 5 phases
• Phase 1: Glossary Enforcement
• Phase 2: Tone & Quality Scoring
• Phase 3: Text Normalization
• Phase 4: Diacritics Validation
• Production Test Summary

API Endpoints

Backend Service (Port 3000)

• POST /translate - Main translation endpoint with NLP
• GET /health - Service health check

NLP Service (Port 8000)

• GET /health - Service health and dependency status
• POST /nlp/postprocess - Phase 1-3 processing (glossary, tone, quality, normalization)
• POST /normalize - Phase 3: Vietnamese text normalization
• POST /validate-diacritics - Phase 4: Diacritics validation

Vietnamese Text Features

Phase 3: Normalization

• Abbreviation expansion (26 common terms: TP.HCM, Q., T2-T7, etc.)
• Vietnamese number formatting (1,000 → 1.000, 3.5% → 3,5%)
• Punctuation normalization (spacing, quotes)

Phase 4: Diacritics Validation

• Coverage calculation (target: 20-35% for Vietnamese text)
• Missing diacritics detection (50+ common words)
• Auto-correction (conservative and aggressive modes)

🏗️ Architecture

┌─────────────────┐
│   User/Client   │
└────────┬────────┘
         │
         ▼
┌─────────────────────────────┐
│  Backend Service (Node.js)  │
│  - Translation API          │
│  - OpenAI integration       │
│  - Request routing          │
└────────┬────────────────────┘
         │
         ▼
┌──────────────────────────────────┐
│  NLP Service (Python/FastAPI)    │
│  - Glossary enforcement (Phase 1)│
│  - Tone analysis (Phase 2)       │
│  - Quality scoring (Phase 2)     │
│  - Text normalization (Phase 3)  │
│  - Diacritics validation (Phase 4)│
│  - PhoBERT (vinai/phobert-base) │
└──────────────────────────────────┘

🤝 Contributing

We welcome contributions! This is an open-source project to improve Vietnamese NLP for marketing content.

Areas for Contribution:

1. Expand Vietnamese diacritics dictionary (currently 50+ words)
2. Add more abbreviation patterns (currently 26 patterns)
3. Improve quality scoring metrics
4. Test with real-world Vietnamese content
5. Report bugs or edge cases

How to Contribute

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Test thoroughly (see test files in root)
5. Commit your changes (git commit -m 'Add amazing feature')
6. Push to the branch (git push origin feature/amazing-feature)
7. Open a Pull Request

Testing

Run the included test suite:

# Test Phase 3 normalization
curl -X POST http://localhost:8000/normalize \
  -H "Content-Type: application/json" \
  -d @test_phase3_comprehensive.json

# Test Phase 4 diacritics
curl -X POST http://localhost:8000/validate-diacritics \
  -H "Content-Type: application/json" \
  -d @test_phase4_detection.json

# Test full pipeline
curl -X POST http://localhost:8000/nlp/postprocess \
  -H "Content-Type: application/json" \
  -d @test_vietnamese_product.json

📊 Project Status

Current Version: 0.5-production-phase4

Phase	Feature	Status
Phase 1	Glossary Enforcement	✅ Complete
Phase 2	Tone Analysis & Quality Scoring	✅ Complete
Phase 3	Vietnamese Text Normalization	✅ Complete
Phase 4	Diacritics Validation	✅ Complete
Phase 5	Smart Features (Hashtags, Emojis)	📋 Planned

🛠️ Technology Stack

Backend:

• Node.js + Express
• OpenAI API (gpt-4o-mini)

NLP Service:

• Python 3.11
• FastAPI
• Transformers (HuggingFace)
• PhoBERT (vinai/phobert-base)
• underthesea (Vietnamese NLP)
• ftfy, langdetect, rapidfuzz, emoji

Infrastructure:

• Docker & Docker Compose
• Uvicorn (ASGI server)

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• PhoBERT by VinAI Research for Vietnamese language models
• underthesea for Vietnamese NLP tools
• OpenAI for translation models
• HuggingFace for transformer infrastructure

📧 Contact & Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions

🗺️ Roadmap

Completed (v0.5):

• ✅ Core translation pipeline
• ✅ Vietnamese glossary enforcement
• ✅ Tone detection with PhoBERT
• ✅ Quality scoring system
• ✅ Text normalization (abbreviations, numbers, punctuation)
• ✅ Diacritics validation and auto-correction

In Progress:

• 🔄 Production usage testing
• 🔄 Dictionary expansion based on real data

Planned (v0.6+):

• 📋 Phase 5: Smart features (hashtags, emojis, CTAs)
• 📋 Web UI for testing
• 📋 API authentication
• 📋 Usage analytics
• 📋 Multi-language support beyond Vietnamese

---

Made with ❤️ for the Vietnamese marketing community