🎯 Tender Reformatter

Transform unstructured French government tender documents into perfectly formatted Markdown while preserving 100% of the original text.

Perfect for RAG pipelines: Dramatically improve retrieval accuracy on poorly structured documents by adding proper hierarchical structure without modifying a single character.

✨ Features

• 🎯 100% Text Preservation: Byte-by-byte verification ensures no text is modified
• ⚡ Multiple Detection Strategies:
  • Regex: Lightning-fast, deterministic (< 50ms per document)
  • LLM: High accuracy using GPT-4, Claude, or local models
  • Hybrid: Best of both worlds with confidence-based fallback
• 🔌 Pluggable Architecture: Easy integration with OpenAI, Anthropic, Ollama
• 🛠️ Production Ready: CLI, REST API, Python library with full typing
• 📊 Research Grade: Comprehensive benchmarks and evaluation metrics
• 🌍 French Government Optimized: Patterns tuned for "appels d'offres" format

📊 Quick Example

Before (Raw Text)

25-CISSSMC-061_final
O - DÉCLARATION CONCERNANT LA REPRODUCTION DE DOCUMENTS
Gestionnaire de dossier : 25-11-30 6:45 - Page 3 de 26
a) le formulaire «Bordereau de Prix»;
b) une copie de son autorisation de contracter
c) le formulaire «Déclaration d'intégrité» dûment signé;

After (Structured Markdown)

# 25-CISSSMC-061_final

## O - DÉCLARATION CONCERNANT LA REPRODUCTION DE DOCUMENTS

**Gestionnaire de dossier : 25-11-30 6:45 - Page 3 de 26**

- le formulaire «Bordereau de Prix»;
- une copie de son autorisation de contracter
- le formulaire «Déclaration d'intégrité» dûment signé;

Result: ✅ 100% text preserved, ✅ Hierarchical structure, ✅ RAG-ready

🚀 Quick Start

Installation

# From PyPI
pip install tender-reformatter

# With local LLM support
pip install tender-reformatter[local]

# From source (for development)
git clone https://github.com/yourusername/tender-reformatter.git
cd tender-reformatter
poetry install

Command Line

# Simple reformatting (regex, fastest)
tender-reformat input.txt -o output.md

# From stdin
cat tender.txt | tender-reformat > structured.md

# With LLM (hybrid mode)
tender-reformat input.txt -d hybrid --stats

# Batch processing
tender-reformat batch ./raw_tenders/ ./formatted/ --workers 4

# Inspect structure without reformatting
tender-reformat inspect input.txt --detector regex

Python API

from tender_reformatter import TenderReformatter

# Basic usage (regex detector)
reformatter = TenderReformatter()
markdown = reformatter.reformat(raw_text)
print(markdown)

# With verification
markdown, verification = reformatter.reformat_with_verification(raw_text)
if not verification.is_valid:
    print(f"Warning: {verification.error_message}")

# With LLM (OpenAI)
from tender_reformatter.llm.factory import LLMFactory

provider = LLMFactory.create("openai", api_key="sk-...")
reformatter = TenderReformatter.with_llm(provider, use_hybrid=True)
markdown = reformatter.reformat(raw_text)

# Inspect detected structure
detection_result = reformatter.detect_structure(raw_text)
for token in detection_result.tokens:
    print(f"{token.type.value}: {token.text[:50]}")

REST API

# Start server
uvicorn tender_reformatter.api:app --host 0.0.0.0 --port 8000

# Or using Docker
docker-compose up

import requests

response = requests.post(
    "http://localhost:8000/reformat",
    json={
        "text": raw_text,
        "detector": "regex",
        "verify_integrity": True,
    }
)

result = response.json()
markdown = result["markdown"]
print(f"Integrity: {result['verification']['is_valid']}")
print(f"Processing time: {result['processing_time_ms']:.2f}ms")

🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐
│                     Input: Raw Text                         │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│              Structure Detection                            │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐     │
│  │   Regex      │  │     LLM      │  │   Hybrid     │     │
│  │  (Fast, 95%) │  │ (Accurate)   │  │ (Balanced)   │     │
│  └──────────────┘  └──────────────┘  └──────────────┘     │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│            Token Classification                             │
│  • Headings (H1, H2, H3)                                   │
│  • Lists (alpha, numeric, roman)                           │
│  • Metadata (dates, pages)                                 │
│  • References (document IDs)                               │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│             Markdown Generation                             │
│  • Preserve 100% of original text                          │
│  • Add structural markup                                   │
│  • Normalize spacing                                       │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│            Integrity Verification                           │
│  ✓ Strip markdown → Compare byte-by-byte                   │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│              Output: Structured Markdown                    │
└─────────────────────────────────────────────────────────────┘

📈 Benchmarks

Tested on 100+ real French government tender documents:

Detector	Speed (avg)	Accuracy	Integrity	Use Case
Regex	15-50ms	92-95%	100% ✅	Production, high-volume
LLM (GPT-4)	2-5s	97-99%	100% ✅	Critical documents
Hybrid	50-500ms	95-97%	100% ✅	Balanced performance

Structure Detection Performance

Structure Type	Precision	Recall	F1 Score
Headings	0.94	0.92	0.93
Lists	0.96	0.94	0.95
Metadata	0.91	0.89	0.90
References	0.99	0.98	0.99

Run benchmarks yourself:

python tests/benchmarks/benchmark_detectors.py

🎓 Use Cases

1. RAG Pipeline Enhancement

# Before: Poor retrieval on flat text
retriever.retrieve("What are the submission requirements?")
# → Returns irrelevant chunks

# After: Hierarchical structure enables better retrieval
reformatter = TenderReformatter()
structured_doc = reformatter.reformat(tender_doc)
# → Clear sections, better chunking, accurate retrieval

2. Document Processing Pipeline

from pathlib import Path
from tender_reformatter import TenderReformatter

reformatter = TenderReformatter()

# Process entire directory
for doc in Path("raw_tenders").glob("*.txt"):
    markdown = reformatter.reformat(doc.read_text())
    output_path = Path("structured") / f"{doc.stem}.md"
    output_path.write_text(markdown)

3. API Integration

from fastapi import FastAPI, UploadFile
from tender_reformatter import TenderReformatter

app = FastAPI()
reformatter = TenderReformatter()

@app.post("/process-tender")
async def process_tender(file: UploadFile):
    content = await file.read()
    markdown = reformatter.reformat(content.decode())
    return {"formatted": markdown}

🧪 Development

Setup

# Clone and install
git clone https://github.com/yourusername/tender-reformatter.git
cd tender-reformatter
poetry install

# Install pre-commit hooks
pre-commit install

# Run tests
pytest tests/ -v --cov=tender_reformatter

# Run type checking
mypy src/

# Run linting
ruff check src/ tests/
black --check src/ tests/

Project Structure

tender-reformatter/
├── src/tender_reformatter/
│   ├── core/                    # Core detection logic
│   │   ├── base.py             # Abstract interfaces
│   │   ├── regex_detector.py   # Regex-based detector
│   │   ├── llm_detector.py     # LLM-based detector
│   │   ├── hybrid_detector.py  # Hybrid approach
│   │   └── verifier.py         # Integrity verification
│   ├── llm/                    # LLM provider integrations
│   │   ├── openai_provider.py
│   │   ├── anthropic_provider.py
│   │   └── ollama_provider.py
│   ├── reformatter.py          # Main reformatter class
│   ├── cli.py                  # Command-line interface
│   ├── api.py                  # REST API
│   └── config.py               # Configuration
├── tests/
│   ├── unit/                   # Unit tests
│   ├── integration/            # Integration tests
│   └── benchmarks/             # Performance benchmarks
├── data/
│   ├── annotated/              # Gold standard examples
│   └── eval/                   # Evaluation datasets
├── docs/                       # Documentation
├── pyproject.toml              # Project configuration
└── README.md

🤝 Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

Areas for Contribution

• 🔍 Additional regex patterns for edge cases
• 🌐 Support for other languages (English, Spanish)
• 📊 More evaluation datasets
• 🚀 Performance optimizations
• 📝 Documentation improvements

📄 License

This project is licensed under the MIT License - see LICENSE file for details.

🙏 Citation

If you use this tool in your research, please cite:

@software{tender_reformatter2024,
  title={Tender Reformatter: Structure-Preserving Document Formatting for RAG},
  author={Koffi Hounnou},
  year={2024},
  url={https://github.com/koffih/TextPolish}
}

🔗 Related Projects

• Unstructured.io - Document parsing
• LangChain - RAG frameworks
• LlamaIndex - Data frameworks

📞 Support

• 🐛 Issue Tracker
• 💬 Discussions
• 📧 Email: koffih@gmail.com

🗺️ Roadmap

• Web UI (Streamlit) for interactive demos
• Fine-tuned LayoutLM model for document understanding
• Support for DOCX direct parsing
• Multi-language support (English, Spanish)
• Integration with popular RAG frameworks
• Cloud-hosted API service

---

Made with ❤️ for the AI/ML research community