Skip to content
BranchesTags

Repository files navigation

AI-Powered Contract Analysis System

📋 Table of Contents

Overview

This project implements an AI-powered contract compliance analysis system that uses Retrieval-Augmented Generation (RAG), Large Language Models (LLMs), and advanced caching to automatically verify if contract clauses meet specified obligations.

Key Features

  • ✅ Semantic Analysis: Pure cosine similarity for confidence scoring
  • ✅ Multilingual Support: Automatic translation to English
  • ✅ RAG-based Retrieval: FAISS vector store for efficient clause matching
  • ✅ LLM Reasoning: GPT-4o-mini for nuanced compliance decisions
  • ✅ Strict Yes/No Output: Binary compliance status (no "Partial")
  • ✅ Intelligent Caching: Avoid re-analyzing identical obligations (NEW)
  • ✅ Batch Processing: 3-5x faster for multiple obligations (NEW)
  • ✅ Configurable Embeddings: Switch between OpenAI models for performance/accuracy (NEW)
  • ✅ Legal Domain Expertise: Advanced prompt engineering for contract analysis (NEW)

System Architecture

┌─────────────────┐
│   User Input    │
│  - Obligations  │
│  - Contract     │
└────────┬────────┘
         │
         ▼
┌─────────────────────────────────────────┐
│         Text Extraction Layer           │
│  - PDF/DOCX/Excel/TXT support          │
│  - Language detection & translation     │
└────────┬────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────┐
│         Chunking & Embedding            │
│  - RecursiveCharacterTextSplitter       │
│  - Chunk size: 2000, Overlap: 200      │
│  - OpenAI embeddings (configurable)    │
└────────┬────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────┐
│         FAISS Vector Store              │
│  - Stores contract clause embeddings    │
│  - Enables similarity search            │
└────────┬────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────┐
│         Cache Check (NEW)               │
│  - Check if obligation already analyzed │
│  - Return cached result if hit          │
└────────┬────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────┐
│    Batch Processing (NEW)               │
│  - Parallel analysis for multiple obs   │
│  - ThreadPoolExecutor (5 workers)       │
└────────┬────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────┐
│         RAG Query Pipeline              │
│  1. Embed obligation                    │
│  2. Retrieve top-k similar clauses      │
│  3. Calculate cosine similarity         │
│  4. Extract keywords (KeyBERT + spaCy) │
└────────┬────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────┐
│         LLM Analysis (GPT-4o-mini)      │
│  - Semantic compliance check            │
│  - Reason generation                    │
│  - Suggestion generation (if No)        │
└────────┬────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────┐
│         Cache Store (NEW)               │
│  - Store result for future use          │
└────────┬────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────┐
│         Final Output                    │
│  - Status: Yes/No                       │
│  - Confidence: Cosine similarity %      │
│  - Reason: LLM explanation              │
│  - Suggestion: Remediation advice       │
└─────────────────────────────────────────┘

Technical Approach

1. Confidence Scoring: Cosine Similarity Only

Decision: Use pure cosine similarity between obligation and contract clause embeddings.

Why?

  • Directly measures semantic similarity
  • Transparent and explainable
  • Avoids complex hybrid formulas

Implementation:

confidence = round(best_score * 100, 1)  # best_score is cosine similarity

2. Chunking Strategy

Configuration:

  • Chunk Size: 2000 characters
  • Overlap: 200 characters

Why larger chunks?

  • Complex obligations span multiple sentences
  • More context improves semantic matching
  • Reduces fragmentation of related clauses

3. Retrieval Strategy

Top-k Retrieval: Fetch 6 most similar chunks

Why?

  • Provides multiple perspectives
  • Handles cases where obligation matches across clauses
  • Gives LLM sufficient context for reasoning

4. LLM for Final Decision

Model: GPT-4o-mini
Temperature: 0.1 (low for consistency)

Why LLM?

  • Handles nuanced language (e.g., "in lieu of", "reasonable efforts")
  • Understands legal implications
  • Generates human-readable explanations

Understanding Similarity Score vs Final Result

Critical Concept: Two Independent Stages

Many users ask: "Why does Obligation 3 return YES with 61% similarity, but Obligation 4 returns NO with 66.8% similarity?"

Answer: Similarity score and final result are independent. Here's why:

The Two-Stage Decision Process

Stage 1: Similarity Score (Confidence Metric)
    ↓
    Measures: Word/semantic similarity
    Output: 0-100% confidence score
    
Stage 2: LLM Semantic Analysis (Final Decision)
    ↓
    Measures: Legal compliance & logical equivalence
    Output: "Yes" or "No"

Stage 1: Similarity Score (Confidence)

What it measures:

  • Lexical similarity (word overlap)
  • Semantic closeness (embedding similarity)
  • How "related" the texts are

What it does NOT measure:

  • Legal compliance
  • Logical equivalence
  • Intent matching

Example:

Obligation: "Vendor must fix bugs"
Contract: "Vendor will repair defects"
Similarity: 45% (different words: fix≠repair, bugs≠defects)

Stage 2: LLM Analysis (Final Result)

What it measures:

  • Legal compliance based on meaning
  • Logical equivalence (do they achieve the same outcome?)
  • Intent matching (does the contract fulfill the obligation?)

Example (same texts as above):

LLM Analysis: "Yes" 
Reason: "Repair defects" achieves the same outcome as "fix bugs"

Real-World Example: Obligation 3 vs 4

Obligation 3: YES (despite only 61% similarity)

Obligation:

"Vendor shall undertake all necessary modifications to remedy infringement"

Contract Clause:

"Vendor will use reasonable commercial efforts to implement modifications 
OR secure licenses in lieu of modification"

Stage 1 - Similarity Score: 61%

  • Different vocabulary ("undertake" vs "implement", "remedy" vs "secure licenses")
  • Moderate lexical overlap

Stage 2 - LLM Analysis: YES ✅

LLM Reasoning (applies Guideline #1):

Guideline: "Reasonable Efforts to Achieve Result"

Analysis:
- Obligation requires: Remedy infringement (RESULT-ORIENTED)
- Contract commits to: Reasonable efforts to modify OR secure licenses
- Key insight: Both paths achieve NON-INFRINGEMENT
  • Path 1: Modify product → no infringement
  • Path 2: Secure licenses → no infringement
- "In lieu of" alternatives are acceptable if same outcome
- Conclusion: COMPLIANT ✅

Why YES despite low similarity?

  • Different words, same legal outcome
  • "Securing licenses" = acceptable alternative to "fixing"
  • Both achieve the obligation's goal (no infringement)
  • Semantic equivalence trumps lexical similarity

Obligation 4: NO (despite 66.8% similarity)

Obligation:

"Licensee must guarantee non-infringement and must:
 (i) secure continued use rights, OR
 (ii) replace with non-infringing products"

Contract Clause:

"Licensee will use reasonable efforts to secure rights or replace products.
If unable, Licensee may reimburse Customer."

Stage 1 - Similarity Score: 66.8%

  • High word overlap ("secure rights", "replace products")
  • Very similar structure

Stage 2 - LLM Analysis: NO ❌

LLM Reasoning (applies Guidelines #2 & #3):

Guideline 2: "Strict Remedy Matching for Guarantees"
Guideline 3: "Refund as Escape Clause"

Analysis:
- Obligation demands: GUARANTEE (strong commitment)
- Contract provides: "may reimburse" (escape clause)
- Key issue: Reimbursement allows TERMINATION of use
  • Obligation: Must secure continued use OR replace
  • Contract: Can refund and walk away
- Refund undermines the guarantee
- Conclusion: NON-COMPLIANT ❌

Why NO despite high similarity?

  • Similar words, different legal outcome
  • "May reimburse" = escape clause (vendor can walk away)
  • Obligation requires continued use, refund terminates use
  • Semantic non-equivalence despite lexical similarity

Side-by-Side Comparison

Aspect Obligation 3 Obligation 4
Similarity Score 61% (lower) 66.8% (higher)
Final Result ✅ YES ❌ NO
Key Phrase "in lieu of" (alternative) "may reimburse" (escape)
Outcome Match ✅ Same (non-infringement) ❌ Different (termination vs continuation)
LLM Guideline #1: Reasonable efforts #2 & #3: Guarantees & escape clauses
Why? Different words, same outcome Similar words, different outcome

The Key Insight

Similarity Score ≠ Compliance

High Similarity does NOT mean Compliance:

Obligation: "You must give me $100"
Contract: "I will give you $100 or give you nothing"
Similarity: 85% (almost identical words)
Compliance: NO (escape clause present)

Low Similarity does NOT mean Non-Compliance:

Obligation: "You must give me $100"
Contract: "I will provide you one hundred dollars"
Similarity: 40% (different words)
Compliance: YES (same meaning)

Why This Two-Stage Design is Correct

Problem with Similarity-Only Approach

If we used only similarity scores:

if similarity > 70%:
    return "Yes"  # WRONG for Obligation 4!
else:
    return "No"   # WRONG for Obligation 3!

Our Approach: Similarity + LLM

# Stage 1: Similarity (for confidence only)
similarity_score = cosine_similarity(obligation, contract)
confidence = similarity_score * 100  # 61% or 66.8%

# Stage 2: LLM (for actual decision)
llm_analysis = analyze_with_legal_guidelines(obligation, contract)
final_result = llm_analysis["is_present"]  # "Yes" or "No"

return {
    "confidence": confidence,      # How similar the texts are
    "is_present": final_result     # Whether legally compliant
}

Decision Flow Diagram

User uploads Obligation + Contract
         ↓
Extract & chunk contract text
         ↓
Embed obligation & chunks
         ↓
Calculate cosine similarity → Similarity Score (61% or 66.8%)
         ↓                    (Used for CONFIDENCE only)
Retrieve top-6 similar chunks
         ↓
Send to LLM with legal guidelines
         ↓
LLM applies reasoning:
  - Guideline #1: Reasonable efforts?
  - Guideline #2: Guarantee undermined?
  - Guideline #3: Escape clause present?
         ↓
LLM returns: "Yes" or "No" → Final Result
         ↓                    (Based on MEANING)
Return both:
  - confidence: 61% or 66.8%
  - is_present: "Yes" or "No"

Summary

Similarity Score (Confidence):

  • Measures: How similar the texts look
  • Range: 0-100%
  • Purpose: Confidence metric only
  • NOT used for final decision

LLM Analysis (Final Result):

  • Measures: Whether texts mean the same thing legally
  • Output: "Yes" or "No"
  • Purpose: Actual compliance decision
  • Independent of similarity score

The Magic:

  • LLM understands "securing licenses" = "fixing" (same outcome)
  • LLM understands "reimbursing" ≠ "guaranteeing" (different outcome)
  • This is semantic understanding, not pattern matching

Real-World Impact:

  • ✅ Catches compliance issues even with similar wording (Ob 4)
  • ✅ Recognizes compliance even with different wording (Ob 3)
  • ✅ Focuses on legal meaning, not just word matching

Key Challenges & Solutions

Root Cause: The LLM interpreted "in lieu of" alternatives (securing licenses) as non-compliance, not recognizing that both paths achieve the same result (non-infringement).

Solution: Strengthened prompt guideline #1:

1. **Reasonable Efforts to Achieve Result**: If the Obligation requires a 
   specific result (e.g., "remedy infringement"), and the Contract commits 
   to "reasonable commercial efforts" to achieve that result, this is 
   ACCEPTABLE. Return "Yes". The contract may also offer alternatives 
   "in lieu of" the primary remedy (e.g., securing licenses instead of 
   fixing) as long as these alternatives achieve the same end result 
   (non-infringement, continued use).

Challenge 2: Low Similarity Threshold Overriding LLM

Issue: Code had a fallback check that overrode the LLM's "Yes" decision when vocabulary differed.

Solution: Removed the threshold check to fully trust semantic analysis.

Challenge 3: Case Sensitivity Bug

Issue: LLM returned "yes" (lowercase), but validation checked for "Yes" (title case).

Solution: Added normalization to handle any case variation.

Challenge 4: Incorrect Suggestions

Issue: Prompt didn't ask for suggestions, causing null or contradictory messages.

Solution: Added suggestion field to prompt with proper logic.

LLM Prompt Engineering

Final Prompt Structure

You are a multilingual contract compliance analyst.

Task: Determine if the 'Obligation' is fully present in the 'Relevant Clauses'.

Guidelines:
1. **Reasonable Efforts to Achieve Result**: If the Obligation requires a 
   result, and the Contract commits to "reasonable commercial efforts" to 
   achieve it, this is ACCEPTABLE. Alternatives "in lieu of" the primary 
   remedy are acceptable if they achieve the same end result.

2. **Strict Remedy Matching for Guarantees**: If the Obligation demands a 
   GUARANTEE with specific remedies, and the Contract adds a REFUND option 
   that allows termination, this undermines the guarantee. Return "No".

3. **Refund as Escape Clause**: If the Obligation requires continued use, 
   but the Contract allows refund and termination, this is NON-COMPLIANCE.

Return JSON:
{
  "is_present": "Yes" or "No",
  "reason": "short 1-2 sentence rationale",
  "suggestion": "If 'No', provide specific clause suggestion. If 'Yes', null."
}

Performance Enhancements

1. Intelligent Caching

How It Works:

  • Cache Key Generation: SHA256(obligation_text + SHA256(contract_content))
  • Storage: In-memory LRU cache (max 1000 entries)
  • Automatic hit rate tracking: Monitors hits/misses
  • Content-based, not filename-based: Uses actual contract content hash

Implementation Details:

# Cache key generation
def _generate_key(obligation: str, contract_hash: str) -> str:
    combined = f"{obligation}|{contract_hash}"
    return hashlib.sha256(combined.encode()).hexdigest()

# Contract hash from content (not filename)
def hash_contract(contract_text: str) -> str:
    return hashlib.sha256(contract_text.encode()).hexdigest()

Critical Edge Cases Handled:

Scenario Obligation Contract Filename Cache Result Why?
1. Exact repeat Same text Same content Same ✅ HIT Identical cache key
2. Updated contract Same text Different content Same ✅ MISS Content hash changed
3. Renamed file Same text Same content Different ✅ HIT Content hash unchanged
4. Slight obligation change Different text Same content Same ✅ MISS Obligation text changed
5. Whitespace only Same (trimmed) Same content Same ✅ HIT Text normalized

Example Scenario:

Upload 1:
- Obligations: "Vendor must remedy infringement"
- Contract: contract_v1.pdf (content: "Vendor will fix...")
- Cache Key: hash("Vendor must remedy..." + hash("Vendor will fix..."))
- Result: Analyzed, cached

Upload 2 (Updated contract, same filename):
- Obligations: "Vendor must remedy infringement" (same)
- Contract: contract_v1.pdf (UPDATED: "Vendor will reimburse...")
- Cache Key: hash("Vendor must remedy..." + hash("Vendor will reimburse..."))
- Result: MISS - Re-analyzes with new content ✅

Why Content-Based is Correct:

  • ✅ Always re-analyzes when contract actually changes
  • ✅ Cache hits even if file is renamed
  • ✅ Detects even single character changes
  • ❌ Filename-based would return stale results for updated contracts

Performance Impact:

First analysis (10 obligations): 30s
Second analysis (same 10): <1s (100% cache hit)
Cache hit rate: 100%
Cost savings: 10x fewer LLM API calls

2. Batch Processing

How It Works:

  • Uses Python's ThreadPoolExecutor for parallel LLM calls
  • Configurable workers via BATCH_SIZE environment variable (default: 5)
  • Maintains original obligation order in results
  • Automatic error handling per obligation

Implementation Details:

with ThreadPoolExecutor(max_workers=BATCH_SIZE) as executor:
    # Submit all obligations in parallel
    future_to_index = {
        executor.submit(query_rag_with_cache, vs, ob, ...): i
        for i, ob in enumerate(obligations)
    }
    
    # Collect results as they complete
    for future in as_completed(future_to_index):
        index = future_to_index[future]
        results[index] = future.result()  # Maintains order

Why Parallel Processing Works:

  • LLM calls are I/O-bound: Waiting for API response, not CPU-intensive
  • ThreadPoolExecutor: Perfect for I/O-bound tasks
  • No GIL issues: Network I/O releases the GIL
  • Optimal worker count: 5 workers balances speed vs. API rate limits

Performance Comparison:

Obligations Sequential Batch (5 workers) Speedup Time Saved
10 30s 8s 3.75x 22s
50 150s 35s 4.29x 115s
100 300s 65s 4.62x 235s

Tuning BATCH_SIZE:

BATCH_SIZE Best For Pros Cons
1 Testing Sequential, easy to debug Very slow
3 Small contracts Moderate speedup Underutilizes API
5 (default) Most cases Good balance Optimal for most
7-10 Large contracts Maximum speed May hit rate limits
10+ Not recommended Marginal gains API throttling risk

Error Handling:

  • Each obligation analyzed independently
  • If one fails, others continue
  • Failed obligations return error message in results
  • No cascading failures

Example:

10 obligations submitted:
Worker 1: Analyzing obligation 1... (3s)
Worker 2: Analyzing obligation 2... (3s)
Worker 3: Analyzing obligation 3... (3s)
Worker 4: Analyzing obligation 4... (3s)
Worker 5: Analyzing obligation 5... (3s)
[Workers 1-5 finish, pick up obligations 6-10]
Total time: ~8s (vs 30s sequential)

3. Configurable Embeddings (Not Custom Fine-Tuning)

What We Did:

  • Made embedding model configurable via environment variable
  • Support switching between OpenAI's pre-trained models
  • No custom fine-tuning on legal documents

Available Models:

  • text-embedding-3-small: Fast, cost-effective, 1536 dimensions (default)
  • text-embedding-3-large: Better semantic understanding, 3072 dimensions

Why text-embedding-3-large is Better for Legal Domain:

  • More parameters = better capture of nuanced legal terminology
  • Better semantic understanding of complex contract clauses
  • Improved handling of legal concepts like "reasonable efforts", "in lieu of"

Configuration: Set EMBEDDING_MODEL in .env

Note: The real "legal domain expertise" comes from our carefully engineered LLM prompt (see LLM Prompt Engineering section), not from custom fine-tuned embeddings.

Setup & Usage

Installation

# Install dependencies
pip install -r backend/requirements.txt

# Create .env file
cp .env.example .env

# Set your OpenAI API key
# Edit .env and add: OPENAI_API_KEY=your-key-here

Configuration

Edit .env file:

# OpenAI API Key (REQUIRED)
OPENAI_API_KEY=your-key-here

# Embedding Model (optional)
# Options: text-embedding-3-small (default), text-embedding-3-large
EMBEDDING_MODEL=text-embedding-3-small

# Caching (optional, default: true)
USE_CACHE=true

# Batch Processing (optional, default: 5)
BATCH_SIZE=5

Running the Backend

uvicorn backend.main:app --reload

Server starts at: http://localhost:8000

API Endpoint

POST /api/analyze

Request:

curl -X POST http://localhost:8000/api/analyze \
  -F "obligations_file=@obligations.xlsx" \
  -F "contract_file=@contract.pdf"

Response:

{
  "status": "success",
  "results": [
    {
      "obligation": "Vendor must remedy infringement",
      "is_present": "Yes",
      "confidence": 87.3,
      "similarity_score": 0.873,
      "reason": "Contract commits to reasonable efforts to remedy",
      "suggestion": null,
      "supporting_clauses": ["..."]
    }
  ],
  "contract_url": "/uploads/...",
  "full_text": "..."
}

Configuration

Environment Variables

Variable Default Description
OPENAI_API_KEY (required) Your OpenAI API key
EMBEDDING_MODEL text-embedding-3-small Embedding model to use
USE_CACHE true Enable/disable caching
BATCH_SIZE 5 Number of parallel workers

For Presentations

To use the best quality model for demos:

# Edit .env
EMBEDDING_MODEL=text-embedding-3-large

# Restart server
uvicorn backend.main:app --reload

API Reference

Endpoints

1. Main Analysis

POST /api/analyze

Analyzes obligations against contract with caching and batch processing enabled.

2. Cache Statistics

GET /api/cache/stats

Returns cache performance metrics.

Response:

{
  "status": "success",
  "cache_stats": {
    "size": 45,
    "max_size": 1000,
    "hits": 120,
    "misses": 45,
    "hit_rate": 72.73
  }
}

3. Clear Cache

POST /api/cache/clear

Clears all cached results.

Technologies Used

  • LLM: OpenAI GPT-4o-mini
  • Embeddings: OpenAI text-embedding-3-small/large (configurable)
  • Vector Store: FAISS (Facebook AI Similarity Search)
  • Keyword Extraction: KeyBERT + spaCy
  • Translation: googletrans + OpenAI fallback
  • Backend: FastAPI + Uvicorn
  • Caching: In-memory LRU cache
  • Batch Processing: ThreadPoolExecutor

Project Structure

ContractAnalysis_V2/
├── backend/
│   ├── main.py              # FastAPI endpoints
│   ├── core.py              # Core analysis logic
│   ├── core_enhanced.py     # Enhanced features (caching, batch)
│   ├── cache.py             # Caching module
│   └── requirements.txt     # Python dependencies
├── .env                     # Configuration (create from .env.example)
├── .env.example             # Configuration template
├── README.md                # This file
├── ENHANCEMENTS.md          # Detailed enhancement documentation
└── uploads/                 # Uploaded files (auto-created)

Performance Tips

  1. Use Caching: Keep USE_CACHE=true for repeated analyses
  2. Tune Batch Size: Adjust BATCH_SIZE based on your system (3-7 recommended)
  3. Monitor Cache: Check /api/cache/stats to track hit rate
  4. Choose Model: Use text-embedding-3-small for speed, text-embedding-3-large for accuracy

Troubleshooting

Low Cache Hit Rate

Cause: Obligations are slightly different each time
Solution: Normalize obligation text before analysis

Batch Processing Not Faster

Cause: BATCH_SIZE too small or too large
Solution: Tune BATCH_SIZE (recommended: 3-7)

Out of Memory

Cause: Too many cached results
Solution: Reduce cache size or clear cache periodically via /api/cache/clear

Future Enhancements

  • Configurable embeddings for legal domain ✅ IMPLEMENTED
  • Caching for repeated obligations ✅ IMPLEMENTED
  • Batch processing for large contracts ✅ IMPLEMENTED
  • Legal domain prompt engineering ✅ IMPLEMENTED
  • Support for more LLM providers (Anthropic, Gemini)
  • Custom fine-tuned embeddings on legal corpus
  • Persistent cache (Redis/Database)
  • Confidence calibration based on historical data

See ENHANCEMENTS.md for detailed documentation on implemented features.

License

MIT License

Contributors

Developed as part of an AI-powered contract intelligence project.

Quick Start

# 1. Install dependencies
pip install -r backend/requirements.txt

# 2. Create .env file
cp .env.example .env

# 3. Add your OpenAI API key to .env
# OPENAI_API_KEY=your-key-here

# 4. Start server
uvicorn backend.main:app --reload

# 5. Upload files via API
curl -X POST http://localhost:8000/api/analyze \
  -F "obligations_file=@obligations.xlsx" \
  -F "contract_file=@contract.pdf"

That's it! The system now runs with all enhancements enabled by default. 🚀

About

This project implements an **AI-powered contract compliance analysis system** that uses **Retrieval-Augmented Generation (RAG)** and **Large Language Models (LLMs)** to automatically verify if contract clauses meet specified obligations.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages