fix: handle long text in embedding requests properly

[Airmomo]

Issue

When processing long documents (>8000 characters), the embedding service truncates the content, resulting in loss of information and potentially inaccurate semantic search results. The issue occurs in document_service.py:

# Current implementation simply truncates content
content = document.raw_content[:8000] if len(document.raw_content) > 8000 else document.raw_content

Cause analysis

1. The original code had two main issues:

# In document_service.py - content was truncated
content = document.raw_content[:8000]

# In llm.py - no handling for long texts
data = {"input": texts, "model": user_llm_config.embedding_model_name}

2. The problems with this approach:

• Information loss: Content beyond 8000 characters is completely ignored
• Semantic incompleteness: Truncated text may lose critical context
• Hard-coded limits: No flexibility to adjust text length limit
• Potential API failures: Long texts could cause embedding API errors

Fix

1. Added configuration for maximum text length:

# Added to .env
EMBEDDING_MAX_TEXT_LENGTH=8000

2. Implemented text chunking in LLMClient:

class LLMClient:
    def __init__(self):
        self.embedding_max_text_length = int(os.getenv('EMBEDDING_MAX_TEXT_LENGTH', 8000))

    def get_embedding(self, texts):
        # Split long texts into chunks
        for text in texts:
            if len(text) > self.embedding_max_text_length:
                chunks = [text[i:i + self.embedding_max_text_length] 
                         for i in range(0, len(text), self.embedding_max_text_length)]
                # ... process chunks and average embeddings

3. Removed content truncation from document_service.py

Improvements

1. Complete Content Processing:

   • All text content is now processed, no information loss
   • Long texts are properly chunked and processed in parts

2. Better Configuration:

   • Text length limit is now configurable via environment variable
   • Default behavior maintains backward compatibility

3. More Robust:

   • Prevents API failures due to text length

[yingapple]

Thanks for contribution.
Very useful fix.