Skip to content

v0.5.0

Choose a tag to compare

View all tags
@federicopalma-pro federicopalma-pro released this 06 Apr 15:46
· 24 commits to main since this release
v0.5.0
f739580

Normalized Semantic Chunker v0.5.0

A cutting-edge tool that processes text documents and splits them into semantically coherent segments while ensuring optimal chunk size for downstream NLP tasks. Ideal for retrieval-augmented generation (RAG) and other token-sensitive applications.

Key Features

  • Adaptive semantic chunking with precise token limit control
  • Parallel multi-percentile optimization and GPU acceleration
  • Intelligent handling of small and oversized chunks
  • REST API with FastAPI

Prerequisites

  • Docker and Docker Compose (for Docker deployment)
  • NVIDIA GPU with CUDA support (recommended)
  • NVIDIA Container Toolkit (for GPU passthrough in Docker)
  • Python 3.10-3.12 (Python 3.11 recommended, Python 3.13 not supported due to dependency compatibility issues)
Assets 2
Loading