Skip to content

v0.6.1

Choose a tag to compare

View all tags
@federicopalma-pro federicopalma-pro released this 03 Aug 19:44
· 28 commits to main since this release
v0.6.1
0b3cd01

Sentences Chunker v0.6.1

A cutting-edge tool that intelligently splits text documents into optimally-sized chunks while preserving sentence boundaries and semantic integrity. Powered by WTPSplit neural models supporting 85+ languages, ideal for retrieval-augmented generation (RAG), embeddings generation, and LLM preprocessing.

Key Features

  • Neural sentence segmentation with configurable token limits and overlap
  • GPU acceleration with automatic CPU fallback and persistent model caching
  • Strict mode with intelligent parameter suggestions for constraint violations
  • REST API with FastAPI and comprehensive metadata

Prerequisites

  • Docker and Docker Compose (for Docker deployment)
  • NVIDIA GPU with CUDA support (recommended for performance)
  • NVIDIA Container Toolkit (for GPU passthrough in Docker)
  • Python 3.10-3.12 (for local installation)

Changelog

  • Torch library to 2.6.0
  • Docker compose fix
  • Bugs fix
Assets 2
Loading