Skip to content

v1.8.0

Latest

Choose a tag to compare

@drittich drittich released this 02 Feb 19:01
· 2 commits to main since this release

New public APIs added:

  • SemanticSlicer.Slicer.SplitDocumentChunksRaw() - Split content directly without preprocessing
    • Treats content exactly as provided (no normalization/HTML stripping/whitespace collapsing/trimming)
    • Offsets relative to exact input string
    • Supports overlap, metadata, and chunk headers
  • SemanticSlicer.TextUtilities - Public preprocessing utilities
    • NormalizeLineEndings(string) - Converts all line endings to Unix-style
    • CollapseWhitespace(string) - Limits consecutive spaces/newlines to max 2
  • SemanticSlicer.Slicer.CountTokens() - Count tokens using configured encoder
  • SemanticSlicer.Slicer.PrepareContentForChunking() - Get preprocessed content separately