fix: sectioner does not consider separator length

[scanny]

sectioner-does-not-consider-separator-length

Executive Summary. A primary responsibility of the sectioner is to minimize the number of chunks that need to be split mid-text. It does this by computing text-length of the section being formed and "finishing" the section when adding another element would extend its text beyond the window size.

When element-text is consolidated into a chunk, the text of each element is joined, separated by a "blank-line" ("\n\n"). The sectioner does not currently consider the added length of separators (2-chars each) and so forms sections that need to be split mid-text when chunked.

Chunk-splitting should only be necessary when the text of a single element is longer than the chunking window.

Example

  elements: List[Element] = [
      Title("Chunking Priorities"),  # 19 chars
      ListItem("Divide text into manageable chunks"),  # 34 chars
      ListItem("Preserve semantic boundaries"),  # 28 chars
      ListItem("Minimize mid-text chunk-splitting"),  # 33 chars
  ]  # 114 chars total but 120 chars with separators

  chunks = chunk_by_title(elements, max_characters=115)

Want:

[
  CompositeElement(
      "Chunking Priorities"
      "\n\nDivide text into manageable chunks"
      "\n\nPreserve semantic boundaries"
  ),
  CompositeElement("Minimize mid-text chunk-splitting"),
]

Got:

[
  CompositeElement(
      "Chunking Priorities"
      "\n\nDivide text into manageable chunks"
      "\n\nPreserve semantic boundaries"
      "\n\nMinimize mid-text chunk-spli"),
  )
  CompositeElement("tting")

Technical Summary

Because the sectioner does not consider separator ("\n\n") length when it computes the space remaining in the section, it over-populates the section and when the chunker concatenates the element text (each separated by the separator) the text exceeds the window length and the chunk must be split mid-text, even though there was an even element boundary it could have been split on.

Fix

Consider separator length in the space-remaining computation.

The solution here extracts both the section.text_length and section.space_remaining computations to a _TextSectionBuilder object which removes the need for the sectioner (_split_elements_by_title_and_table()) to deal with primitives (List[Element], running text length, separator length, etc.) and allows it to focus on the rules of when to start a new section.

This solution may seem like overkill at the moment and indeed it would be except it forms the foundation for adding section-level chunk combination (fix: dissociated title elements) in the next PR. The objects introduced here will gain several additional responsibilities in the next few chunking PRs in the pipeline and will earn their place.

[Coniferish]

LGTM!