Geearl/6323 large tables

[georearl]

This PR relates to GitHub issues #410 and #353.

The root cause is we add sequentially add paragraphs to a chunk until we hit the max token limit, 750 by default. Now with regular text when we reach a para that will push us over, we break it down sentence by sentence and add these to the chunk until we hit the limit, then add the remaining sentences to the next chunk.

This doesn't work for tables.

When we have a table that spans 2 pages the code to break it down by sentence doesn't work, it sees the whole table as a single sentence. The outcome is we have a single chunk holding the tables from each page. The user experience is a failure based on chunks being too large if they are returned as part of a RAG request.

The change includes the following:

• We now split tables by rows when they cross a max token count. We considered keeping a table as a single chunk using a page boundary, but Daylan requested we keep the max chunk size for tables at 1024 tokens.
• We repeat top rows that FR identifies as heading cells. We tag these rows as
• If a 2 tables are identified as a single table (which is signified by consecutive tables with no text based paragraphs between them) then we include the section form the first table to all subsequent tables. The scenario is useful if a table spans multiple pages, but the header rows are only on the first page of the table, this context would be lost as the table is chunked.