Large documents cause stack overflow in TextChunker.SplitPlainTextParagraphs() #1633
Comments
5 tasks
github-merge-queue bot
pushed a commit
that referenced
this issue
Jun 27, 2023
### Motivation and Context TextChunker.BuildParagraph is unnecessarily recursive and can lead to stack overflows on large inputs. ### Description Fixes #1633 Also happens to make it more efficient with a few tweaks. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows SK Contribution Guidelines (https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) - [x] The code follows the .NET coding conventions (https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions) verified with `dotnet format` - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 --------- Co-authored-by: Dmytro Struk <13853051+dmytrostruk@users.noreply.github.com>
shawncal
pushed a commit
to shawncal/semantic-kernel
that referenced
this issue
Jul 6, 2023
### Motivation and Context TextChunker.BuildParagraph is unnecessarily recursive and can lead to stack overflows on large inputs. ### Description Fixes microsoft#1633 Also happens to make it more efficient with a few tweaks. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows SK Contribution Guidelines (https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) - [x] The code follows the .NET coding conventions (https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions) verified with `dotnet format` - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 --------- Co-authored-by: Dmytro Struk <13853051+dmytrostruk@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
When calling
TextChunker.SplitPlainTextLines()followed byTextChunker.SplitPlainTextParagraphs()with a large document (7k+ lines output fromSplitPlainTextLines()call.) The internal chunking logic causes a stack overflow exception in the final line ofTextChunker.BuildParagraph().To Reproduce
Steps to reproduce the behavior:
Expected behavior
Document should upload, chunk, and process without issue.
Screenshots
Desktop (please complete the following information):
Additional context
Document stats:
Characters: 650k+
Words: 115k+
Output from
TextChunker.SplitPlainTextLines(): 7,214 linesThe text was updated successfully, but these errors were encountered: