Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TrinoSplitManagerBase's single read batch is too large #37

Closed
groobyming opened this issue Nov 17, 2023 · 2 comments
Closed

TrinoSplitManagerBase's single read batch is too large #37

groobyming opened this issue Nov 17, 2023 · 2 comments

Comments

@groobyming
Copy link
Contributor

groobyming commented Nov 17, 2023

The nextPage method of TrinoPageSourceBase reads a batch of data each time. Through debugging, I found that the batch is very large, which leads to the use of DynamicSliceOutput.writeBytes of a VariableWidthBlockBuilder object when processing strings. String writing will cause the size attribute to continue to expand and eventually exceed the maximum value of int and overflow.
debug code:
image

batch size print:
image

Exception occurrence code:
image

image
@groobyming
Copy link
Contributor Author

groobyming commented Nov 17, 2023

@JingsongLi jin Please help me look into this issue.

@JingsongLi
Copy link
Contributor

Thanks @groobyming
Already fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants