Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bound amount of work done in constant folding #7564

Merged
merged 5 commits into from
Jul 27, 2021

Conversation

frankmcsherry
Copy link
Contributor

@frankmcsherry frankmcsherry commented Jul 27, 2021

This PR converts TableFunc to evaluate to a Box<Iterator>. For most of the variants of this function, that avoids a Vec allocation of all the of the results. The FlatMap reduction logic is also updated to only produce a bounded number of records, using a threshold (10,000) that was chosen only to minimally disrupt tests.

Fixes #6790
Fixes #7008

Copy link
Member

@benesch benesch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

@frankmcsherry
Copy link
Contributor Author

The last PR adds in support for limiting Join unfolding as well. It's not great, but fixes #7008 .

@frankmcsherry frankmcsherry changed the title Flatmap iterators Bound amount of work done in constant folding Jul 27, 2021
@frankmcsherry frankmcsherry merged commit 9cfbafb into MaterializeInc:main Jul 27, 2021
@frankmcsherry frankmcsherry deleted the flatmap_iterators branch March 8, 2022 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants