Skip to content

Avoid stack overflow in Datafusion #1444

@xudong963

Description

@xudong963

Rewrite recursive procedures into iterative procedures, suggested by @houqp @alamb

This is an acceptable short term workaround, but I think it would be more efficient and more elegant if we rewrite these recursive procedures into iterative procedures.

Do you mean to use push-based model?

@xudong963 , I think what @houqp is suggesting is to rewrite code that is recursive to not be recursive.

The pattern for Datafusion probably looks like taking code like

fn visit(expr: &expr)  {
  for child in expr.children() {
    visit(child)
  }
  // do actual expr logic
}

And changing it so the state is tracked with a structure on the heap rather than a stack. I think VecDeque is a good one for rust:

fn visit(expr: &expr)  {
  let mut worklist = VecDequeue::new();
  worklist.push_back(expr);
  while !worklist.is_empty() {
    let parent = worklist.pop_front();
    for child in parent.children() {
      worklist.push_back(child)
    }
    // do actual expr logic on parent
  }
}

(aka avoid the call back to visit)

Originally posted by @alamb in #1437 (comment)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions