datafusion-proto: logical-plan serialization overflows the stack in debug builds

[mithuncy]

datafusion_proto::bytes::logical_plan_to_bytes can abort with a native stack overflow in a debug build while serializing a small, valid LogicalPlan.

The overflow occurs during recursive logical-plan-to-protobuf conversion in:

LogicalPlanNode::try_from_logical_plan(...)

Reproducer

Tested on macOS/aarch64 with DataFusion 54.0.0.

use std::sync::Arc;

use datafusion_common::DFSchema;
use datafusion_expr::logical_plan::{EmptyRelation, LogicalPlan, LogicalPlanBuilder};
use datafusion_proto::bytes::logical_plan_to_bytes;

fn main() {
    let mut plan = LogicalPlan::EmptyRelation(EmptyRelation {
        produce_one_row: false,
        schema: Arc::new(DFSchema::empty()),
    });

    // SubqueryAlias x 10 -> EmptyRelation.
    // This requires 11 try_from_logical_plan invocations during serialization.
    for level in 0..10 {
        plan = LogicalPlanBuilder::from(plan)
            .alias(format!("level_{level}"))
            .unwrap()
            .build()
            .unwrap();
    }

    std::thread::Builder::new()
        .name("two-megabyte-stack".into())
        .stack_size(2 * 1024 * 1024)
        .spawn(move || logical_plan_to_bytes(&plan).unwrap())
        .unwrap()
        .join()
        .unwrap();
}

The default debug profile aborts:

thread 'two-megabyte-stack' has overflowed its stack
fatal runtime error: stack overflow, aborting

The same reproducer completes with cargo run --release.

Measurement

I added a temporary AArch64 stack-pointer probe at entry to LogicalPlanNode::try_from_logical_plan.

In the instrumented debug build, each nested entry adds approximately 194 KB of stack:

depth  2:   193,952 bytes
depth  3:   387,936 bytes
depth 10: 1,745,824 bytes
depth 11: 1,939,808 bytes
depth 12: 2,133,792 bytes

The 2 MiB test thread aborts while attempting to enter depth 11. With a 4 MiB test thread, depth 11 completes and the 11-alias plan reaches depth 12.

In the identically instrumented release build, each nested entry adds 14,976 bytes; depth 12 completes at 164,736 bytes.

The exact sizes are target/toolchain dependent, but the large debug-versus-release difference is reproducible.

Expected behavior

Serializing this valid logical plan should complete, or return a recoverable error. It should not abort the process through native stack overflow.

Possible direction

try_from_logical_plan is a large recursive match over all LogicalPlan variants. Reducing the size of the recursive path, or using iterative post-order conversion, would make logical-plan serialization stack-safe.

Related prior art:

• Avoid stack overflow by reducing stack usage of BinaryExpr::evaluate in debug builds #1047 reduced a large recursive debug stack frame by separating non-recursive work from the recursive path.
• Logical Expr serde should be stack safe #5693 tracks stack-safe logical-expression serde.
• Restructure core codepaths to prevent stack overflows #16788 tracks broader DataFusion stack-safety restructuring.

[mithuncy]

@alamb^