Log pollution from `Record batch memory usage exceeds the expected limit`

[xudong963]

The warning message in datafusion/physical-plan/src/spill/mod.rs (lines 157-162) is being triggered very aggressively in production environments, causing log noise.

warn!(
    "Record batch memory usage ({actual_size} bytes) exceeds the expected limit ({max_record_batch_memory} bytes) \n\
        by more than the allowed tolerance ({SPILL_BATCH_MEMORY_MARGIN} bytes).\n\
        This likely indicates a bug in memory accounting during spilling.\n\
        Please report this issue in https://github.com/apache/datafusion/issues/17340."
);

Since this is a known issue (tracked in #17340) and doesn't affect functional correctness, we should consider one of the following approaches:

Option 1: Downgrade to debug level

Change warn! to debug! to reduce production noise while keeping the diagnostic information available for development.

Option 2: Increase tolerance margin

Adjust SPILL_BATCH_MEMORY_MARGIN from 4096 bytes to a more realistic value that accounts for expected Arrow IPC overhead. But the realistic value could be case by case and needs efforts to investigate.

I lean towards Option 1, it's the easiest way to avoid the influence.

[2010YOUY01]

+1 for option 2, here are the reasons:

This warning is useful for catching memory bloat in external sorts. For example, if a batch is 10 MB before spilling, after spilling and reading it back it may become 12 MB. The external sorter's merge phase then reserves memory based on the original 10 MB estimate, while the actual memory usage exceeds the memory pool’s expectation.

Since DataFusion’s memory pool already allows a certain margin of error between estimated and actual memory usage, I think Option 2 is a good idea. We can tolerate, for example, a 20% estimation error, but order-of-magnitude errors (e.g., 10×) should still be treated as bugs, warned about, and fixed.

[alamb]

I suggest we do both

We definitely shouldn't be warning for a known issue.