Skip to content

FFI: confirm FFI_RecordBatchStream does not need library_marker_id #22337

@timsaucer

Description

@timsaucer

Question

FFI_RecordBatchStream in datafusion/ffi/src/record_batch_stream.rs exposes poll_next and schema and a clone/release pair, but does not appear to carry a library_marker_id field — unlike most other FFI_X structs in the crate. Confirm whether this is intentional.

Why it matters

Without library_marker_id, the local-bypass optimization is unavailable: when a RecordBatchStream originates from the same library that consumes it, conversion back to Box<dyn RecordBatchStream> cannot short-circuit to the original concrete type and instead always wraps it in a Foreign adapter that pays the FFI hop on every poll_next.

If by design (e.g., streams are always consumed by a different library than the one that produced them, or the bypass is unsafe for Stream types specifically), this is a no-op — please document the rationale here and we will update the datafusion-ffi skill accordingly so the audit checklist no longer flags it.

Possible outcomes

  1. Intentional omission. Document why in the wrapper's module-level doc comment, and update the skill so the marker-id check is listed as "not required for FFI_RecordBatchStream because ".
  2. Genuine gap. Add library_marker_id: extern \"C\" fn() -> usize and the corresponding marker-id check in From<&FFI_RecordBatchStream> for Box<dyn RecordBatchStream>. ABI change → api change label, target main only, no back-port to branch-<major>, integration test required.

Generated from an audit performed for PR #22327 (datafusion-ffi agent skill). If the intentional-omission outcome applies, please propose the skill update along with the close-as-not-planned.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ffiChanges to the ffi cratequestionFurther information is requested

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions