[C++] IPC Stream Reader doesn't check if extra fields are present for RecordBatches

[asfimport]

I looked through recent commits and I don't think this issue has been patched since:

import pyarrow as pa
with pa.output_stream("/tmp/f1") as sink:
  with pa.RecordBatchStreamWriter(sink, rb1.schema) as writer:
    writer.write(rb1)
    end_rb1 = sink.tell()

with pa.output_stream("/tmp/f2") as sink:
  with pa.RecordBatchStreamWriter(sink, rb2.schema) as writer:
    writer.write(rb2)
    start_rb2_only = sink.tell()
    writer.write(rb2)
    end_rb2 = sink.tell()

# Stitch to togher rb1.schema, rb1 and rb2 without schema.
with pa.output_stream("/tmp/f3") as sink:
  with pa.input_stream("/tmp/f1") as inp:
     sink.write(inp.read(end_rb1))
  with pa.input_stream("/tmp/f2") as inp:
    inp.seek(start_rb2_only)
    sink.write(inp.read(end_rb2 - start_rb2_only))

with pa.ipc.open_stream("/tmp/f3") as sink:
  print(sink.read_all())

Yields:

{{pyarrow.Table
c1: int64
----
c1: [[1],[1]]

I would expect this to error because the second stiched in record batch has more fields then necessary but it appears to load just fine.

Is this intended behavior?

Reporter: Micah Kornfield / @emkornfield

_{Note: This issue was originally created as ARROW-16160. Please see the migration documentation for further details.}

[asfimport]

Micah Kornfield / @emkornfield:
The opposite direction returns an error "ArrowInvalid: Ran out of field metadata, likely malformed". If we agree this case should be an error it would an error message that indicates too many fields were provided.

I can provide a patch if we think we want to change this behavior.

[asfimport]

Micah Kornfield / @emkornfield:
It appears in on master branch we get:
"pyarrow.lib.ArrowInvalid: Tried reading schema message, was null or length 0"

It seems like the error message could be improved here.

[asfimport]

David Li / @lidavidm:
We should definitely improve those errors, and I would consider the behavior in the OP to be a bug as well.

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.