[Failing Test]: Python postcommits are failing due to tests in bigquery_read_it_test

[ahmedabu98]

What happened?

Seeing EOF errors. Looks like they're not failing on Dataflow, but they are failing for flink and direct runners.

Example stacktrace from apache_beam.io.gcp.bigquery_read_it_test.ReadTests.test_table_schema_retrieve_with_direct_read:

E       RuntimeError: Pipeline BeamApp-jenkins-1004001524-fad74b9a_641cdb33-f74d-452f-8929-87441cffdedc failed in state FAILED: java.lang.RuntimeException: Error received from SDK harness for instruction 60: Traceback (most recent call last):
E         File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python39/src/sdks/python/apache_beam/runners/worker/sdk_worker.py", line 297, in _execute
E           response = task()
E         File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python39/src/sdks/python/apache_beam/runners/worker/sdk_worker.py", line 372, in <lambda>
E           lambda: self.create_worker().do_instruction(request), request)
E         File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python39/src/sdks/python/apache_beam/runners/worker/sdk_worker.py", line 625, in do_instruction
E           return getattr(self, request_type)(
E         File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python39/src/sdks/python/apache_beam/runners/worker/sdk_worker.py", line 663, in process_bundle
E           bundle_processor.process_bundle(instruction_id))
E         File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python39/src/sdks/python/apache_beam/runners/worker/bundle_processor.py", line 1051, in process_bundle
E           input_op_by_transform_id[element.transform_id].process_encoded(
E         File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python39/src/sdks/python/apache_beam/runners/worker/bundle_processor.py", line 232, in process_encoded
E           self.output(decoded_value)
E         File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python39/src/sdks/python/apache_beam/runners/worker/operations.py", line 572, in output
E           _cast_to_receiver(self.receivers[output_index]).receive(windowed_value)
E         File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python39/src/sdks/python/apache_beam/runners/worker/operations.py", line 266, in receive
E           self.consumer.process(windowed_value)
E         File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python39/src/sdks/python/apache_beam/runners/worker/operations.py", line 1076, in process
E           delayed_applications = self.dofn_runner.process_with_sized_restriction(
E         File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python39/src/sdks/python/apache_beam/runners/common.py", line 1465, in process_with_sized_restriction
E           return self.do_fn_invoker.invoke_process(
E         File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python39/src/sdks/python/apache_beam/runners/common.py", line 830, in invoke_process
E           residual = self._invoke_process_per_window(
E         File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python39/src/sdks/python/apache_beam/runners/common.py", line 994, in _invoke_process_per_window
E           self.output_handler.handle_process_outputs(
E         File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python39/src/sdks/python/apache_beam/runners/common.py", line 1609, in handle_process_outputs
E           for result in results:
E         File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python39/src/sdks/python/apache_beam/io/gcp/bigquery.py", line 1312, in __next__
E           return fastavro.schemaless_reader(self.bytes_reader, self.avro_schema)
E         File "fastavro/_read.pyx", line 1126, in fastavro._read.schemaless_reader
E         File "fastavro/_read.pyx", line 1153, in fastavro._read.schemaless_reader
E         File "fastavro/_read.pyx", line 743, in fastavro._read._read_data
E         File "fastavro/_read.pyx", line 616, in fastavro._read.read_record
E         File "fastavro/_read.pyx", line 735, in fastavro._read._read_data
E         File "fastavro/_read.pyx", line 526, in fastavro._read.read_union
E         File "fastavro/_read.pyx", line 176, in fastavro._read.read_long
E       EOFError

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 2 (backlog / disabled test but we think the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[ahmedabu98]

I see there's already a GH issue for this in #28811. Closing this so any discussion goes on the other one.