[Failing Test]: Many BigQuery direct read Python PostCommit failing possibly due to fastavro regression

[Abacn]

What happened?

Possibly due to 1.8.4 released today (Oct 3)

apache_beam/runners/common.py:1609: in handle_process_outputs
    for result in results:
apache_beam/io/gcp/bigquery.py:1312: in __next__
    return fastavro.schemaless_reader(self.bytes_reader, self.avro_schema)
fastavro/_read.pyx:1126: in fastavro._read.schemaless_reader
    ???
fastavro/_read.pyx:1153: in fastavro._read.schemaless_reader
    ???
fastavro/_read.pyx:743: in fastavro._read._read_data
    ???
fastavro/_read.pyx:616: in fastavro._read.read_record
    ???
fastavro/_read.pyx:735: in fastavro._read._read_data
    ???
fastavro/_read.pyx:526: in fastavro._read.read_union
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   EOFError

fastavro/_read.pyx:176: EOFError

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[yupbank]

This one is hitting us in production, any work arounds ? e.g. any health version ? currently using apache_beam[gcp]==2.50.0

[Abacn]

If you are using our released docker image it should not be hit because fastavro dependency is included. If you are building custom environment, explicitly install fastavro<1.8.4 (pip install "fastavro<1.8.4")

[tvalentyn]

@yupbank see recommendations about building reproducible environments on https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/ to prevent an issue like this affecting your production.

[riteshghorse]

Should we limit pin fastavro to <1.8.4?

beam/sdks/python/setup.py

Line 245 in af9da31

'fastavro>=0.23.6,<2',

[tvalentyn]

we need a bug on fastavro side; to unblock tests, we could temporarily add !=1.8.4 to the condition.

[tvalentyn]

they might be working on patch release to remediate.

[riteshghorse]

Created an issue for fastavro fastavro/fastavro#720

[kennknowles]

But the root danger is still here: if fastavro releases a bad version in our allowed range, all users break. Is our dependency pinning strategy wrong?

[AnandInguva]

We tend to follow SemVer approach for pinning dependencies for most of the packages. https://semver.org/#why-use-semantic-versioning. Usually, we set the versions of a dependency to be less than the next major upgrade. In this case, its 2.0. It is up to the fastavro library not to make backward compatible changes until there is an major upgrade.
https://devhints.io/semver

Also, we don't want to pin a tighter upper bounds on the dependencies since users will miss out on using latest minor releases until the next beam release comes out

[kennknowles]

We have to choose. Either:

1. We pin dependencies and we know that transitive dependencies cannot break users.
2. We do semver loose dependencies and know that transitive dependencies can break users.

In scenario 2 we can't really treat transitive dependency changes as a panic situation or release blocker, because the behavior is actually by design.

To be very clear: I like scenario 2 better and I think dependency pinning should wait until the final deployment of an application.

For test coverage: Even though we claim we are compatible with a semver range, we have not tested that whole range. In reality it just means the same thing as being pinned to the top of the range. We have no idea if Beam 2.51.0 actually works with lower versions of fastavro. It was never tested.

[Abacn]

Currently we release the docker image per release which contains fixed version of dependency, which partly mitigate the potential issue of choice 2. In this case, the same test does not break on Dataflow, which builds docker image via :sdks:python:container:py3x:docker task. Other portable runners in DOCKER mode would not be impacted either (most production use cases). It fails test because the direct runner and flink runner (LOOPBACK mode) determines beam's dependency from setup.py during the test.