[Bug][Go SDK]: Gap in DoFn pre-processing, interface{} in iterator type of DoFn.

[lostluck]

What happened?

The contributed parquetio write function has a signature of: parquetWriteFn) ProcessElement(ctx context.Context, _ int, iter func(*interface{}) bool) which is incorrect, as "interface{}" isn't a supported PCollection type.

beam/sdks/go/pkg/beam/io/parquetio/parquetio.go

Line 150 in 10de67e

func (a *parquetWriteFn) ProcessElement(ctx context.Context, _ int, iter func(*interface{}) bool) error {

This should have been caught at pipeline construction time regardless of whether it was registered or not, regardless of the runner used.

The resolution is two fold: Ensure that this kind of signature gets caught at pipeline construction and then also fixing the Parquetio write DoFn to not have this error.

Issue Priority

Priority: 2

Issue Component

Component: sdk-go

[yyy1000]

Hi! I'm new to this project and want to contribute. :)
Could you give me some detailed instructions on this issue?

[lostluck]

Thank you for your interest and patience!

Specifically this issue is about improving the validation that the SDK provides. In particular, it's not possible for Beam to encode arbitrary interface{} or any types as part of a PCollection. PCollections are required to have a static type at Pipeline Construction time. This avoids runtime type errors at pipeline execution time, and allows the execution layer to optimize how it's decoding types.

So the error here is that the SDK isn't validating the types in emitter functions (like func(T) or func(K, V)) iterators function (like func(*T) bool or func(*K, *V) bool, or func(K) func(*V) bool, that the TorKorV` in those are a known, registered type that Beam knows how to encode and decode.

In short, the goal is to make the type signatures for the main DoFn method ProcessElement fail pipeline construction when those are plain interface{}.

This would involve updating the funcx and typex packages to perform this validation on the emitter and emitter types and failing them accordingly.

The trick however is that Universal types, like beam.T are any. They are allowed, but only if during pipeline construction they are inferable to a concrete type, or they are bound concretely to a specific type.

If you have specific questions, let me know. "details instructions" is tantamount to simply doing the work to me, so smaller, specific questions will get better responses.

[yyy1000]

.take-issue