[BEAM-7808] Add method to convert avro field to beam field

[bmv126]

Currently we have static methods in AvroUtils class, which exposes methods like
toBeamSchema(), toBeamRowStrict(). These methods work only on generic record.

Method is now added which supports converting avroField to BeamField and vice vesra

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go		---	---		---	---
Java
Python		---			---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

[bmv126]

R: @lukecwik R: @reuvenlax R: @akedin Can you have a look in to this.

[sheshananda]

cc: @reuvenlax cc: @akedin We have a requirement were we have to add an avro field to the beam row. So this helper function will be helpful to convert the avro field to beam field and then add it to the beam row.

[akedin]

LGTM.

Not sure I understand the use case though. You have some Beam rows with a schema. And you also have an Avro schema that you want to merge (partially) with that existing Beam schema? Would it make sense to eventually build a more generic projection framework? Something that would allow you to merge the two schemas and then pick only the fields you need. And if this happens as part of some kind of Join implementation then it probably should be supported there as well in a more generic fashion.

[akedin]

r: @TheNeuralBit

[sheshananda]

LGTM.

Not sure I understand the use case though. You have some Beam rows with a schema. And you also have an Avro schema that you want to merge (partially) with that existing Beam schema? Would it make sense to eventually build a more generic projection framework? Something that would allow you to merge the two schemas and then pick only the fields you need. And if this happens as part of some kind of Join implementation then it probably should be supported there as well in a more generic fashion.

Below is the scenario we are trying:
Kafka Source (AvroSchema based Topic) -> Beam UDF Transform -> Kafka Sink (AvroSchema based Topic)

We read incoming data stream from Kafka and then apply a Beam Transform. After tranformation again the data is written to Kafka Sink.
The data that flows between the transform is of type PCollection.
Now as part of UDF transform, we have exposed a configuration with which user can provide info on how the incoming Row has to be enriched. This information is also provided by an avro schema.

So we needed this function which can convert the avro Field to Beam Field and finally merge it with incoming row.

[bmv126]

@akedin @TheNeuralBit If changes are fine, can this be merged.

[TheNeuralBit]

Apologies for the delay. LGTM and thanks for the contribution!

I think this is great to include if it unblocks you, but like Anton I'm curious if we can find a way to support this pattern more generally. If I'm understanding correctly it sounds like maybe the Join transform should do what you want? If not maybe there's a feature request in there.