Dataflow to BQ

[bmd-andyputra]

Hi,

I'm trying to use dataflow as bridge from kafka to bigquery, but for each topic created in kafka, i need to create 1 dataflow and that will result in huge cost. is there any way to read multiple topic to produce multiple table in bigquery using only 1 dataflow?

i'm using this dataflow template
https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/main/java/com/google/cloud/teleport/templates/KafkaToBigQuery.java

[bvolpato]

The parameters support reading from multiple topics:

  @TemplateParameter.Text(
      order = 2,
      optional = true,
      regexes = {"[,a-zA-Z0-9._-]+"},
      description = "Kafka topic(s) to read input from.",
      helpText = "Kafka topic(s) to read input from.",
      example = "topic1,topic2")
  String getKafkaReadTopics();

Please use topic1,topic2, etc, to read from many topics, instead of creating multiple jobs.

--

This issue has been stale for some time now. Please reopen it if there is a follow up or any related questions.