Support Google cloud dataproc kafka sink

[qafro1]

Kafka dataproc sink is very useful to subscribe from Kafka topic into Google cloud Dataproc (Apache spark) .
Dataproc is a fast, easy-to-use, fully managed cloud service for running Apache Spark and Apache Hadoop clusters in a simpler, more cost-efficient way.
Any ideas of how to create it?

[oscerd]

You need to open an issue to general camel project: https://issues.apache.org/jira/projects/CAMEL/issues

A new component for dataproc should be created. After that we can think about the related kamelet.

[qafro1]

No problem.

[qafro1]

Is there any way of subscribing from Kafka topic into Google cloud Dataproc?

[oscerd]

Not with Camel at the moment as far as I know

[qafro1]

For a temp solution, using camel-google-pubsub-kafka-sink-connector to subscribe from Kafka topic into Google cloud Pub then use Google pud/sub to Dataproc or any other GCloud.
Is there any way to integrate it into Kamelet?

[oscerd]

You're mixing up the projects. Camel-kafka-connector is a separated project, camel-kamelets is a different one, they just share Camel, but they are different.

You can use the google pubsub one, but you'll need to wait for Camel-k 1.5.0 release