Support for new Producer/Consumer without Zookeeper connection

[barrycommins]

Hi,

is there any plan to support using the new Consumer / new Producer configuration without having to specify the connection to Zookeeper? http://kafka.apache.org/documentation.html#producerconfigs

It looks like it will default to localhost in KafkaBinderConfigurationProperties, and is an expected setting.

In some Kafka configurations, particularly when using ACLs, the connection to Zookeeper will be firewalled and the producer/consumer will only be allowed to connect to Kafka. This is often the case when management of topics is restricted.

[nitantbhartia]

@barrycommins did you find a way around this? I am seeing the same issue and was wondering if you found a workaround

[nitantbhartia]

@mbogoevici can we get some love here? Is there any way currently to bypass using zookeeper?

[barrycommins]

@nitantbhartia Hi, I've only gotten back to looking at this today, and it looks like there's a very simple solution.

just set spring.cloud.stream.kafka.binder.auto-create-topics=false and the binder will never attempt to connect to Zookeeper. It still sets the default to localhost, but it isn't used.

I've tried this on 1.1.0.RELEASE

[mbogoevici]

@barrycommins Thanks for checking and responding! I suppose we can close this issue now?

[barrycommins]

@mbogoevici Sure, I'll close it now, thanks.

[barrycommins]

One thing that might be worth noting though is that even if spring.cloud.stream.kafka.binder.auto-create-topics is set to false, the KafkaProducer / KafkaConsumer will still autocreate the topic on the initial connection, using the default properties on the Kafka broker.

This is default behaviour in Kafka unless auto.create.topics.enable is set to false in server.properties.
It will just be done by Kafka code instead of explicitly by the Spring Cloud Stream Kafka binder

[mbogoevici]

Yes - this is documented in the setting description (see http://docs.spring.io/spring-cloud-stream/docs/Brooklyn.BUILD-SNAPSHOT/reference/htmlsingle/#_configuration_options_2) . That is the behaviour of the Kafka broker and there's nothing that can be do on our side to control it - so the only thing we can do is to document it.

[barrycommins]

Oops, RTFM I guess :-)
Sorry about that.

[mbogoevici]

No worries! Thanks for staying in touch! Looking forward to more :)!

[jensdt]

Just a heads up if you, like me, came upon this issue and still have problems... @barrycommins made a small typo in the property - 'kakfa.binder' should of course be 'kafka.binder'.

Serves me right for blindly copy-pasting but hopefully this saves somebody else the 15 minutes I spent wondering why this didn't work.

[barrycommins]

@jensdt Sorry about that, I'd say I'd type 'kakfa' instead of 'kafka' about 50% of the time!

I've updated the original comment now to correct it.

[daggerok]

Thanks for spring.cloud.stream.kafka.binder.auto-create-topics=false good to know!

[garyrussell]

Just to close this out; the upcoming 2.0 release has no dependencies on zookeeper at all. Even though the bindings have used the pure java client for a long time now, we still needed to go via Zookeeper to provision topics.

There is now a pure Java AdminClient (it was added in 0.11 but was not fully functional - for the binder's needs - until 1.0.0).

[vbasem]

@garyrussell is this part of any of the current milestones? As of M5 I am still seeing zookeeper connections being opened.

[garyrussell]

There is no M5; what do you mean?

2.0.0.M4 was released last week and no longer has zookeeper dependencies.

This commit: 50b8955

As you can see from the tag it is in M4

[vbasem]

Sorry I was not being accurate, M5 was referring to Spring Cloud Finchley.M5 release train.
Do you think M4 will be part of the next release train?

[garyrussell]

See the project page.

Finchley.M5 includes Elmhurst.M3; you need to override the stream dependencies to Elmhurst.M4.

Presumably it will be in the next Finchley milestone.