[phpkafka] How to make persistent connections?

[viniciuswebdev]

Hey guys,

I'm testing some libs to publish with PHP to kafka, but I noticed that every message published a connection is open on Kafka, there is a way to use the same connection open at the first message?

Or is it a configuration on Kafka?

Thanks!!!

[edenhill]

I dont know much about the PHP bindings, but it sounds like a new client handle is constructed for each produced message, which definately isn't right.

Check to see if you are properly holding on and reusing your kafka handle.

I think you should file this issue on phpkafka instead.
Right @EVODelavega ?

[viniciuswebdev]

Thank you @edenhill! I think this is the problem, as we are publishing messages on a web based aplication, every request we need to create a new connection, but i'm trying to find a way to reuse the same connection. If you guys @edenhill @EVODelavega @arnaud-lb have some suggestion, will be nice, thanks!

[arnaud-lb]

Hi @viniciuswebdev,

In php-rdkafka, there is only one rd_kafka_t instance per RdKafka\Producer instance; so you can send multiple messages through a single connection by re-using the sameRdKafka\Producer instance.

Connections are not persistent across multiple PHP requests though; however it could be easy to change php-rdkafka to allow re-using rd_kafka_t instances across PHP requests.

[EVODelavega]

@viniciuswebdev Reusing the same connection over various requests isn't really feasible. PHP is a stateless environment, so keeping a connection alive in between requests is not really supported. Of course, if you're using PHP-FPM, it could be possible for us to add a kafka connection pool to the extension, that can be accessed in a specific way. But features like that require aren't considered high-priority to implement.

What you could do instead, though, is to use something like react/reactPHP to have a php process with an active connection to Kafka and interact with that. React will keep the PHP processes (and all of its resources) alive.

[EVODelavega]

@arnaud-lb The risk with re-using kafka connections are quite similar to, for example, the old mysql extension:

1. Will we add support for default connections (eg mysql_query('some_string'); defaulted to the latest connection resource). I say no, some might say yes. A choice has to be made
2. What is the expected behaviour when the code creates a new Kafka connection to the exact same brokers? Do we return the pooled connection, or not?
3. What about configuration? Should the rd_kafka_conf_t persist, too? Do we close all connections on PHP_MSHUTDOWN or do we expose a method/function to the user to manually close these connections?
4. If a connection persists over several requests, should queue's and topic resources persist, too to finalize async produce/consume calls until after PHP_RSHUTDOWN? When do we cleanup those resources? If the consume/produce queue's are all empty? But what if a given request keeps hitting the same queue because of the pooling?

I Really don't think this is going to be an easy thing to implement, because it changes the way rd_kafka_t resources are managed, and that is the resource that we need to open topics, queues, produce/consume messages etc...

In my opinion, persistent connections are not a good idea, certainly not in PHP. A stateless environment shouldn't keep resources alive in between requests. The only thing PHP can persist in between requests is extension/module setup that you can safely put in PHP_MINIT, and you can safely cleanup in PHP_MSHUTDOWN

[bo2018]

@EVODelavega as described in this url :arnaud-lb/php-rdkafka#14, I found phpkafka also has the similar problem, can you help me ,thanks