Reconnect issues after broker restart

[mubarak]

It appears from our environment that when brokers goes down and comes back up, ekaf client is keep trying to create more connections, we have seen around 16K ekaf connections to brokers (TCP connection state is ESTABLISHED) and we have to restart an app (which embeds ekaf client, number of ekaf connections went down to around 100 after restart of brokers).

Are we missing any config(s) to reconnect ekaf client after broker restart or is it an expected behavior?
Please let us know. Thanks.

[qrilka]

Probably you could get hit by scenario described in #6 (which is the result of #9 ) ?

[bosky101]

Can you verify the commit if your deployment. Ekaf env will also help.

~B

On 30-Jan-2015, at 12:18 am, Mubarak Seyed notifications@github.com wrote:

It appears from our environment that when brokers goes down and comes back up, ekaf client is keep trying to create more connections, we have seen around 16K ekaf connections to brokers (TCP connection state is ESTABLISHED) and we have to restart an app (which embeds ekaf client, number of ekaf connections went down to around 100 after restart of brokers).

Are we missing any config(s) to reconnect ekaf client after broker restart or is it an expected behavior?
Please let us know. Thanks.

—
Reply to this email directly or view it on GitHub.

[mubarak]

We are using commit e45f136

[bosky101]

Can you share your ekaf env entry (swapping out the broker host for **) - i can exactly comment then.

[mubarak]

[{ekaf, [ {ekaf_per_partition_workers, 20}, {ekaf_per_partition_workers_max, 100}, {ekaf_buffer_ttl, 1000} ]} ].

      %%% Callbacks for server state and worker (connection) state
      application:set_env(ekaf, ?EKAF_CALLBACK_DOWNTIME_REPLAYED, {ecu_kafka_client, kafka_callback}),
      application:set_env(ekaf, ?EKAF_CALLBACK_DOWNTIME_SAVED, {ecu_kafka_client, kafka_callback}),
      application:set_env(ekaf, ?EKAF_CALLBACK_FLUSH, {ecu_kafka_client, kafka_callback}),
      application:set_env(ekaf, ?EKAF_CALLBACK_FLUSHED_REPLIED, {ecu_kafka_client, kafka_callback}),
      application:set_env(ekaf, ?EKAF_CALLBACK_TIME_TO_CONNECT, {ecu_kafka_client, kafka_callback}),
      application:set_env(ekaf, ?EKAF_CALLBACK_WORKER_DOWN, {ecu_kafka_client, kafka_callback}),
      application:set_env(ekaf, ?EKAF_CALLBACK_WORKER_UP, {ecu_kafka_client, kafka_callback}),

      PartitionWorkers = ekaf_per_partition_workers,
      PartitionWorkersMax = ekaf_per_partition_workers_max,
      BufferTtl = ekaf_buffer_ttl,

      application:set_env(ekaf, PartitionWorkers, application:get_env(ekaf, PartitionWorkers, 20)),
      application:set_env(ekaf, PartitionWorkersMax, application:get_env(ekaf, PartitionWorkersMax, 100)),
      application:set_env(ekaf, ekaf_bootstrap_broker, {Host,Port}),
      application:set_env(ekaf, BufferTtl, application:get_env(ekaf, BufferTtl, 1000)),

      {ok, _} = application:ensure_all_started(ekaf);

[bosky101]

and how many partitions do you have for your topic ?

N partitions * M number of workers per partition = N * M tcp connections per partition

did you happen to log how many messages were saved during the downtime (in your kafka_callback).
also how long was the downtime ( can be graphed by WORKER_DOWN, WORKER_UP if you plot this)

once connection is regained after a downtime - all unsent messages are then sent. if the downtime was long, there may be lot of pending messages - so this could contribute.

[nwt]

@bosky101: I'm going to attempt to pick this up where @mubarak left off.

I did some experimentation and noticed that ekaf_server will quickly fill its message queue with connect and {timeout, _, <<"reconnect">>} events when #ekaf_server.broker is down. This happens because ekaf_server:ready({timeout, _, <<"reconnect">>, _) will start two <<"reconnect">> timers–one directly and the other indirectly when it sends the connect event (since ekaf_socket:open/1 will fail in the resulting ekaf_server:ready(connect, _) call).

[bosky101]

@nwt Sorry couldn't reply earlier. will look into this as soon as possible.

this pull request #14, also identifies & fixes the issue - once i add some tests - and add some docs - will look to merge into master.

[nwt]

@bosky101: I can confirm that #14 (and specifically ericliang@8ec0619) resolves this issue for us. Thanks for pointing it out.