synchronize can't be called from trap context

[jiteshonce]

We are replacing our existing ruby-kafka library with rdkafka-ruby in our rails application. We are running rails application with puma. But we are facing multiple exception warnings,

/usr/share/rvm/gems/ruby-3.3.0/gems/set-1.0.3/lib/set.rb:246: warning: Exception in finalizer #<Proc:0x00007f2717b45ad8 /usr/share/rvm/gems/ruby-3.3.0/gems/rdkafka-0.15.0/lib/rdkafka/native_kafka.rb:82 (lambda)>
/usr/share/rvm/gems/ruby-3.3.0/gems/rdkafka-0.15.0/lib/rdkafka/native_kafka.rb:70:in `synchronize': can't be called from trap context (ThreadError)

[mensfeld]

Please provide full readable stacktrace as a text. Otherwise will not be assessed.

[mensfeld]

Also please prepare a workable POC if your expectation is to get help. Help me, so I can help you.

[jiteshonce]

rdkafka-ruby-stacktrace.txt
https://github.com/jiteshonce/rdkafka-syncronize
Here is the attached backtrace and repo link

We are getting issue with our puma 5.6.7 but not getting with puma 5.0

[mensfeld]

@jiteshonce can you explain how this reproduction should be used? exact steps to trigger the behaviour etc?

[mensfeld]

@jiteshonce your stacktrace points be to app/entities/destination_entity.rb - I cannot find such object in the example repro app. Once again I will kindly ask you to provide me with all information and clear path for me to be able to help you.

[jiteshonce]

I am getting this exception on running rails console. I added the related code in https://github.com/jiteshonce/rdkafka-syncronize/blob/main/config/initializers/kafka_queues_init.rb

[mensfeld]

Ok I can reproduce. It is not rdkafka-ruby, it is you running topic creation on initialization and catching errors without explicit admin close.

This is not something you want to do anyway in an initializer the way you do. You should create an explicit rake task for topics mangement or you can use karafka declarative topics available in OSS: https://karafka.io/docs/Declarative-Topics/

that provide bundle exec karafka topics migrate and a DSL for topics definition:

class KarafkaApp < Karafka::App
  routes.draw do
    topic :a do
      config(
        partitions: 6,
        replication_factor: 3,
        'retention.ms': 86_400_000 # 1 day in ms,
        'cleanup.policy': 'delete'
      )

      consumer ConsumerA
    end

    topic :b do
      config(
        partitions: 2,
        replication_factor: 3
        # The rest will be according to the cluster defaults
      )

      consumer ConsumerB
    end
  end
end

if you really want to do what you are doing and I highly discourage you from doing so, you should ensure admin is closed:

# Create Kafka Topic if it does not exist
Rdkafka::Config.logger = Rails.logger
begin
  admin = Rdkafka::Config.new({:"bootstrap.servers" => "localhost:9092"}).admin
  create_topic_handle = admin.create_topic('our.new.topic2', 1, 1)
  create_topic_handle.wait(max_wait_timeout: 1.0)
rescue Rdkafka::RdkafkaError => e
  Rails.logger.warn 'Kafka topic already exists'
rescue StandardError => e
  Rails.logger.error "Kafka error- (kafka_queues_init.rb): #{e.message}"
ensure
  admin.close
end

[mensfeld]

also please note that by doing what you are doing likely you will hit this bug as well: #266

[jiteshonce]

Thanks @mensfeld for help, on closing the admin our problem got resolved.