NameError: instance variable @cluster not defined

[hydrogen18]

Judging from the if statement, this must be the result of a race condition. We're running Cequel in sidekiq when this happens

Stack trace of the frames that are in the cequel gem

/gems/cequel-6b652daac361/lib/cequel/metal/
keyspace.rb: 252:in `remove_instance_variable'
…/gems/cequel-6b652daac361/lib/cequel/metal/
keyspace.rb: 252:in `clear_active_connections!'
…/gems/cequel-6b652daac361/lib/cequel/metal/
keyspace.rb: 318:in `rescue in client_retry'
…/gems/cequel-6b652daac361/lib/cequel/metal/
keyspace.rb: 313:in `client_retry'
…/gems/cequel-6b652daac361/lib/cequel/metal/
keyspace.rb: 214:in `block in execute_with_options'
…cequel-6b652daac361/lib/cequel/metal/
request_logger.rb:  33:in `block in log'
…cequel-6b652daac361/lib/cequel/metal/
request_logger.rb:  33:in `log'
…/gems/cequel-6b652daac361/lib/cequel/metal/
keyspace.rb: 213:in `execute_with_options'
…ler/gems/cequel-6b652daac361/lib/cequel/metal/
batch.rb:  67:in `apply'
…/cequel-6b652daac361/lib/cequel/metal/
batch_manager.rb:  52:in `block in batch'
…/cequel-6b652daac361/lib/cequel/metal/
batch_manager.rb:  52:in `tap'
…/cequel-6b652daac361/lib/cequel/metal/
batch_manager.rb:  52:in `batch'
…ems/cequel-6b652daac361/lib/cequel/record/
callbacks.rb:  34:in `save'
…s/cequel-6b652daac361/lib/cequel/record/
validations.rb:  53:in `save'
…er/gems/cequel-6b652daac361/lib/cequel/record/
dirty.rb:  48:in `save'
…s/cequel-6b652daac361/lib/cequel/record/
persistence.rb:  47:in `block in create'
…s/cequel-6b652daac361/lib/cequel/record/
persistence.rb:  47:in `tap'
…s/cequel-6b652daac361/lib/cequel/record/
persistence.rb:  47:in `create'
…reams/.rbenv/versions/2.3.0/lib/ruby/2.3.0/
delegate.rb:  83:in `method_missing'
…ms/cequel-6b652daac361/lib/cequel/record/
record_set.rb: 930:in `block in method_missing'
…r/gems/cequel-6b652daac361/lib/cequel/record/
scoped.rb:  34:in `with_scope'
…ms/cequel-6b652daac361/lib/cequel/record/
record_set.rb: 930:in `method_missing'
…reams/.rbenv/versions/2.3.0/lib/ruby/2.3.0/
delegate.rb:  83:in `method_missing'

[hydrogen18]

This is the method in question

      def clear_active_connections!
        if defined? @client
          remove_instance_variable(:@client)
        end
        if defined? @client_without_keyspace
          remove_instance_variable(:@client_without_keyspace)
        end
        if defined? @cluster
          @cluster.close
          remove_instance_variable(:@cluster)
        end
      end

I don't see how this could ever work. In MRI, no two threads can run at the same time, but a thread can still be preempted anywhere. Thus, if two threads wind up in this method at the same time they are likely to both wind up inside the 'if block.

[hydrogen18]

This is a bit farther off topic, but the way we use this gem is in a ruby on rails application that runs sidekiq jobs. Each sidekiq job basically just writes things into Cassandra. The web requests read things from Cassandra. I've looked at the underlying architecture of the cassandra driver gem, and it appears to be a "better than naive" implementation. Other than the lock in this class, this gem basically is just a wrapper around that gem.

The behavior in this method also explains other behavior I've seen, namely that the policies you can specify in the Cassandra driver seem to have little effect. Since this gem clears @cluster on the first sign of a problem, this short circuits any change the policy objects have to try and deal with a failing on unresponsive Cassandra node.

MRI isn't exactly known for being a great parallel processing platform, but what I discovered is that Sidekiq has a default of 25 threads. I was seeing the production system deadlock completely when upgrading to Cequel 2.0.x from 1.x. I dropped the thread count to 5 threads in Sidekiq. The deadlocks went away, the system throughput remained the same, and the time spent dropped significantly.

This isn't really a problem, but is something to be aware of. Unless the memory footprint of your application is enormous you are probably better off running a large number of MRI instances each with a small number of application threads.

[alxgsv]

@pezra @hydrogen18 does it possible to use connection pooling somehow to avoid this?

AFAIK, pool was a config attribute in cequel v1.

Should I manage pool by myself now with https://github.com/mperham/connection_pool or it's not possible with cequel v3?

Problem is that I can't decrease concurrency of Sidekiq to 5 :-(

[alxgsv]

@hydrogen18 I've found that underlying https://github.com/datastax/ruby-driver supports pooling since early 2016. Have you had a chance to test it with your setup?

To manage pool, I can configure it like this:

development:
  host: '127.0.0.1'
  port: 9042
  cassandra_options:
    connections_per_remote_node: 10
    connections_per_local_node: 10

But I'm curious if it plays well with cequel, since it should solve your problem, but you didn't mention this.