Updated Cassandra to 1.0.6 #4

wants to merge 2 commits into


None yet
2 participants

nmilford commented Jan 6, 2012

  • Updated the url to the Cassandra tarball.
  • Changed the JMX port to 7199, which has changed since 0.8 from 8080.
  • Changed the clustername directive in cassandra.yaml

Passes 'mvn test' and also successfully spawns a 1.0.6 cluster.


andreisavu commented Jan 6, 2012

Integration tests are failing both on aws-ec2 & cloudservers-uk.


andreisavu commented Jan 6, 2012

We also need to upgrade the test deps:

-      <version>0.7.0</version>
+      <version>1.0.6</version>

nmilford commented Jan 7, 2012

Integration tests fail because initial_token needs to be explicitly set. Complete WHIRR-469 and you'll be good to go.

The first two nodes get spawned, each getting 50% of the ring

nathan@ip-10-195-47-159:~$ nodetool -h $(hostname) -p 7199 ring
Address DC Rack Status State Load Owns Token datacenter1 rack1 Up Normal 13.48 KB 50.00% 66545260261209709411633801669351785558 datacenter1 rack1 Up Normal 6.67 KB 50.00% 151615851991444325277477453527293838422

This is a successful start up.

nathan@ip-10-116-134-146:~$ tail -15 /var/log/cassandra/system.log
INFO [FlushWriter:1] 2012-01-07 01:02:15,249 Memtable.java (line 277) Completed flushing /var/lib/cassandra/data/system/LocationInfo-hc-3-Data.db (87 bytes)
INFO [main] 2012-01-07 01:02:15,250 StorageService.java (line 627) JOINING: sleeping 30000 ms for pending range setup
INFO [main] 2012-01-07 01:02:45,257 StorageService.java (line 627) JOINING: Starting to bootstrap...
INFO [main] 2012-01-07 01:02:45,269 ColumnFamilyStore.java (line 692) Enqueuing flush of Memtable-LocationInfo@6923467(53/66 serialized/live bytes, 2 ops)
INFO [FlushWriter:1] 2012-01-07 01:02:45,270 Memtable.java (line 240) Writing Memtable-LocationInfo@6923467(53/66 serialized/live bytes, 2 ops)
INFO [FlushWriter:1] 2012-01-07 01:02:45,310 Memtable.java (line 277) Completed flushing /var/lib/cassandra/data/system/LocationInfo-hc-4-Data.db (163 bytes)
INFO [CompactionExecutor:1] 2012-01-07 01:02:45,316 CompactionTask.java (line 113) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-hc-2-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-hc-1-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-hc-4-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo-hc-3-Data.db')]
INFO [main] 2012-01-07 01:02:45,326 StorageService.java (line 948) Node / state jump to normal
INFO [main] 2012-01-07 01:02:45,327 StorageService.java (line 589) Bootstrap/Replace/Move completed! Now serving reads.
INFO [main] 2012-01-07 01:02:45,338 Mx4jTool.java (line 72) Will not load MX4J, mx4j-tools.jar is not in the classpath
INFO [main] 2012-01-07 01:02:45,440 CassandraDaemon.java (line 125) Binding thrift service to /
INFO [main] 2012-01-07 01:02:45,444 CassandraDaemon.java (line 134) Using TFastFramedTransport with a max frame size of 15728640 bytes.
INFO [main] 2012-01-07 01:02:45,448 CassandraDaemon.java (line 161) Using synchronous/threadpool thrift server on / : 9160
INFO [Thread-5] 2012-01-07 01:02:45,451 CassandraDaemon.java (line 213) Listening for thrift clients...

Then it appears that any subsequent Cassandra node fails to start when called.

This is on the 3rd node that failed to start on the first try.

nathan@ip-10-242-47-147:~$ tail -15 /var/log/cassandra/system.log
INFO [main] 2012-01-07 01:02:17,217 StorageService.java (line 627) JOINING: getting bootstrap token
ERROR [main] 2012-01-07 01:02:17,222 AbstractCassandraDaemon.java (line 354) Exception encountered during startup
java.lang.RuntimeException: Every node is a bootstrap source! Please specify an initial token manually or wait for an existing bootstrap operation to finish.
at org.apache.cassandra.dht.BootStrapper.getBootstrapSource(BootStrapper.java:194)
at org.apache.cassandra.dht.BootStrapper.getBalancedToken(BootStrapper.java:150)
at org.apache.cassandra.dht.BootStrapper.getBootstrapToken(BootStrapper.java:145)
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:535)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:472)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:385)
at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:215)
at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:337)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:107)

INFO [StorageServiceShutdownHook] 2012-01-07 01:02:17,251 MessagingService.java (line 478) Shutting down MessageService...
INFO [ACCEPT-/] 2012-01-07 01:02:17,252 MessagingService.java (line 635) MessagingService shutting down server thread.
INFO [StorageServiceShutdownHook] 2012-01-07 01:02:17,252 MessagingService.java (line 494) Waiting for in-progress requests to complete

I previously missed this, because I went into the instances and started the dead nodes manually after Whirr fired it up, not thinking anything of it.

After manually starting the third node it'll take half of another node's tokens.

nathan@ip-10-195-47-159:~$ nodetool -h $(hostname) -p 7199 ring
Address DC Rack Status State Load Owns Token datacenter1 rack1 Up Normal 20.09 KB 25.00% 24009964396092401478711975740380759126 datacenter1 rack1 Up Normal 6.7 KB 25.00% 66545260261209709411633801669351785558 datacenter1 rack1 Up Normal 11.08 KB 50.00% 151615851991444325277477453527293838422


andreisavu commented Jan 7, 2012

Thanks for investigating this. I will start to work on WHIRR-469.


andreisavu commented Jan 14, 2012

I have attached this pull as a .patch to https://issues.apache.org/jira/browse/WHIRR-407 + some changes for node balancing.

@nmilford nmilford closed this Dec 28, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment