Cassandra not starting properly in multi-nic environment

[viglesiasce]

When using a machine that has 2 NICs, a fatal issue occurs when starting cassandra:

ERROR 06:57:52,308 Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
    at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1296)
    at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:457)
    at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:671)
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:623)
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:515)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:424)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:554)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:643)
java.lang.RuntimeException: Unable to gossip with any seeds
    at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1296)
    at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:457)
    at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:671)
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:623)
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:515)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:424)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:554)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:643)
Exception encountered during startup: Unable to gossip with any seeds

Which leads to:
https://stackoverflow.com/questions/20690987/apache-cassandra-unable-to-gossip-with-any-seeds

@dmccue found that:

Reason why this doesn't work is because odc-f-28.prc.eucalyptus-systems.com is resolving to the public interface and not the private interface. What is the best way to address this, change the /etc/hosts file or modify cassandra.yaml listen_address to use the private interface IP?

This would need to be altered from node['fqdn'] to something that allows overriding to private ip address
https://github.com/eucalyptus/midokura-cookbook/blob/master/recipes/cassandra.rb#L13

[viglesiasce]

@dmccue if we made the value a an IP pulled from another attribute would that work? We can also default it to node['ipaddress'](which is the default gw interface IP) for the base case.

[dmccue]

Excellent ... just the node['fqdn'] isn't going to work long-term, so swapping to ip based will work best ... thanks vic!

For reference:
https://github.com/eucalyptus/midokura-cookbook/blob/master/recipes/cassandra.rb#L13

execute "CASSANDRA: set listening address" do
 command "sed -i -e 's/localhost/#{node['fqdn']}/g' /etc/cassandra/conf/cassandra.yaml"
end