New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

can not add node with cassandra ami #107

Closed
asias opened this Issue Jun 8, 2016 · 4 comments

Comments

Projects
None yet
2 participants
@asias
Contributor

asias commented Jun 8, 2016

Lucas, we have issue adding the 4th node with cas AMI, it hangs there
when I login into it. I saw this twice.

Cluster started with these options:
--clustername asias-test-long-cassandra-db-cluster-a455508d
--bootstrap true --totalnodes 1 --seeds 172.30.0.19 --version
community --release 2.1.8

Waiting for nodetool...
The cluster is now in it's finalization phase. This should only take a moment...

Note: You can also use CTRL+C to view the logs if desired:
AMI log: ~/datastax_ami/ami.log
Cassandra log: /var/log/cassandra/system.log

The logs says:


INFO  [SharedPool-Worker-3] 2016-06-03 06:14:51,705 Gossiper.java:954
- InetAddress /172.30.0.233 is now UP
INFO  [SharedPool-Worker-14] 2016-06-03 06:14:51,707 Gossiper.java:954
- InetAddress /172.30.0.18 is now UP
ERROR [main] 2016-06-03 06:14:52,487 CassandraDaemon.java:541 -
Exception encountered during startup
java.lang.UnsupportedOperationException: Other
bootstrapping/leaving/moving nodes detected, cannot bootstrap while
cassandra.consistent.rangemovement is true
        at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:552)
~[apache-cassandra-2.1.8.jar:2.1.8]
        at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:777)
~[apache-cassandra-2.1.8.jar:2.1.8]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:714)
~[apache-cassandra-2.1.8.jar:2.1.8]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:605)
~[apache-cassandra-2.1.8.jar:2.1.8]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378)
[apache-cassandra-2.1.8.jar:2.1.8]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:524)
[apache-cassandra-2.1.8.jar:2.1.8]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:613)
[apache-cassandra-2.1.8.jar:2.1.8]
WARN  [StorageServiceShutdownHook] 2016-06-03 06:14:52,553
Gossiper.java:1418 - No local state or state is in silent shutdown,
not announcing shutdown
INFO  [StorageServiceShutdownHook] 2016-06-03 06:14:52,554
MessagingService.java:708 - Waiting for messaging service to quiesce
INFO  [ACCEPT-/172.30.0.233] 2016-06-03 06:14:52,558
MessagingService.java:958 - MessagingService has terminated the
accept() thread

If I restart cassandra

sudo /etc/init.d/cassandra restart

it can join the cluster with no problem.

@asias

This comment has been minimized.

Show comment
Hide comment
@asias

asias Jul 26, 2016

Contributor

@lmr ping

Contributor

asias commented Jul 26, 2016

@lmr ping

@lmr

This comment has been minimized.

Show comment
Hide comment
@lmr

lmr Jul 26, 2016

Contributor

Sorry. I think we can then make the cassandra- cluster class to incorporate the restart procedure?

Contributor

lmr commented Jul 26, 2016

Sorry. I think we can then make the cassandra- cluster class to incorporate the restart procedure?

@lmr

This comment has been minimized.

Show comment
Hide comment
@lmr

lmr Jul 31, 2016

Contributor

I was looking into this... it seems that we need to implement some sort of smart log watching - when an error is detected and the service could not be started, try to restart it again. Let me see if I can think of something.

Contributor

lmr commented Jul 31, 2016

I was looking into this... it seems that we need to implement some sort of smart log watching - when an error is detected and the service could not be started, try to restart it again. Let me see if I can think of something.

@lmr

This comment has been minimized.

Show comment
Hide comment
@lmr

lmr Aug 1, 2016

Contributor

Solved per #153.

Contributor

lmr commented Aug 1, 2016

Solved per #153.

@lmr lmr closed this Aug 1, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment