Writing tags failed with NoHostAvailableException and does not recover

[umbreak]

Tag writer fails to write tags. This happens if the cassandra cluster is started after our application is started. When the cassandra cluster starts afterwards the service, the failure does not recover (even though we keep seeing error logs as if attempting to connect).

It seems to happen also if there is some temporary networking issue with the cluster. However, even when the connectivity issue is fixed, the TagWriter seems to be unable to restore the session.

Weirdly, writes and gets to the CassandraJournal through our application still work. So this seems to affect only the TagWriter.

These are the error log we are seeing (repeated all over):

a.p.cassandra.journal.TagWriter - Writing tags has failed. This means that any eventsByTag query will be out of date. The write will be retried. Reason com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: *** (com.datastax.driver.core.exceptions.TransportException: ***] Cannot connect), *** (com.datastax.driver.core.exceptions.TransportException: [***] Cannot connect), ***(com.datastax.driver.core.exceptions.TransportException: [***] Cannot connect))

 As it can be observed from the logs, we are having a cassandra cluster with 3 nodes.

We are running the latest akka-persistence-cassandra version 0.91.

We think it is something related to the cassandra session object being messed up somehow.

[darshak28]

@umbreak I have one question regarding Journal writes. We are testing behaviour of write-retries and read-retries of Cassandra-journal and the question is whether it works as intended in-case of NoHostavailable exceptions? We are following below test steps and we are seeing journal writes fails. [since Persistent Actor dies]

Scenario:
1)Start the application (this will cause the keyspaces to be created)
2)Send event [ which started Persistent actor]
3)Event processed and seeing data in persistence table
4)Kill cassandra
5)Send event
6) Wait for a minute and start cassandra - [write failed no data in Persistence table for new event] [seeing NoHosatAvailable exception]

Note :There is only one Caasnadra node.

What my understading is regarding write-retries is : It will try to attempt write operation in Casandra DB no.of times whatever value we set of write-retries [i.e. write-reties = 12345].
Please correct me if my understanding is wrong and provide your thoughts on (akka-persistence Cassandra) journal write-retries and read-retries.
[ Consider me as new bie in this area :) ]

[umbreak]

We keep seeing NoHosatAvailable on attempting to write to the TagWriter even after cassandra is back online. Yes, that's the problem.