Skip to content

Commit

Permalink
Create a new session on 404 when refreshing (closes #5)
Browse files Browse the repository at this point in the history
The consul agent can report a false negative serfHealth status.
When that happens the consul server removes the faulty member.
This has two consequences :
1 - all the sessions associated with the node are deleted
2 - the creation of a new session might still fail with
500,Internal Server Error, Check 'serfHealth' is in critical state

This commit is an attempt to be more resilient in this cases and keep
trying to create a new session. If it succeded the system will
keep working as if nothing happened, otherwise after the max num of retry
the constructr machine will terminate the akka system.

The default constructr machines configuration don't give consul enough
time to recover under these circumstances. I suggest to change
the configuration for production env like to something like :

  coordination-timeout = 10 seconds
  nr-of-retries        = 10
  refresh-interval     = 60 seconds
  retry-delay          = 10 seconds
  ttl-factor           = 5.0
  • Loading branch information
Berardino la Torre committed Sep 16, 2016
1 parent b3f3258 commit 18ae928
Showing 1 changed file with 5 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,11 @@ final class ConsulCoordination(
val sessionId = stateSession.getOrElse(throw new IllegalStateException("It wasn't possible to get a valid Consul `sessionId` for refreshing"))
val uri = sessionUri.withPath(sessionUri.path / "renew" / sessionId)
send(Put(uri)).flatMap {
case HttpResponse(OK, _, entity, _) => ignore(entity).map(_ => Done)
case HttpResponse(OK, _, entity, _) => ignore(entity).map(_ => Done)
case HttpResponse(NotFound, _, entity, _) => ignore(entity).flatMap { _ =>
logger.warning("Unable to refresh, session {} not found. Creating a new one", sessionId)
createIfNotExist(self, ttl)
}.map(_ => Done)
case HttpResponse(other, _, entity, _) => ignore(entity).map(_ => throw UnexpectedStatusCode(uri, other))
}
}
Expand Down

0 comments on commit 18ae928

Please sign in to comment.