Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Transition from Active to NoState not allowed #29383

Closed
johanandren opened this issue Jul 14, 2020 · 3 comments
Closed

Transition from Active to NoState not allowed #29383

johanandren opened this issue Jul 14, 2020 · 3 comments
Assignees
@johanandren
Labels
3 - in progress Someone is working on this ticket bug t:cluster-sharding
Milestone
2.6.8

Comments

@johanandren
Copy link
Member

johanandren commented Jul 14, 2020 •
edited

Reported in gitter by @mikelalvarezgo

java.lang.IllegalArgumentException: Transition from Active(Actor[akka://ActorSystem/system/sharding/class/72/fc8975ce-44f6-4bd4-95df-31d8c39bfa41#2055504332]) to NoState not allowed at 
akka.cluster.sharding.Shard$Active.transition(Shard.scala:230) at 
akka.cluster.sharding.Shard$Entities.removeEntity(Shard.scala:289) at 
akka.cluster.sharding.Shard.entityTerminated(Shard.scala:850) at 
akka.cluster.sharding.Shard$$anonfun$idle$1.applyOrElse(Shard.scala:569) at 
akka.actor.Actor.aroundReceive(Actor.scala:537) at akka.actor.Actor.aroundReceive$(Actor.scala:535) at 
akka.cluster.sharding.Shard.akka$actor$Timers$$super$aroundReceive(Shard.scala:386) at 
akka.actor.Timers.aroundReceive(Timers.scala:56) at akka.actor.Timers.aroundReceive$(Timers.scala:41) at 
akka.cluster.sharding.Shard.aroundReceive(Shard.scala:386) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577) at 
akka.actor.dungeon.DeathWatch.$anonfun$receivedTerminated$1(DeathWatch.scala:70) at 
akka.actor.dungeon.DeathWatch.$anonfun$receivedTerminated$1$adapted(DeathWatch.scala:64) at 
scala.Option.foreach(Option.scala:407) at akka.actor.dungeon.DeathWatch.receivedTerminated(DeathWatch.scala:64) at 
akka.actor.dungeon.DeathWatch.receivedTerminated$(DeathWatch.scala:63) at 
akka.actor.ActorCell.receivedTerminated(ActorCell.scala:410) at 
akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:562) at akka.actor.ActorCell.invoke(ActorCell.scala:545) at 
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) at akka.dispatch.Mailbox.run(Mailbox.scala:231) at 
kamon.instrumentation.executor.ExecutorInstrumentation$InstrumentedForkJoinPool$TimingRunnable.run(ExecutorInstrume
ntation.scala:662) at 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48) at 
java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at 
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at 
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at 
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
@johanandren johanandren added bug 0 - new Ticket is unclear on it's purpose or if it is valid or not t:cluster-sharding labels Jul 14, 2020
@johanandren johanandren self-assigned this Jul 14, 2020
@johanandren
Copy link
Member Author

I think this is because we don't allow termination without prior passivation, when not using remember entities, but I'm guessing you used to be able to do that, so a regression.

A workaround until we fix is to passivate (https://doc.akka.io/docs/akka/current/cluster-sharding.html#passivation ) instead of immediate stopp of actors.

@johanandren johanandren added 3 - in progress Someone is working on this ticket and removed 0 - new Ticket is unclear on it's purpose or if it is valid or not labels Jul 14, 2020
@mikelalvarezgo
Copy link

mikelalvarezgo commented Jul 14, 2020 •
edited

Hi @johanandren , here is the configuration we hace that creates the issue:

cluster {
    sharding {
      waiting-for-state-timeout = 5 s
      passivate-idle-entity-after = off
      state-store-mode = ddata

      least-shard-allocation-strategy {
        rebalance-threshold = 2
        max-simultaneous-rebalance = 1
      }

      use-dispatcher = "akka.sharding-dispatcher"
    }

Anyway we will try to activate the passivation

johanandren added a commit to johanandren/akka that referenced this issue Jul 14, 2020
@johanandren
Allow entities to stop by terminating in sharding without remember en…
64426e3 
…tities akka#29383

We missed an allowed transition from running/active to stopped/NoState in shard
when the logic was rewritten.
@johanandren johanandren mentioned this issue Jul 14, 2020
Merged
johanandren added a commit to johanandren/akka that referenced this issue Jul 14, 2020
@johanandren
Allow entities to stop by terminating in sharding without remember en…
5807409 
…tities akka#29383

We missed an allowed transition from running/active to stopped/NoState in shard
when the logic was rewritten.
johanandren added a commit that referenced this issue Jul 15, 2020
@johanandren
Allow entities to stop by terminating without remember entities (#29384)
01edcab 
* Allow entities to stop by terminating in sharding without remember entities #29383
  We missed an allowed transition from running/active to stopped/NoState in shard. 
  when the logic was rewritten.
* Add a toggle to opt-in crash shard on illegal state transitions
  Default is logging an error and not crashing shard and all other entities, our tests have the toggle enabled.
* A fix for passivation when not using remember entities fixing #29359 and possibly #27549
@johanandren johanandren added this to the 2.6.8 milestone Jul 15, 2020
@johanandren
Copy link
Member Author

johanandren commented Jul 16, 2020 •
edited

We have now released Akka 2.6.8 with a fix for this, thanks for reporting so that we could do a swift fix @mikelalvarezgo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@johanandren johanandren

Labels
3 - in progress Someone is working on this ticket bug t:cluster-sharding
Projects
None yet
Milestone
2.6.8
Development

No branches or pull requests
2 participants
@johanandren @mikelalvarezgo