Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression in 2.5.20 CoordinatedShutdown fails when a stream involving StreamRefs is running #26326

Closed
fredfp opened this Issue Feb 4, 2019 · 3 comments

Comments

Projects
None yet
3 participants
@fredfp
Copy link
Contributor

fredfp commented Feb 4, 2019

Regression introduced by 96692b2, see #26327 for reproducer

Coordinated shutdown times out and leaves actor system running when a stream involving StreamRefs is running.

Can be worked around by adding a kill switch to the stream, and triggering it from a coordinated shutdown hook.

@patriknw

This comment has been minimized.

Copy link
Member

patriknw commented Feb 7, 2019

I'm looking into this

@patriknw patriknw self-assigned this Feb 7, 2019

@patriknw

This comment has been minimized.

Copy link
Member

patriknw commented Feb 7, 2019

It's a deadlock (the synchronized access wasn't that innocent):

"ClusterSpec2-akka.actor.default-dispatcher-4" Id=38 BLOCKED on akka.actor.FunctionRef@1677b0d8 owned by "ClusterSpec2-akka.actor.default-dispatcher-6" Id=40
	at akka.actor.FunctionRef.addWatcher(ActorRef.scala:788)
	-  blocked on akka.actor.FunctionRef@1677b0d8
	at akka.actor.FunctionRef.sendSystemMessage(ActorRef.scala:727)
	at akka.actor.FunctionRef.watch(ActorRef.scala:844)
	-  locked akka.actor.FunctionRef@37ca1eac
	at akka.stream.stage.GraphStageLogic$StageActor.watch(GraphStage.scala:246)
	at akka.stream.impl.streamref.SourceRefStageImpl$$anon$1.observeAndValidateSender(SourceRefImpl.scala:221)
	at akka.stream.impl.streamref.SourceRefStageImpl$$anon$1.$anonfun$initialReceive$1(SourceRefImpl.scala:151)
	at akka.stream.impl.streamref.SourceRefStageImpl$$anon$1.$anonfun$initialReceive$1$adapted(SourceRefImpl.scala:148)
	at akka.stream.impl.streamref.SourceRefStageImpl$$anon$1$$Lambda$824/359665177.apply(Unknown Source)
	at akka.stream.stage.GraphStageLogic$StageActor.internalReceive(GraphStage.scala:232)
	at akka.stream.stage.GraphStageLogic$StageActor.$anonfun$callback$1(GraphStage.scala:198)
	at akka.stream.stage.GraphStageLogic$StageActor.$anonfun$callback$1$adapted(GraphStage.scala:198)
	at akka.stream.stage.GraphStageLogic$StageActor$$Lambda$826/571757874.apply(Unknown Source)
	at akka.stream.impl.fusing.GraphInterpreter.runAsyncInput(GraphInterpreter.scala:454)
	at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:468)
	at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:563)
	at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:745)
	at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:760)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:670)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
	at akka.actor.ActorCell.invoke(ActorCell.scala:561)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)


"ClusterSpec2-akka.actor.default-dispatcher-6" Id=40 BLOCKED on akka.actor.FunctionRef@37ca1eac owned by "ClusterSpec2-akka.actor.default-dispatcher-4" Id=38
	at akka.actor.FunctionRef.addWatcher(ActorRef.scala:788)
	-  blocked on akka.actor.FunctionRef@37ca1eac
	at akka.actor.FunctionRef.sendSystemMessage(ActorRef.scala:727)
	at akka.actor.FunctionRef.watch(ActorRef.scala:844)
	-  locked akka.actor.FunctionRef@1677b0d8
	at akka.stream.stage.GraphStageLogic$StageActor.watch(GraphStage.scala:246)
	at akka.stream.impl.streamref.SinkRefStageImpl$$anon$1.observeAndValidateSender(SinkRefImpl.scala:193)
	at akka.stream.impl.streamref.SinkRefStageImpl$$anon$1.preStart(SinkRefImpl.scala:93)
	at akka.stream.impl.fusing.GraphInterpreter.init(GraphInterpreter.scala:295)
	at akka.stream.impl.fusing.GraphInterpreterShell.init(ActorGraphInterpreter.scala:557)
	at akka.stream.impl.fusing.ActorGraphInterpreter.tryInit(ActorGraphInterpreter.scala:679)
	at akka.stream.impl.fusing.ActorGraphInterpreter.preStart(ActorGraphInterpreter.scala:727)
	at akka.actor.Actor.aroundPreStart(Actor.scala:528)
	at akka.actor.Actor.aroundPreStart$(Actor.scala:528)
	at akka.stream.impl.fusing.ActorGraphInterpreter.aroundPreStart(ActorGraphInterpreter.scala:670)
	at akka.actor.ActorCell.create(ActorCell.scala:656)
	at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:527)
	at akka.actor.ActorCell.systemInvoke(ActorCell.scala:549)
	at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:283)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

patriknw added a commit that referenced this issue Feb 7, 2019

patriknw added a commit that referenced this issue Feb 7, 2019

@fredfp

This comment has been minimized.

Copy link
Contributor Author

fredfp commented Feb 7, 2019

Thanks Patrik, I'm really looking forward to 2.5.21 with your fix included!

patriknw added a commit that referenced this issue Feb 12, 2019

patriknw added a commit that referenced this issue Feb 12, 2019

@patriknw patriknw added this to the 2.5.21 milestone Feb 12, 2019

@patriknw patriknw closed this Feb 12, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.