Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Disassociated error without any reason. #2141

Closed
VitaliiPerederii opened this issue Jul 6, 2016 · 4 comments
Closed

Disassociated error without any reason. #2141

VitaliiPerederii opened this issue Jul 6, 2016 · 4 comments
Milestone
1.1.0

Comments

@VitaliiPerederii
Copy link

VitaliiPerederii commented Jul 6, 2016 •
edited

Hi all.
I've encountered problem with sudden disassociation when developing system using akka.net remote. Server actor system application is sending a lot of messages (10-20 k/sec )to remote actor. CPU utilization on the server machine is about 100%, thought heartbeat messages is delivered in time. After 3 minutes of working message flow is stopped by the disassociation error. Here is config:

akka {
    stdout-loglevel = DEBUG
    loglevel = DEBUG
    actor {
        debug {  
          receive = on 
          autoreceive = on
          lifecycle = on
          event-stream = on
          unhandled = on
          fsm = on
        }   
        serialize-creators = on
        akka.actor.guardian-supervisor-strategy = "Akka.Actor.SupervisorStrategy.StoppingStrategy"
        provider = "Akka.Remote.RemoteActorRefProvider, Akka.Remote"
        serializers {
          wire = "Akka.Serialization.WireSerializer, Akka.Serialization.Wire"
        }
        serialization-bindings {
            "System.Object" = wire
        }
        deployment {
            /CDRRouterActor/Actor0 {
              remote = "akka.tcp://DeployTarget@127.0.0.1:8090"
            }
        }
    }
    remote {
        helios.tcp {
            port = 0
            hostname = 127.0.0.1
        }
        watch-failure-detector {
            threshold = 100
            acceptable-heartbeat-pause = 1000 s
        }
        transport-failure-detector {
            heartbeat-interval = 1 s # default 4s
            acceptable-heartbeat-pause = 1000 s # default 10s
            min-std-deviation = 12 s
        }
    }
}

The log is pretty the same on both nodes:

[DEBUG][7/6/2016 2:26:43 PM][Thread 0014][[akka://Reconcilation/system/remote-watcher]] Sending Heartbeat to [akka.tcp://DeployTarget@127.0.0.1:8090]
[DEBUG][7/6/2016 2:26:44 PM][Thread 0016][[akka://Reconcilation/system/remote-watcher]] Sending Heartbeat to [akka.tcp://DeployTarget@127.0.0.1:8090]
[DEBUG][7/6/2016 2:26:45 PM][Thread 0016][[akka://Reconcilation/system/remote-watcher]] Sending Heartbeat to [akka.tcp://DeployTarget@127.0.0.1:8090]
[DEBUG][7/6/2016 2:26:45 PM][Thread 0015][[akka://Reconcilation/system/remote-watcher]] Received heartbeat rsp from [akka.tcp://DeployTarget@127.0.0.1:8090]
[DEBUG][7/6/2016 2:26:45 PM][Thread 0017][[akka://Reconcilation/system/remote-watcher]] Received heartbeat rsp from [akka.tcp://DeployTarget@127.0.0.1:8090]
[DEBUG][7/6/2016 2:26:45 PM][Thread 0017][[akka://Reconcilation/system/remote-watcher]] Received heartbeat rsp from [akka.tcp://DeployTarget@127.0.0.1:8090]

[DEBUG][7/6/2016 2:26:46 PM][Thread 0017][[akka://Reconcilation/system/remote-watcher]] Sending Heartbeat to [akka.tcp://DeployTarget@127.0.0.1:8090]
[DEBUG][7/6/2016 2:26:46 PM][Thread 0017][akka://Reconcilation/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3a%2f%2fDeployTarget%40127.0.0.1%3a8090-1] Stopped
[DEBUG][7/6/2016 2:26:46 PM][Thread 0015][akka://Reconcilation/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fDeployTarget%40127.0.0.1%3a8090-1/endpointWriter/endpointReader-akka.tcp%3a%2f%2fDeployTarget%40127.0.0.1%3a8090-1] Stopped
[DEBUG][7/6/2016 2:26:46 PM][Thread 0015][akka://Reconcilation/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fDeployTarget%40127.0.0.1%3a8090-1/endpointWriter] received AutoReceiveMessage <Terminated>: [akka://Reconcilation/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fDeployTarget%40127.0.0.1%3a8090-1/endpointWriter/endpointReader-akka.tcp%3a%2f%2fDeployTarget%40127.0.0.1%3a8090-1] - ExistenceConfirmed=True
[DEBUG][7/6/2016 2:26:46 PM][Thread 0015][[akka://Reconcilation/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fDeployTarget%40127.0.0.1%3a8090-1/endpointWriter]] Disassociated [akka.tcp://Reconcilation@127.0.0.1:23423] -> akka.tcp://DeployTarget@127.0.0.1:8090
[WARNING][7/6/2016 2:26:46 PM][Thread 0017][[akka://Reconcilation/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fDeployTarget%40127.0.0.1%3a8090-1]] Association with remote system akka.tcp://DeployTarget@127.0.0.1:8090 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
   at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)]
[DEBUG][06.07.2016 14:39:11][Thread 0011][[akka://DeployTarget/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fReconcilation%40127.0.0.1%3a23775-1/endpointWriter]] Disassociated [akka.tcp://DeployTarget@127.0.0.1:8090] <- akka.tcp://Reconcilation@127.0.0.1:23775
[DEBUG][06.07.2016 14:39:11][Thread 0011][akka://DeployTarget/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fReconcilation%40127.0.0.1%3a23775-1/endpointWriter] Stopped
[INFO][06.07.2016 14:39:11][Thread 0004][akka://DeployTarget/deadLetters] Message HeartbeatRsp from akka://DeployTarget/system/remote-watcher to akka://DeployTarget/deadLetters was not delivered. 1 dead letters encountered.
[INFO][06.07.2016 14:39:11][Thread 0004][akka://DeployTarget/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3a%2f%2fDeployTarget%40127.0.0.1%3a23776-1] Message DisassociateUnderlying from akka://DeployTarget/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fReconcilation%40127.0.0.1%3a23775-1/endpointWriter to akka://DeployTarget/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3a%2f%2fDeployTarget%40127.0.0.1%3a23776-1 was not delivered. 2 dead letters encountered.
[INFO][06.07.2016 14:39:11][Thread 0004][akka://DeployTarget/deadLetters] Message Heartbeat from akka://DeployTarget/system/remote-watcher to akka://DeployTarget/deadLetters was not delivered. 3 dead letters encountered.
[INFO][06.07.2016 14:39:11][Thread 0004][akka://DeployTarget/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fReconcilation%40127.0.0.1%3a23775-1/endpointWriter] Message BackoffTimer from akka://DeployTarget/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fReconcilation%40127.0.0.1%3a23775-1/endpointWriter to akka://DeployTarget/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fReconcilation%40127.0.0.1%3a23775-1/endpointWriter was not delivered. 4 dead letters encountered.
[DEBUG][06.07.2016 14:39:11][Thread 0010][akka://DeployTarget/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fReconcilation%40127.0.0.1%3a23775-1] received AutoReceiveMessage <Terminated>: [akka://DeployTarget/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fReconcilation%40127.0.0.1%3a23775-1/endpointWriter] - ExistenceConfirmed=True
[DEBUG][06.07.2016 14:39:12][Thread 0009][[akka://DeployTarget/system/remote-watcher]] Sending Heartbeat to [akka.tcp://Reconcilation@127.0.0.1:23775]

Could you please tell me what is the possible reason of EndpointDisassociatedException? Any other exceptions in my code did not appear. Network connections if OK either. Testing within one machine gives the same result.
@Aaronontheweb
Copy link
Member

We're releasing Akka.NET 1.1 which completely overhauls Akka.Remote and the underlying transport layer. We were supposed to stage it yesterday but have paused until #2138 is resolved. 1.1 resolves this issue.

@Aaronontheweb
Copy link
Member

In the meantime if you need immediately help, try the nightly builds: http://getakka.net/docs/akka-developers/nightly-builds

@VitaliiPerederii
Copy link
Author

Thanks a lot!

@Aaronontheweb
Copy link
Member

No problem @VitaliiPerederii - and thanks for reporting this issue!

@alexvaluyskiy alexvaluyskiy added this to the 1.1.0 milestone Jul 6, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
1.1.0
Development

No branches or pull requests
3 participants
@Aaronontheweb @alexvaluyskiy @VitaliiPerederii