Safely stop agent heartbeat and handle Disconnected exception

[BrianJKoopman]

Description

This PR adds a check to make sure the heartbeat is still running before trying to stop the looping call, self.heartbeat_call. We also add exception handling for the autobahn.exception.Disconnected exception we've been seeing occasionally when an agent loses its connection to the crossbar server and goes to flush its feed buffer.

Motivation and Context

If an agent loses its connection to the crossbar server the heartbeat looping call might crash before it is stopped when onLeave() is triggered. This hits an assertion error in twisted.

Additionally, when the agent goes to flush its buffer by publishing any data it has that publish call will fail with an autobahn.exception.Disconnected exception. We were already handling the TransportLost exception, so we just add Disconnected.

An example of hitting both of these at the same time can be see in #417.

Resolves #417.

How Has This Been Tested?

I haven't been able to reproduce #417 locally. I've tried by running a small ocs network with the registry and several fake data agents. I then cycle the crossbar server over and over again, hoping to hit the exception just right. Cycling a couple dozen times never hit it.

I'm still going to try to think of a way to reproduce this, but wanted to get the PR open.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.

[jlashner]

Thanks Brian, these changes seem to me like they should help!