Skip to content

HubNet Internals

Jump to bottom Edit New page
estberg edited this page Aug 9, 2017 · 1 revision

(the "I" here is Josh Cough)

This is rather broad, but that's ok with me. There is a lot there that could use writing up. I'll break it down into a few categories now, and maybe more later. Some of these may not need to be documented at all:

Protocol design (including messages and session)

HubNet, like any network protocol, consists of messages (in a particular format) and rules for exchanging messages (the session).

Message

HubNet uses Java object serialization. Scala case classes are very much like documentation since (at least in this case), they contain only the name of the message and the fields (as well as the types of the fields). Given these two important facts, it suffices to simply show the case classes here for documentation on what the messages are. A few simple notes though. 1) I've intentionally left the plot messages out of the code below. They are too confusing. 2) The first message sent by both the client and the server is a java.lang.String (documented in the Session section), and so that is also left out of this list.

trait Message extends Serializable
case class HandshakeFromClient(userId: String, clientType: String) extends Message
case class HandshakeFromServer(activityName: String, interfaceSpecList: Iterable[AnyRef]) extends Message
case class LoginFailure(content: String) extends Message
case class OverrideMessage(data: Any, clear: Boolean) extends Message
case class ExitMessage(reason:String) extends Message
case object EnterMessage extends Message
case class WidgetControl(content: AnyRef, tag: String) extends Message
case object DisableView extends Message
case object ClearOverrideMessage extends Message
case class ActivityCommand(tag: String, content: AnyRef) extends Message
case class AgentPerspectiveMessage(bytes: Array[Byte]) extends Message
case class ViewUpdate(worldData: Array[Byte]) extends Message
object Text {
  sealed trait MessageType extends Serializable
  object MessageType {
    case object USER extends MessageType
    case object TEXT extends MessageType
    case object CLEAR extends MessageType
  }
}
case class Text(content: String, messageType: Text.MessageType) extends Message

Session

Clients connect to HubNet by opening a socket connection to a particular port (the port is chosen automatically by HubNet when the hubnet-reset primitive is called). Once the connection is established, users must complete the following handshake:

  1. Client sends a java.lang.String which contains the version of NetLogo they they are running. Ex: "NetLogo 5.0RC2".
  2. Server sends a java.lang.String which contains the version of NetLogo they they are running. Ex: "NetLogo 5.0RC2".

Note: If the versions don't match, the server should probably just send a logout message instead, and disconnect. This situation is handled slightly differently on the abmplus branch, explained below.

  1. Client sends a HandshakeFromClient message containing the desired username and the client type.
  2. Server responds.
  • If the version of the client (send in message 1) matches the version of the server, then a LoginFailure is sent.
  • If the username of the client is null or empty, then a LoginFailure is sent.
  • If the username of the client is already in use by another user, then a LoginFailure is sent.
  • Otherwise, the server responds with a one, or many messages.
    • First, it sends a HandshakeFromServer message containing the model name, and the client interface specification. The client interface spec will be documented later.
    • Then, if view mirroring is on, the server will send every client a ViewUpdate message containing a full view update (more on that later).
    • Finally, if plot mirroring is on, the server might send a bunch of plot related messages. The design of plot mirroring in HubNet is so bad that it isn't worth documenting. It should be scrapped entirely, or redone in a way similar to the view mirroring, where only diffs are sent.
  1. The client sends an EnterMessage. At this time, the enter message is put into the HubNet queue, and can be processed with the HubNet primitives. Essentially, the client is in, the handshake is complete.

After the handshake is complete, the server and client are free to send ActivityMessages and WidgetControl messages, as well as a few others, all explained below.

Headless HubNet

Directions for running HubNet headlessly are already written up in the formal documentation. See "Example (HubNet headless)" at Controlling API.

Getting HubNet to run headlessly was a relatively straightforward refactoring of HubNet code, and so should require very little documentation here. Essentially, in 4.1 and below, all of the HubNet server code was terribly entangled with GUI code. I simply managed to unentangle it.

Future of HubNet

My brief view of the future of HubNet is pretty simple. I'd like to replace the Java client with a JavaScript (or Dart) client.

Text protocol

The text protocol (aka Raw protocol, and Binary protocol) is an experimental replacement for Java Object Serialization for HubNet messages. It currently exists in two different places:

This was the first implementation. The branch supports both Java Object Serialization and the Text Protocol. Upon connecting, clients first send which protocol they want to use (I believe that this change means that connecting to a running HubNet server with an older version of HubNet would cause the client to fail with an ugly error message, but this is something that needs to be tested). The idea behind supporting both was that we would not have to change our existing clients over from Java Object Serialization to the Text protocol. The primary reason for not doing that is that many messages (including view and plot messages) are currently unsupported in the Text protocol. This does currently mean that any clients using the text protocol won't be able to support a view. Obviously, I'd like to fix that at some point.

It should be noted here that the Text protocol is simply something I whipped up. While it works, it could easily be replaced with something like Protobuf. It's even possible that it could be replaced with JSON, but that could result in a lot more network traffic. The Text protocol is designed to be as compact as possible.

This was the second implementation and is basically a copy of the original with a few simple modifications. It exists in order to support the hubnet-proxy extension which I will explain below.

Proxy extension

The hubnet-proxy extension works in conjunction with the Text protocol to allow NetLogo 5.0 HubNet clients to be written in different languages. The extension lives in https://github.com/NetLogo/HubNetClients. The extension is designed pretty simply. First, hubnet-proxy:start 9999 is called, which opens up a ServerSocket on port 9999. Clients can connect to this port, and they must send and receive messages using the Text protocol described above. When a client connects, the proxy then creates its own socket connection to the HubNet server (usually on port 9173). Any messages from the client are translated from the Text protocol into Java (actually Scala) messages, and sent to the HubNet server via the socket connection using Java Object Serialization. Any messages sent from the server to the client are first translated to the Text protocol and written back out to the client.

Client sends {text message} -> hubnet-proxy sends {java message} -> HubNet
HubNet sends {java message} -> hubnet-proxy sends {text message} -> Client

As previously mentioned, not all messages are supported in the Text protocol. As a result, clients using the hubnet-proxy can't currently support the view.

The build the hubnet-proxy extension, clone the repo, put it into the extensions directory of NetLogo 5.0 (last tested with RC2 from source though, not the actual release), but rename the directory hubnet-proxy instead of HubNetClients. Then go into the directory and type make.

Python client

I wrote a simple command-line python client that uses the Text Protocol to send messages to HubNet. The client works with the Template model and simply sends up, down, left, or right messages to HubNet, moving its turtle around at random. It ignores all messages coming from the server.

In https://github.com/NetLogo/HubNetClients/tree/hubnet-protocols/src/main/python there is an example client that works directly with the hubnet-protocols branch in git on Assembla. I believe it should work with the Template model in the models library, requiring no modifications.

In https://github.com/NetLogo/HubNetClients/tree/master/src/main/python there is an example client that works with the hubnet-proxy extension to connect to NetLogo 5.0RC2. It can be run with any machine running Python. The client must connect to a modified version of the Template model which can be found here: https://github.com/NetLogo/HubNetClients/blob/master/src/main/netlogo/Template-Python.nlogo.

UPDATE

Below, I describe using JSON for messages sent from JavaScript. Maybe we should abandon the Text Protocol and just use JSON instead, from Python. The advantage of using the Text protocol is that it's more compact (basically maximally compact). It is something to consider. I'm around to ask questions to about this stuff.

Web Sockets and JavaScript Client

Unfortunately, JavaScript cannot open a regular Socket. WebSockets are a new (HTML 5, I think) form of bi-directional communication between JavaScript clients and a web server. I've create an extension called hubnet-web (https://github.com/NetLogo/HubNet-Web-Extension) that uses WebSockets to allow a JavaScript client to connect to HubNet. It works with NetLogo 5.0RC2 and is very similar to the HubNet proxy extension.

Users first call hubnet-reset, and then call hubnet-web:start 9999. hubnet-web:start 9999 creates a WebSocket server in the same JVM as NetLogo. JavaScript clients can then connect to the WebSocket and start sending messages. Unlike the hubnet-proxy extension, which uses the Text protocol for its messages, the hubnet-web extension encodes its messages in JSON, which is much easier from JavaScript. Example messages can be found in client.html which resides in the root directory of the project. I'm not going to document the messages here, for a couple of reasons: 1) I'm not sure if this project is going anywhere 2) The messages are probably a moving target 3) I can answer questions about them if anyone asks 4) Examples are available in the code.

Eventually, it might be nice to replace Java sockets and Java serialization altogether, and just use web sockets and JSON messages for HubNet entirely. If we still want the Java client to connect, we can replace the socket calls in it with a WebSocket client. Simple google searches show that such clients exist.

Client Extension

The hubnet-client extension allows for P2P communication between HubNet servers. It allows NetLogo to connect to HubNet as if it was a client. This allows for the possibility of building intelligent clients in NetLogo. Like the hubnet-proxy extension, the code lives at https://github.com/NetLogo/HubNetClients/tree/master/src/main/scala

https://github.com/NetLogo/HubNetClients

https://github.com/NetLogo/HubNetClients contains not only the proxy and client extensions, and the python client, but examples of building clients in Java, and Scala (using Java Object Serialization). These clients have been tested against 5.0RC2.

Add a custom footer

Home

Advanced Users

Contributors

Internal Home

NetLogo Web

Wiki Issues, Questions, Comments

Licenses

Known Issues

Clone this wiki locally