Socket API in ZeroMQ

[xp1632]

https://zeromq.org/socket-api/

What's the difference with conventional socket?

• While sockets are standard API for network programming. The socket-based API in ZeroMQ that we used in
  juptyer_client protocol, is different.

1. Discrete messages

• Conventional Sockets present a synchronous interface to SOCK_STREAM (connection-oriented reliable byte streams) or SOCK_DGRAM(connection-less unreliable datagrams).
  • In comparison, ZeroMQ sockets present an abstraction of asynchronous message queue, with exact queueing semantics depending on socket type in use.
  • ZeroMQ sockets transfer discrete messages where conventional sockets transfer streams of bytes or discrete datagrams

2. Asynchronous

• Also, ZeroMQ sockets being asynchronous means that the timing of physical connection setup, tear down, reconnect and effective delivery are transparent to the use and organized by ZeroMQ itself.
• Further, while a peer is unavailable to receive messages, the messages may be queued in the event

3. Many to many

• Conventional sockets allow only strict one-to-one (two peers), many-to-one(many clients, one server), or in some
  cases one-to-many(multicast) relationships.
• ZeroMQ sockets may be connected to multiple endpoints with the exception of PAIR sockets,
  while simultaneously accepting incoming connections from multiple endpoints bound to the socket, thus allow
  many to many relationships.

---

This many to many , reminds me of the characteristic of Jupyter, where a kernel could deal with multiple frontends
at the same time.

[xp1632]

Since in the jupyter_client where we define the protocol of communicating with Kernel and FrontEnd, only Socket
with Type: ROUTER, DEAL, SUB,PUB are used. We'll focus on these four types.

---

ZeroMQ Socket Lifetime:

• ZeroMQ sockets have a life in four parts:

1. Creating and destroying sockets
2. Configuring sockets by setting option on them and checking them if necessary
3. Plugging sockets into the network by creating connections to and from them
4. Using the sockets to carry data by writing and receiving messages on them

---

Bind vs Connect

• With ZeroMQ sockets, it doesn't matter who binds and who connects.
• Normally the server used Bind while client used Connect.
• A general Rule is to use bind from the most stable points in the architecture, and use connect from dynamic
  components with unstable endpoints.
• If in peer-to-peer, we can't figure out which parts are more stable, we could consider a stable device in the middle
  which all sides can connect to.

Queue underlying ZeroMQ connection

• ZeroMQ creates queues per underlying connection, if a socket is connected to three peer sockets, then there are
  three messages queues behind the scenes.

-With Bind, you allow peers to connect to you, thus you don't know how many peers there will be
and you cannot create the queues in advance.

• Queues are created when individual peers connect to the bound socket

-With Connect, ZeroMQ would create a single queue immediately because it knows that there's going to be
at least a single peer.

• This applies to all socket types except ROUTER, because for ROUTER, queues are only created after the peer we connect to has acknowledge our connection

-Thus, there's no queue to store message to, when sending a message to bound socket with no peers, or a ROUTER with no live connections.

[xp1632]

Messaging Patterns based on sockets with matching types

• ZeroMQ patterns are implemented by pairs of sockets with matching types

The core ZeroMQ patterns are:

1. Request-reply:

• which connects a set of client to a set of services,
• This is a remote procedure call and task distribution pattern.

2. Pub-sub:

   • which connects a set of publishers to a set of subscribers.
   • This is a data distribution pattern

3. Pipleline:

• which connects nodes in a fan-out/fan-in pattern that can have multiple steps and loops.
• This is a parallel task distribution and collection pattern.

4. Exclusive pair, which connects two sockets exclusively,

• this is a pattern for connecting two threads in a processm not to be confused with 'normal' pairs of sockets.

---

We' ll be focusing on the request-reply and pub-sub.

---

Request-reply pattern

• The request-reply pattern is intended for service-oriented architectures, we have two types：

1. Synchronous -- REQ and REP socket types
2. Asynchronous -- DEALER and ROUTER socket types

In Jupyter, we use the asynchronous type with DEALER and ROUTER socket types

DEALER socket:

• DEALER socket type sends and receives messages to a set of anonymous peers using round-robin algorithms

  • round-robin algorithm: a First Come First Serve CPU Scheduling algorithm
  • where each process in ready state get a fixed time of the CPU turn by turn

• Messages received by a DEALER are fair-queued from all connected peers

• When a DEALER socket reached the high water mark(top-limits) for all peers, it enters mute state

  • any send operation on the socket will block until mute state ends
  • message are not discarded

---

ROUTER socket:

• The ROUTER socket type talks to a set of peers, using explicit addressing

• so that each outgoing message is sent to a specific peer connection.
  -ROUTER is often used as the basis for servers that talk to DEALER clients

• Receiving messages

• When receiving messages, a ROUTER socket will add in the front a message part that contains the routing id
  of which peer the message coming from, before passing the message to the application.

• The received messages are fair-queued from all connected peers.

• Sending messages

• When sending messages, a ROUTER will remove the first part of the message(that we add ourself when we receive the message), and use the containing information to determine the routing id of the peer the message shall be routed to.

• If the peer does not exist anymore, or has never existed, the messages shall be silently discarded.

• Mute State

• When ROUTER socket enters mute state due to having reached high water mark(top-limits) for all peers

• Any messages sent to the sockets will be dropped until the mute state ends

• If the peer we are routing to reaches high water mark, the messages will also be dropped

---

Publish-subscribe pattern

• publish-subscribe pattern is used for one-to-many distribution of data
• which is from a single publisher to multiple subscriber in a fan out fashion

-Fan in, Fan out

• In software engineering, fan in means how many upper level modules imports to current module
• fan out means, current modules is used by how many lower level modules:

---

ZeroMQ supports Pub/Sub by way of four sockets types:

• PUB
• XPUB
• SUB
• XSUB

---

• Topics

• To convey topic information, ZeroMQ uses messages in multipart

• Topics are expressed as an array of bytes though we may use a string with suitable text encoding.

-The following is an example of how publisher publish a "status" message to subscribers of "status" topic

//  Send a message on the 'status' topic
zmq_send (pub, "status", 6, ZMQ_SNDMORE);
zmq_send (pub, "All is well", 11, 0);

• Prior to the message payload: "All is well",
• a publisher must include the topic, in our case, "status" in the message's first frame

- For Subscribers:

• Subscribers can filter the subscriptions by setting interested topic

//  Subscribe to the 'status' topic
zmq_setsockopt (sub, ZMQ_SUBSCRIBE, "status", strlen ("status"));

• In this example, subscriber is interested in topic "status"
• A subscriber socket can have multiple subscription filters
• And a message's topic is compared against subscribers' subscriptions using a prefix check.
• That is, a subscriber who subscribe to topic would receive messages with topics:
  - topic, topic/subtopic, topical
• but it will not receive messages with topics: topi,TOPIC(because it's a byte-wise comparison so case-sensitive)
• Because of prefix matching, by subscribing with an empty topic string, we can receive all published messages

---

PUB socket

• A PUB socket is used by a publisher to distribute data and is not able to receive any messages

• Messages sent are distributed in a fan out fashion to all connected peers

• Mute State

• When a PUB socket reaches the high water mark for a subscriber and enters the mute state.

• then any messages shall sent to the subscriber shall be dropped until the mute state ends

---

SUB socket

• A SUB socket is used by a subscriber to subscribe to data distributed by a publisher.
• Initially, a SUB sockets is not subscribed to ant messages
• The send function is not implemented for SUB socket so it can only receive messages

[xp1632]

The rest socket types and patterns in ZeroMQ are not used in Jupyter, thus we'll omit the rest parts