I have used python in-memory dictionary data structure to store messages for each topic. The dictionary contains topic-messages pairs. Flask library is used to implement REST endpoints. The below REST endpoints are implemented:
The topic endpoint is used to create a topic and get a list of topics.
Used to create a new topic.
Flask endpoint
@app.route('/topic', methods=['PUT'])
Body:
{'topic' : str}
Returns:
{'success' : bool}
Returns True if the topic was created, False if the topic already exists or was not created for another reason.
Used to get a list of topics
Flask endpoint
@app.route('/topic', methods=['GET'])
Returns:
{'success' : bool, 'topics' : [str]}
If there are no topics it returns an empty list.
The message endpoint is to add a message to a topic and get a message from a topic.
Flask endpoint
@app.route('/message', methods=['PUT'])
Body:
{'topic' : str, 'message' : str}
Returns:
{'success' : bool}
returns failure if topic does not exists
Used to pop a message from the topic. Notice that the topic name is included in the URL.
Flask endpoint
@app.route('/message/<topic>', methods=['GET'])
Returns:
{'success' : bool, 'message' : str}
It returns False if:
- the topic does not exist
- there are no messages in the topic that haven’t been already consumed
The status endpoint is used for testing the leader election algorithm.
Flask endpoint
@app.route('/status', methods=['GET'])
Returns:
{'role' : str, 'term' : int}
Role is a string with one of three options:
- Leader
- Candidate
- Follower
Term return the term that this node is currently in as INT (to the best of its knowledge)
Each node starts as a follower. Every node has a timer with timeout value picked randomly from uniform distribution (500, 1000) milliseconds. When node timers time out and it does not know who is the leader, it starts election process.
Flask endpoint
@app.route('/election/vote', methods=['POST'])
The follower after starting election process moves to candidate role and increases current election term value. It calls the above REST endpoint for each node in the list. It votes for itself and wait for votes from others. If it receives votes from majoirty of nodes it moves to leader state. While during election process, if it finds out the current leader or any node with term greter than its term value, it moves back to follower state. A leader must have most updated logs.
Flask endpoint
@app.route('/logs/append', methods=['POST'])
After a node becomes leader, it syncs its own logs with other nodes by calling the above endpoint. Only leader can communicate with the clients. When the leader receives a request, it appends an entry to its own logs. Then it tries to send append entry messages to other nodes to sync logs. Once more than half of the nodes append the entry to their logs, leader commits the entry in its logs. After leader commits, other nodes start committing the entry. Committed entries are applied to state machine asynchronously. After a entry is applied to leader's state machine, leader replies back to the client.
This implementation is almost similar to what has been proposed on raft paper.
References: