You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is the start of Scale's new messaging passing architecture to improve scalability.
All of this should be in a Django app, called "messaging"?
Base class CommandMessage
self.succeeded - used by manager to know whether to ack success for this message
self.new_messages - list of CommandMessage, used by manager to know if new messages should be sent
self.type - type of message to use in factory method to create correct subclass, will need to be sent/received as part of overall payload in the CommandMessageManager methods
to_json()
from_json(json_dict)
execute() - docstring should mention that this should be idempotent and able to be run out of order
Subclasses register in Django app methods and have no-arg init()
Factory method to generate CommandMessage objects from JSON payloads
Basic test CommandMessage (log "Hello Scale" or something)
Class CommandMessageManager
init() sets up and initializes actual backend (SQS, RabbitMQ, Kafka) from settings/env vars
send_message(cmd_msg) - uses cmd_msg.to_json() to create payload
singleton access as cmd_msg_mgr
process_messages() - generically handles "loops" of message processing, should handle multiple in a loop (so we can do 10 at a time with SQS batch), gets JSON payloads from backend and calls factory method and from_json() to set up cmd_msg objects, and processes them
Django management command to run cmd_msg_mgr.process_messages()
Classes that abstract/implement backends (SQS, RabbitMQ, Kafka, etc)
SQS backend should use long-polling and SQS batch (10 at a time)
Need to updates docs to discuss configuring Scale to use whatever messaging backend.
The text was updated successfully, but these errors were encountered:
So after some reading on the subject, I'm leaning toward RabbitMQ for the out-of-the-box solution. RabbitMQ appears to be the nearest match for what we are trying to accomplish and has excellent Python (and other cross-language support), as a bonus its usage is very similar to SQS. Kafka isn't a first choice as it is favors an event stream approach with no cleanup following message processing (message roll-off happens by queue size / TTL). It is up to the consumer to manage their position in the log instead of just popping off the front of the queue.
This is the start of Scale's new messaging passing architecture to improve scalability.
All of this should be in a Django app, called "messaging"?
Base class CommandMessage
Subclasses register in Django app methods and have no-arg init()
Factory method to generate CommandMessage objects from JSON payloads
Basic test CommandMessage (log "Hello Scale" or something)
Class CommandMessageManager
Django management command to run cmd_msg_mgr.process_messages()
Classes that abstract/implement backends (SQS, RabbitMQ, Kafka, etc)
SQS backend should use long-polling and SQS batch (10 at a time)
Need to updates docs to discuss configuring Scale to use whatever messaging backend.
The text was updated successfully, but these errors were encountered: