forked from mfp/ocamlmq
-
Notifications
You must be signed in to change notification settings - Fork 0
Small footprint, easy to use STOMP message broker designed for task queues and IPC
License
orbitz/ocamlmq
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
ocamlmq is a STOMP message broker with features that make it especially suitable for implementing task queues and communication between subsystems: * persistent queues, scaling over arbitrarily large numbers of queues with constant memory usage (i.e. supports millions of queues) * strong durability guarantees: messages are guaranteed to have been saved to disk by the time the sender gets a message receipt * message priorities * per-subscription prefetch limit for queue messages * error handling and ACK timeout: if a subscriber doesn't ACK a message after a (message-specific) timeout, the message will be sent to another subscriber. Messages are also resent automatically if a subscriber dies and its connection expires. * topic subscriptions (messages broadcast to all subscribers) with prefix matching * support for high numbers of subscriptions: millions of subscriptions pose no problem * simple extensions within the STOMP protocol to report the number of messages in a queue and the number of subscribers to a queue or topic ocamlmq is written in OCaml, in less than 1200 lines of code. It is easy to extend and fairly efficient. The server is abstracted over a storage backend; currently only PostgreSQL's is implemented (< 150 lines of code). Scalability =========== ocamlmq has been designed to support millions of queues and topic subscriptions without undue memory usage. This table summarizes the time complexity of some STOMP operations: SEND to queue O(log subscribers) SEND to topic O(subscribers + log (total subs)) SUBSCRIBE to queue O(log (total subs)) SUBSCRIBE to topic O(log subscribers) ACK O(1) ocamlmq needs typically around 150 bytes per subscription, so 1 million subscriptions will not take much more than 150 MB of memory. No extra memory is needed for queues, so you can use lots of them with no concerns for memory usage. Limitations =========== ocamlmq works well in the intended use case (persistent queues and transient topic destinations, with possibly many queues and subscriptions), but it has some limitations which preclude its use in other domains: * ocamlmq is not designed to scale beyond several hundred / a few thousand simultaneous connections (it will work, but performance will be affected) * there is no flow control for topic messages (in the intended use case, topic messages are assumed to be relatively small and processed fast) * messages are limited to 16 MB on 32-bit platforms * the PostgreSQL storage backend can only persist a few thousand messages per second (note that ocamlmq allows >50K/s persistent message bursts in async mode) * ocamlmq does not support very high message rates (ocamlmq delivers only ~20000 messages/second on a 3GHz AMD64 box) If you need complex routing rules, scalability to many thousand simultaneous connections or other _enterprise_ messaging features, you'll be better served by AMPQ or JMS brokers. ActiveMQ, in particular, is very configurable, so it'll most likely fit the bill if memory consumption and scaling to many subscriptions are not a concern. Building ======== You'll need a working OCaml environment plus the following libraries: * Lwt * extlib * PGOCaml Additionally, ocamlmq requires PostgreSQL both at compile- and run-time. If you have omake, just do $ omake Otherwise, run $ sh build.sh If the compilation fails with the following error File "./mq_schema.ml", line 3, characters 21-327 (end at line 11, character 3): Camlp4: Uncaught exception: Unix.Unix_error (20 | CstTag21, "connect", "") you will have to provide some information using env. variables so that PGOCaml can connect to the PostgreSQL server (PGOCaml does this to check statically that all the SQL statements are valid), e.g.: PGHOST=localhost PGUSER=myself PGPASSWORD=myself omake --verbose or PGHOST=localhost PGUSER=myself PGPASSWORD=myself sh build.sh This is the full list of options recognized by PGOcaml: PGHOST PGPORT PGUSER PGPASSWORD PGDATABASE UNIX_DOMAIN_SOCKET_DIR A temporary table will be created in the database while ocamlmq is compiled; no permanent changes will be made. Running ======= ocamlmq's configuration is given via the command line: $ ./ocamlmq -help Usage: ocamlmq [options] -dbhost HOST Database server host. -dbport HOST Database server port. -dbdatabase DATABASE Database name. -dbsockdir DIR Database UNIX domain socket dir. -dbuser USER Database user. -dbpassword PASSWORD Database password. -dbmaxconns NUM Maximum size of DB connection pool. -port PORT Port to listen at (default: 61613). -login LOGIN Login expected in CONNECT. -passcode PASSCODE Passcode expected in CONNECT. -initdb Initialize the database (create required tables). -debug Write debug info to stderr. -help Display this list of options --help Display this list of options ocamlmq can create the required tables (currently, only one, named "ocamlmq_msgs") in the PostgreSQL table indicated with -dbdatabase (defaulting to one with the same name as the user) with the -initdb option, so you can try it with $ ./ocamlmq -dbdatabase mydatabase -initdb which will listen on port 61613 by default. STOMP protocol specifics ======================== ocamlmq uses the STOMP protocol as specified in http://stomp.codehaus.org/Protocol and uses a trailing newline (after the NULL byte) to delimit frames. SEND ---- The ACK timeout period, after which queue messages are sent to another subscriber, can be specified in the "ack-timeout" header (as a float in seconds), e.g. SEND destination:/queue/test ack-timeout:3.14 just testing ^@ SUBSCRIBE --------- Clients can subscribe to topics (/topic/xxx) which have broadcast, non-durable semantics, or to queues (/queue/xxx), which are persistent. It is also possible to subscribe to all the topics matching a prefix, by using /topic/someprefix* as the destination. The prefetch limit (max. number of unacknowledged messages allowed by the server) can be specified in the "prefetch" header. BEGIN / COMMIT -------------- They are not implemented, since they are ill-specified. Control messages ================ A client can send messages to special "control" destinations and obtain the response in the message receipt (iow., nothing is returned unless the "receipt" header is set): /control/count-msgs/queue/name-of-the-queue returns the number of messages in the "num-messages" header /control/count-subscribers/queue/name-of-the-queue returns the number of suscribers in the "num-subscribers" header /control/count-subscribers/topic/name-of-the-topic returns the number of suscribers in the "num-subscribers" header
About
Small footprint, easy to use STOMP message broker designed for task queues and IPC
Resources
License
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published
Languages
- OCaml 92.0%
- Shell 7.0%
- Ruby 1.0%