Import docs from OmniTI Labs

CONCEPTS.md

@@ -0,0 +1,101 @@
+# JLog Concepts
+
+A JLog is durable message queue; records (messages) are written to a JLog and
+will remain logged until subscribers have read past that message and issued a
+checkpoint.
+
+JLogs are implemented as a directory on disk that contains a handful of control
+files and a number of segment files. The segment files contain the messages
+that were written to the JLog; each segment file can contain multiple messages
+and grow to be 4MB in size. If a new message written to the JLog would cause
+this limit to be exceeded, a new segment file is created. Segment files are
+deleted automatically when all subscribers have consumed the contents of that
+segment file and issued a checkpoint.
+
+## Subscribers
+
+In order to manage data in the JLog, the JLog needs to track who is
+subscribing. Each subscriber has a name; the name must be unique for each
+consuming process, otherwise the behavior of JLog is undefined as two different
+processes will conflict over their state.
+
+It is recommended that the writer of a jlog have knowledge of the subscriber
+list before it writes data to the jlog, so that segments are not pruned away
+before a given subscriber starts to read--such a subscriber would effectively
+be "late" to the party and miss out on the data.
+
+## Using the C API
+
+Here's a quick overview of how to use the C API for writing and reading:
+
+### Writer
+
+Here, the writer is appending data to the jlog, which we're storing at
+/var/log/jlogexample. We have two subscribers, named "one" and "two":
+
+    jlog_ctx *ctx;
+    const char *path = "/var/log/jlogexample";
+    int rv;
+
+    // First, ensure that the jlog is created
+    ctx = jlog_new(path);
+    if (jlog_ctx_init(ctx) != 0) {
+      if(jlog_ctx_err(ctx) != JLOG_ERR_CREATE_EXISTS) {
+        fprintf(stderr, "jlog_ctx_init failed: %d %s\n", jlog_ctx_err(ctx), jlog_ctx_err_string(ctx));
+        exit(1);
+      }
+      // Make sure it knows about our subscriber(s)
+      jlog_ctx_add_subscriber(ctx, "one", JLOG_BEGIN);
+      jlog_ctx_add_subscriber(ctx, "two", JLOG_BEGIN);
+    }
+
+    // Now re-open for writing
+    jlog_ctx_close(ctx);
+    ctx = jlog_new(path);
+    if (jlog_ctx_open_writer(ctx) != 0) {
+       fprintf(stderr, "jlog_ctx_open_writer failed: %d %s\n", jlog_ctx_err(ctx), jlog_ctx_err_string(ctx));
+       exit(0);
+    }
+
+    // Send in some data
+    rv = jlog_ctx_write(ctx, "hello\n", strlen("hello\n");
+    if (rv != 0) {
+      fprintf(stderr, "jlog_ctx_write_message failed: %d %s\n", jlog_ctx_err(ctx), jlog_ctx_err_string(ctx));
+    }
+    jlog_ctx_close(ctx);
+
+### Reader
+
+Using the reader for subscriber "one" looks like this:
+
+    jlog_ctx *ctx;
+    const char *path = "/var/log/jlogexample";
+    int rv;
+    jlog_id begin, end;
+    int count;
+
+    ctx = jlog_new(path);
+    if (jlog_ctx_open_reader(ctx, "one") != 0) {
+      fprintf(stderr, "jlog_ctx_open_reader failed: %d %s\n", jlog_ctx_err(ctx), jlog_ctx_err_string(ctx));
+      exit(1);
+    }
+
+    count = jlog_ctx_read_interval(ctx, &begin, &end);
+    if (count > 0) {
+      int i;
+      jlog_message m;
+
+      for (i = 0; i < count; i++, JLOG_ID_ADVANCE(&begin)) {
+         end = begin;
+
+         if (jlog_ctx_read_message(ctx, &begin, &m) == 0) {
+            printf("Got: %.*s\n", m.mess_len, (char*)m.mess);
+         } else {
+           fprintf(stderr, "jlog_ctx_read_message failed: %d %s\n", jlog_ctx_err(ctx), jlog_ctx_err_string(ctx));
+         }
+      }
+
+      // checkpoint (commit) our read:
+      jlog_ctx_read_checkpoint(ctx, &end);
+    }
+    jlog_ctx_close(ctx);

README.md

@@ -0,0 +1,58 @@
+# JLog
+
+JLog is short for "journaled log" and this package is really an API and implementation that is libjlog.
+What is libjlog? libjlog is a pure C, **very simple** durable message queue with multiple subscribers and publishers
+(both thread- and multi-process safe). The basic concept is that publishers can open a log and write messages to it
+while subscribers open the log and consume messages from it. "That sounds easy." libjlog abstracts away the need to
+perform log rotation or maintenance by publishing into fixed size log buffers and eliminating old log buffers when
+there are no more consumers pending.
+
+## Implementation Goal
+
+The goal of libjlog is to provide a dead-simple API, and a very small, uncomplicated implementation. 
+The goal of the JLog is to have others use this project for adding durability to some of their asynchronous 
+notification needs.
+
+## Sample Use Case
+
+We run an application on server A and would like to write logs to server B. If server B is down, we don't want to lose
+logs and we do want A to continue operating. So A writes to a JLog and B consumes from that JLog. Sounds simple, but JLog
+is a low-level API. The implementor must provide a service on A to which B can connect to consume the JLog messages over
+the network. If you want to have you cake and eat it too, become a baker and use this fine ingredient!
+
+Read through [Concepts](./CONCEPTS.md) for a bit more detail on core concepts and a brief overview of how to use the C API.
+
+## Installing
+
+FreeBSD port `databases/jlog`:
+
+        pkg install jlog
+
+MacOS via [Homebrew](http://brew.sh/):
+
+        brew install jlog
+
+If JLog is not packaged by your OS vendor, read on for build instructions.
+
+The LZ4 library is required if you want support for compressed logs.
+
+Java 6 or newer is required if you want to support using JLog from a Java application.
+
+To build from source, clone this repo and then:
+
+    autoconf
+    ./configure
+    make
+    make install
+
+## Team
+
+* Wez Furlong
+* Alec Peterson
+* George Schlossnagle
+* Theo Schlossnagle
+* Alexey Toptygin (current maintainer)
+
+## License
+
+JLog is released under a new BSD license. See our [license](./LICENSE) for details.