This package helps you log zillions of events {cheaply,efficiently,reliably}.
An EventBucket is a set of events.
Key JSONValue'*, ** Type uint16 or unicode Value JSONObject (e.g. via JSON, protobuf (needing a .proto file to view/analyze)) Client Time ms since 1970 client-specified timestamp Server Time ms since 1970 when the event was received by the logging server
* no non-integer numbers, no nested structures, no \u0000
** Keys must be distinct so the server can be idempotent.
{LoggingServer, EventServer, EventClient} = require 'logging-system'
POST /api/post-events?bucket=TOKEN Content-Type: "application/json" or "application/eventbuf-v2" Request body: See "Formats" section Response status: 2xx or 5xx TODO: 2xx only after all data has been fully persisted (TEMP: 2xx immediately)
GET /events.json?bucket=TOKEN {"events": [{...see JSON Event Format...}...]} GET /reset Delete all events.
- Logs all data of all incoming HTTP requests in batches
- HTTP requests are not parsed. Not even a little. // TEMP: requiring UTF-8 for now
- Batches are POSTed to S3
- One batch every 10 sec implies 2.6 USD/month for POST requests
server = new LoggingServer { s3: { AWSAccessKeyId: "..." policy64: "..." signature64: "..." bucket: "..." customUrl: "https://my-s3-clone:12345" # OPTIONAL } batchSeconds: 10 # OPTIONAL } server.listen PORT, () -> console.log "Listening on #{PORT}..." # AND/OR: ...(req, res) -> if ... server.handleRequest req, res
server = new EventServer() server.listen PORT, () -> console.log "Listening on #{PORT}..." server.reset() # delete all events
[EVENTS, bucket, k_string] -> eventJson // later protobuf [INDEXED_EVENTS, bucket, indexId, ...index...] -> eventJson // later protobuf
{ "key": ..., "type": ..., "value": ..., "clientTime": 1317062859638 }
{ "events": [...] }
TODO
Just concatenate 'em.
"v1/%Y-%m-%d/%H-%M-%S-%L-Z-" + serverToken + "-" + randomToken(8, BASE58_ALPHABET) + "-" + batchNumber + "-v2" ...where, when the server starts, serverToken := randomToken(8, BASE58_ALPHABET) If the body is gzipped, append ".gz".
Concatenated HTTP events, each of which is:
msgpack(length_of_the_following) # because msgpack libraries don't support streaming msgpack(http_event)
REQ_DATA_EVENT = 1 REQ_END_EVENT = 2 { 1: REQ_DATA_EVENT 2: reqId 3: ms_timestamp_of_first_fragment 8: fragment_id 4: data } { 1: REQ_END_EVENT 2: reqId 3: ms_timestamp_of_first_fragment 8: fragment_id 5: remote IP UTF-8 6: content-type UTF-8 7: path UTF-8 }
{PUT,LIST}s 100,000 for 1 USD => (every 10 sec => 2.6 USD/month) {GET}s 1,000,000 for 1 USD storage >= 7.14 GB-month for 1 USD