Pull request Compare This branch is even with cloudfoundry-attic:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bin
doc
include
src
test
upgrades
.gitignore
.gitmodules
.preferred_otp_version
Procfile
README.md
public.rebar.config
rebar
rebar.config

README.md

Table of Contents

  1. Build
  2. Test
  3. Develop
  4. Supervision Tree
  5. Processes

Build

$ ./rebar --config public.rebar.config get-deps compile

Test

Given an empty local redis (v2.6ish):

$ INSTANCE_NAME=`hostname` \
  LOGPLEX_CONFIG_REDIS_URL="redis://localhost:6379" \
  LOCAL_IP="127.0.0.1" \
  LOGPLEX_COOKIE=123 \
  rebar skip_deps=true ct

Runs the common test suite for logplex.

Develop

run

$ INSTANCE_NAME=`hostname` \
  LOGPLEX_CONFIG_REDIS_URL="redis://localhost:6379" \
  LOCAL_IP="127.0.0.1" \
  LOGPLEX_COOKIE=123 \
  LOGPLEX_AUTH_KEY=123 \
  erl -name logplex@`hostname` -pa ebin -env ERL_LIBS deps -s logplex_app -setcookie ${LOGPLEX_COOKIE}

create creds

1> logplex_cred:store(logplex_cred:grant('full_api', logplex_cred:grant('any_channel', logplex_cred:rename(<<"Local-Test">>, logplex_cred:new(<<"local">>, <<"password">>))))).
ok

hit healthcheck

$ curl http://local:password@localhost:8001/healthcheck
OK

create a channel

$ curl -d '{"tokens": ["app"]}' http://local:password@localhost:8001/channels
{"channel_id":1,"tokens":{"app":"t.feff49f1-4d55-4c9e-aee1-2d2b10e69b42"}}

post a log msg

$ curl -v \
-H "Content-Type: application/logplex-1" \
-H "Content-Length: 119" \
-d "115 <134>1 2012-12-10T03:00:48Z+00:00 erlang t.feff49f1-4d55-4c9e-aee1-2d2b10e69b42 console.1 - Logsplat test message 1" \
http://local:password@localhost:8601/logs

create a log session

$ curl -d '{"channel_id": "1"}' http://local:password@localhost:8001/v2/sessions
{"url":"/sessions/9d53bf70-7964-4429-a589-aaa4df86fead"}

fetch logs for session

$ curl http://local:password@localhost:8001/sessions/9d53bf70-7964-4429-a589-aaa4df86fead?srv=1
2012-12-10T03:00:48Z+00:00 app[console.1]: test message 1

Supervision Tree

logplex_app logplex_sup logplex_db
config_redis (redo)
logplex_drain_sup logplex_http_drain
logplex_tcpsyslog_drain
nsync
redgrid
logplex_realtime redo
logplex_stats
logplex_tail
logplex_redis_writer_sup (logplex_worker_sup) logplex_redis_writer
logplex_read_queue_sup (logplex_queue_sup) logplex_queue
logplex_reader_sup (logplex_worker_sup) logplex_reader
logplex_shard redo
logplex_api
logplex_syslog_sup tcp_proxy_sup tcp_proxy
logplex_logs_rest

Processes

logplex_db

Starts and supervises a number of ETS tables:

channels
tokens
drains
creds
sessions

config_redis

A redo redis client process connected to the logplex config redis.

logplex_drain_sup

An empty one_for_one supervisor. Supervises HTTP and TCP drain processes.

nsync

An nsync process connected to the logplex config redis. Callback module is nsync_callback.

Nsync is an Erlang redis replication client. It allows the logplex node to act as a redis slave and sync the logplex config redis data into memory.

redgrid

A redgrid process that registers the node in a central redis server to facilitate discovery by other nodes.

logplex_realtime

Establishes a connection to the logplex stats redis. Owns the logplex_realtime ETS table and flushes the contents to tempo via redis pub/sub every second.

Metrics published to tempo:

message_received
message_processed
drain_delivered
drain_dropped
git_branch
availability_zone

logplex_stats

Owns the logplex_stats ETS table. Prints channel, drain and system stats every 60 seconds.

logplex_tail

Maintains the logplex_tail ETS table that is used to register tail sessions.

logplex_redis_writer_sup

Starts a logplex_worker_sup process, registered as logplex_redis_writer_sup, that supervises logplex_redis_writer processes.

logplex_read_queue_sup

Starts a logplex_queue_sup process, registered as logplex_read_queue_sup, that supervises logplex_queue processes.

logplex_reader_sup

Appears to start a logplex_worker_sup that supervises logplex_reader processes. That model doesn't seem to exist?!?

logplex_shard

Owns the logplex_shard_info ETS table. Starts a separate read and write redo client for each redis shard found in the logplex_shard_urls var.

logplex_api

Blocks waiting for nsync to finish replicating data into memory before starting a mochiweb acceptor that handles API requests for managing channels/tokens/drains/sessions.

logplex_syslog_sup

Supervises a tcp_proxy_sup process that supervises a tcp_proxy process that accepts syslog messages over TCP.

logplex_logs_rest

Starts a cowboy_tcp_transport process and serves as the callback for processing HTTP log input.