statistics system with realtime and trend view.
JavaScript Python Other
Switch branches/tags
Nothing to show
Pull request Compare This branch is 24 commits ahead of pedia:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
agent
misc
server
.gitignore
README.md
setup.py

README.md

Usage

pip install flask tornado simplejson
python server/main.py

Core / Engine

We find that in the real world every data-source can be represented as a basic structure, a TREE of all types of data, and the tree is efficiency for communication between sub-systems. Based on this idea, we start to design a new realtime monitor system, which contains a HUGE tree in memroy, for realtime usage. History trend data is store in databases.

Here is an example of such structure:

[
{'cpu': {'10.3.1.12': {'idle': 32,
                       'sytem': 29,
                       'user': 900,
                       'marker': 'sth. wrong happened'}},
  'defaultaction': 'average'},
{'br':  {'ie':  {'6': 3},
                {'7': 8, {'venders':{'sogou':2}}},
                {'7' : {'venders':{'sogou':1}}}},
      {'webkit': {'14.0': 40},
                {'13.0': 20}},
  'defaultaction': 'add'}
]

Query by key name as form like:

  • br,webkit

  • br,ie,,venders,sogou

More detail in server/tree.py

server

directory layout

server/
  main.py
  tree.py -> async store in trend db
  treedb.py
  pushviahttp.py
  pushviasocket.py

server/pipes/*.py
server/web
  web/pages.py
  web/trend.py
server/web/static/*.js *.css *.html
server/web/templates/*.html
server/graph
server/graph/styles.py
  cpu|net|url|default
  input: tree
    bar|line|x-axis|marker range,picture|theme...|format
  output: png|pdf


# relay server
--relay
? agent/conf/relay.conf

url map

request argument name style like: e or a,b or a,,c

/api
  /api/q/a,b.json current tree
  

/push
  form field support json/accesslog
  json -> merge into core tree
  accesslog -> pipes -> tree -> merge into core tree

/pull/?key-br
/pull/?key-br,ie
/pull/?key-br,ie,7
  comet, repsonse in json list
    [{br,ie,7 : 2328},
    {br,ie,sogou,3 : 323}]
/get/
  response immediatly in json list
/realtime/
  page, init with /get/
  then comet by /pull/
/realtime/view/
  view by name, per user's specify

/trend/view/
  [default in one year]
  left tree | right (multi) js graph (zoomable highcharts/examples/dynamic-master-detail.htm)
/trend/view/?
  one js graph
  add compare | zoomable | setting style ...
/trend/?time
/setting/view
  add | remove view


/diff/keya/keyb
/graph/[.jpg|.png|.svg]

pipes

web access log
  browser type  -> tree
  os type       -> tree
  upstream time  -> tree
  request time
  response bytes
  url count
  error count
  ? geo

write your own pipe.py
  def init()
  def process(arr)
  def cleanup()

running mode:
init:
  for m in pipes/*.py:
    pipes.push(m)
    m.init()

run:
  for line in log.read():
    arr - weblog.parse(line)
    for m in pipes:
      m.process(arr)	

server push

  • udp

  • tcp

  • http post in form

Agent

directory

agent/
agent/modules/*.py *.so  
             gmod[32|64]/support ganglia module
agent/conf/*.conf *.on *.off
agent/agent.conf

agent/setup.py

modules

web access log(raw)
url(time to first byte, status code, bytes, ...)
  nginx stats

write your own module(same as [ganglia python module](http://sourceforge.net/apps/trac/ganglia/wiki/ganglia_gmond_python_modules#WritingcustomPythonmodules)):
  def metric_init({})
  def metric_cleanup()

reminder

A: use ganglia-plugin. build a package named 'ganglia-plugin' ? use ganglia-plugin only

  1. LD_PRELOAD=libganglia.so python modload.so modcpu.so

  2. rebuild modxxx.so

  3. ganglia install path/*.so

A: PUT/POST, socket

x post file in form ? HTTP POST raw body ? HTTP PUT socket tornado socket libevent socket + flask x web socket

A: 1 web log => graph