Skip to content
/ tega Public

Config management, pubsub and rpc for IoT networking

Notifications You must be signed in to change notification settings

araobp/tega

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

tega db

travis

Project start: 2014/8/8

tega_idb

Distributed document-oriented database for small PCs (incl. Raspberry Pi), written in Python:

  • Tornado-based
  • CRUD/RPC supported
  • Transaction supported
  • JSON-oriented (not YANG)
  • Schema-less
  • Extensible with plugins
  • NAT traversal (HTTP/WebSocket)
  • Python and Go driver
  • Easy to use
$ tega-cli
tega CLI (q: quit, h:help)
[tega: 0] put a.b.c
1

[tega: 1] put a.x
2

[tega: 2] get a
b: {c: 1}
x: 2

Design policy

  • Simplicity rather than high-performance and rigid schema
  • NOSQL for non big data (small data)
  • Embeddable data base
  • Focus on config and operational data management
  • Support various integration patterns: data change notification, messaging and rpc
  • Easy-to-use APIs
  • Concurrenty support with coroutine
  • NAT/HTTP-Proxy traversal: REST and WebSocket

Why graph(tree) structure is suitable for network config in some cases

  • In general, you had better use SQL (or SQL-like db) for storing config of a whole network.
  • You define Nodes table, Links table, Subnets table... a lot of tables.
  • A subset (a set of rows from those tables) of the network config should be stored in a local db at each router. In general, such data has a tree structure (e.g., SNMP MIB).
  • There are two choices: SQL or graph(tree) database. It depends... In my case, I have chosen grap(tree) database, since I want version controll for the config (like "git").

Project goal

goal

Use cases

  • NLAN
  • BBR remote config
  • OpenWrt remote config
  • Instant VPN (L2 or L3)
  • IOT

Deployment

Installation

You need to have python3.5 installed on your Debian/Ubuntu Linux.

$ pip install tornado
$ pip install httplib2
$ pip install pyyaml
$ pip install readline
$ git clone http://github.com/araobp/tega
$ cd tega
$ python setup.py install

Start tega server with sample options, like this:

$ cd tega/scripts
$ ./global

And start tega CLI to use CLI commands:

$ tega-cli

Current architecture

                           [cli.py]
                               |
                          [driver.py]
                               |  |
         +---------------------+  +-----------------------+
         |                                                |
     Tornado                                          Tornado 
     [server.py] ------------ REST/WebSocket ---------[server.py]
         :                                                : 
     [idb.py]...[tree.py]                             [idb.py]...[tree.py]
         :                                                :
     --------                                         --------
    /tega.db/                                        /tega.db/
   ---------                                        ---------
   commit-log                                       commit-log

Documentation

Specification

Usage

CRUD operations

operations to the tree

Rollback

rollback

Consistency

CP

Collision detection

Collision

Garbage collection

garbage collection

RPC routing

rpc routing

Integration patterns

integration patterns

For example,

  • My neutron-lan project has taken the pattern [D]. The implementation of the agent is not simple.
  • My nlan project has been taking the pattern [A], since Docker is for an immutable infrastructure. The implementation of the agent is quite simple.
  • In general, commercial-grade network controllers take the pattern [E]. It requires a development team with a lot of software engineers.
  • Signalling protocol developers think differently... My another drawing

You had better study how databases (incl. NOSQL ones) realize data synchronization in a cluster.

Note

Implementation

  • Tega db is still in alpha release and part of the documentation here has not been implemented yet.

Performance

it is a schema-less database and the performance (especially write-performance) of this database is not good:

  • dynamic typing
  • nested hash map (hash collisions, rehashing...)
  • it makes use of for-loop a lot (Python's for-loop performance is not good)
  • it saves every commit log onto a storage

References

About

Config management, pubsub and rpc for IoT networking

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published