Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Traffic stats collection/analysis system
Fetching latest commit…
Cannot retrieve the latest commit at this time.
|Type||Name||Latest commit message||Commit time|
|Failed to load latest commit information.|
Traffic analysis system (in search of a better name) This is a small project I did to track my traffic based on the traffic of the web server - bytes sent, packets sent, tcp retransmits. The code is very simple and should be easy to understand, but here are some docs, anyway. The DB structure has a few logical components: - routers - for each router that handles the traffic from the web servers there's an entry in the "router" table, with the network it serves. ! see the note about bgpupd.sh below - traffic type - there's also an entry for each type of traffic in the "traffic" table. This can be set in the originator of the udp packets, probably on a per-vhost or anything basis (in another incarnation of the code ther was a check if the file was an video file and that was another type of traffic). - current routing table - in the "pfxtmp" table, for each router there's its full routing table (prefix and ASpath). - current traffic - in the "iptraf" table there's an entry for each prefix/router/traffic type combination, that gets updated by one of the cron scripts. - historical data - in the "iptraf_arch" table there's a dump of the data from the "iptraf" table each hour, with the rows with zero traffic removed. - country - there's a table "cntry" with all the prefixes for every country that gets updated by another script, with data from http://ip.ludost.net/ (which is a geolocation db based on the RIR databases). - there are two aggregate tables, "iptraf_cn" (per-country data) and "iptraf_dt" (per-asn data). NOTE: aspath field in the DB is only 255 chars, should be fixed. The workflow looks like this: - cron/countryupd.sh updates the table with the countries - cron/bgpupd.sh updates the prefixes for the router. It needs a quagga installation, and usually one quagga per router (most of the time this means another machine). I didn't have the time to try making something with views and having only one quagga to take the data from all routers. It uses a very (badly) stripped down version of the bgpdump tool from RIPE. - cron/processdwlog.sh moves the current log, sends HUP to the ipdwl daemon (so it reopens the log), processes it into queries and sends them in the db. It uses the readlog php script, which aggregates the data per ip/router/traffic triplet in separate queries. - on the other side, the module in the web server (nginx or something else) for each request takes the information from the tcp_info struct (to get the retransmits), creates an UDP packet, signs it with SHA1 and sends it to the ipdwl server. - the ipdwl server daemon receives the packets and writes them to a log file. (brief) Installation instructions: Install postgres, dev Install ip4r http://pgfoundry.org/projects/ip4r/ create a database (ipr) as postgres, load the ip4r type psql -U postgres < ip4r.sql create a database, load the ipr.schema psql ipr < ipr.schema install quagga give someone access to the quaggavty and quagga groups to run bgpupd from configure bgp neighbor compile and install bgpdump in INSTDIR. In all the sources grep for and replace: SOMESECRET, LOGDIR and INSTDIR. replace IP.IP.IP.IP with the ip of the server that will listen