Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Fetching contributors…

Cannot retrieve contributors at this time

59 lines (49 sloc) 2.74 KB
What is the directory structure?
+- bin/
| +- services/
| | +- Services, it is possible to start them by hand but the initscripts do normally the job
| +- Some scripts/wrappers to generate some particular queries
+- doc/
| +- UML Diagrams, dump of the wiki, documentation on third party softwares, blog articles...
+- etc/
| +- init.d/
| | +- Initscripts
| +- bgp-ranking.conf config file used by the initscripts and define all the paths of the application and some constraints
| +- redis.conf config file of redis
+- lib/
| +- db_init/
| | +- Scripts to initialize the databases (whois client and ranking)
| +- db_models/
| | +- Databases models (whois client and ranking)
| +- helpers/
| | +- some helpers (for the initscripts and the manipulation of the ips) used very often in the project
| +- modules/
| | +- all the modules to parse the differents raw files
| +- whois_client/
| | +- whois fetcher and connector for the whois client
| +- whois_parser/
| | +- parsers to extract particular informations of the whois response
| +- : push the RIS whois responses in the database
| +- : push the whois responses in the database
| +- : pop the new entries from redis and push it in MySQL
+- scripts/ : bunch of scripts used to start the system
+- var/
| +- raw_data/
| | +- source_name/ : name of the source of the file
| | | +- old/ : directory to put the old files (already parsed)
| | | +- temp/ : directory where the lists are downloaded
| | | +- file(s) to parse
| +- run/
| | +- pid files of the unning processes
| +- log/
| | +- log files
Why do we need many redis servers ? Is it not overkill ? Redis can handle a lot of query/sec.
tl;dr: if you do not use the daily dataset of dshield, one redis server is enough. Set all the redis ports to 6379 in the config file.
And comment the line with DshieldDaily!
Detailled answer:
It depends. If you use the whole system with all the sources, not. At all. Or you will see redis die.
The system is massively multiprocessed, with massively I meen around 40/50 processes when it is busy. Each of this processes interract more or less with redis. Some of them write stuff but the most "redis-intensif" work is reading information from the databases.
That is why there is two slave databases which only handle this part.
The master server handle all the write processes.
The second master redis server contains the temporary (RIS) Whois entries, it is quite intensive but a single server doing only this is enough. It is gread to have this second server because many information are added/removed from the server and the other master server was quite busy to replicate the entries on the slaves.
Jump to Line
Something went wrong with that request. Please try again.