Make Database initialization faster

[jatrost]

Current the downloading of snort rules and initialization of the database take too long. We should be able to improve this.

[gregcmartin]

Agreed. Maybe threaded snort rule downloads from the multiple files instead single thread download of the big snort rule file from ET?

[gregcmartin]

Another way would be to only download snort rules when user initiates that from a button in the UI. This would make install go much faster.

[ no rules loaded currently, would you like to pull a fresh copy? ]

[jatrost]

Switching to compressed version would be faster for the download http://rules.emergingthreats.net/open/snort-2.9.0/emerging.rules.tar.gz. We could also host a preprocessed mhn.db or rules file as well. Lots of options. Maybe the lazy load makes sense though.

[gregcmartin]

did not know we were not using the compressed, lets start there.

On Wed, Jul 9, 2014 at 6:10 PM, Jason Trost notifications@github.com
wrote:

Switching to compressed version would be faster for the download
http://rules.emergingthreats.net/open/snort-2.9.0/emerging.rules.tar.gz.
We could also host a preprocessed mhn.db or rules file as well. Lots of
options. Maybe the lazy load makes sense though.

—
Reply to this email directly or view it on GitHub
https://github.com/threatstream/mhn/issues/20#issuecomment-48554761.

[serpulga]

I don't think it's the download that takes that long, it is parsing the file and and storing each rule into the database. We could add a command line option: Do you want to setup Snort Rules now?.

[jatrost]

The problem with this is we need a way to eventually download the rules if they intend to deploy a snort box (and the rules need to download before deploying one).

I think for many users who tested this download speed was their issue. When I asked them to attempt to manually download the file using wget it would timeout or hang. When running over vagrant with my fast Internet connection my DB initialization only takes a few minutes (3-5 min). We had users report 12+ hours for this step.

[d1str0]

Haven't noticed a prohibitively long database init. Issue seems to be resolved.