Msfcrawler improvement (and bug fixes)

[flibustier]

• Fixing StoreDB function on msfcrawler

• Add sub-module for catching links in html comments

• Fixing errors in 'data/msfcrawler/forms.rb' sub-module :

[*] ERROR
[*] undefined method `empty?' for nil:NilClass: ["(eval):35:in `block in parse'", 

and

encoding error : output conversion failed due to conv error, bytes 0xA4 0x31 0x2E 0x30
I/O error : encoder error

• Remove old banners, "rubygems" requirements, executable marks

Verification

List the steps needed to make sure this thing works

• Start msfconsole
• use auxiliary/crawler/msfcrawler
• set StoreDB true
• set the other options required
• run
• Verify It should work, you can verify with wmap_sites -l for example

[wvu]

Ye olde msfcrawler. Why are these files still in the data directory? Does anyone still use this?

[wvu]

Can we just rm the whole thing? I know you put a lot of work into this, but I'm skeptical anyone still uses this enough for us to keep it.

[h00die]

Googling msfcrawler the results are all rapid7's, git, and a blog post over a year old... Don't think unleashed has this in it. I'm not sure many would notice it being gone...

[wvu]

I'm sure someone uses it. I also hate to rm someone's hard work. We'll decide what to do at a later date...

[wvu]

Release Notes

Fixes and improvements were made to msfcrawler.