Adding support for concurrent crawling

[shivamanipatil]

Hey,

I think it would a great feature to have if we supported taking multiple URLs at the same time and then spawning multiple crawlers using goroutines. I think there won't be any change to our crawler package due to this.

Can be divided into :

• Support for taking multiple inputs
• Concurrency support to spawn multiple crawlers

@zpeters What do you think about this? And can you assign me this issue?

[zpeters]

This sounds like a good plan. Lets see what it looks like

[zpeters]

please make sure and update README.md if the command line options change.

Thanks again, have fun