WIP infinite crawl, flat files to DB

[leonstafford]

per #85

[leonstafford]

@thegulshankumar this may be of interest to check:

• all URLs detected during crawl
• code output looks OK

Note: it's only tested with ZIP deployment at the moment, others may not work (won't show progress at least)

static-html-output-plugin-wip-zip-only.zip

Sitting down too long, will take a break and start on the other deployers

[thegulshankumar]

Tried, unable to save options or click reset settings button.
So unable to set max speed for crawling and deployment.

[leonstafford]

@thegulshankumar thanks for checking - also shows me some UI issues when smaller screen size!

Are you getting some browser console errors or server errors with that?

[thegulshankumar]

There was little timeout issue but I could fix by adding

fastcgi_read_timeout 10800;