Visual Analytics : Create admin dashboard to crawldb to visualize the stats in realtime

[thammegowda]

Get stats from solr crawldb by querying it and visualize the results using charts like D3js (or similar)

[jorgelbg]

Perhaps there is no need of building something from scratch, since the stats are stored on Solr, we could use something like https://github.com/lucidworks/banana (port of Kibana) so adding an exported version of the panel could be enough, plus every body can customize it.

[thammegowda]

👍 @jorgelbg Thanks.
+1 for Banana.

[karanjeets]

@jorgelbg: Thanks for the suggestion. Banana is definitely a good visual aid.

However, apart from visuals, we are also looking forward to some interaction with the Crawler through a user interface. This can provide many benefits, for eg: guiding the crawl.

[manishdwibedy]

Hi, I would love to pitch in. If anyone can help me what exactly needs to be done.. Thanks.

[karanjeets]

Hi @manishdwibedy
Thanks for joining in. There are two user interfaces we are targeting here:

• Banana Dashboard connected to CrawlDB (Solr).
• An interactive user interface (preferably in angularJS) which can start, monitor and visualize the crawl.

I would recommend you starting the banana dashboard and add graphs which can help visualize the crawl data. Start by running Sparkler on your local system and analyzing the Solr schema.

[karanjeets]

@manishdwibedy
Thanks for your time today. Since you have Sparkler working on your local machine now, I am really looking forward to the first draft of Banana dashboard.

Once again, thanks for your interest in Sparkler!

[manishdwibedy]

@karanjeets Check the PR #38 for the banana dashboard.

[karanjeets]

Thanks @manishdwibedy!! 👍

I will review and merge. For the next steps, please create sub-tasks (as GitHub issues) for Banana dashboard, so that others can also contribute in this. :)

[thammegowda]

Resolved by #39