Skip to content
Simple dashboard for Kimurai web scraping framework
Branch: master
Clone or download
Latest commit 73af31b Dec 30, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin Init Sep 15, 2018
lib/kimurai
test Init Sep 15, 2018
.gitignore Init Sep 15, 2018
.travis.yml Init Sep 15, 2018
Gemfile Init Sep 15, 2018
LICENSE.txt Init Sep 15, 2018
README.md Update README Dec 30, 2018
Rakefile Init Sep 15, 2018
kimurai-dashboard.gemspec Set required version of Kimurai to >= 1.3.0 Dec 3, 2018

README.md

Kimurai::Dashboard

Simple Dashboard for Kimurai web scraping framework. Required version of Kimurai >= 1.3.0.

Installation

Add this line to your Kimurai project's Gemfile:

# add this line after `gem 'kimurai'`
gem 'kimurai-dashboard', require: false

and then execute $ bundle.

Configuration

You need to provide stats_database_url to enable stats and save info about project spiders runs and sessions to a database. Format for a database url: https://sequel.jeremyevans.net/rdoc/files/doc/opening_databases_rdoc.html. You can use sqlite, postgres or mysql database (check Kimurai's project Gemfile and uncomment preferred gem).

Example for SQlite:

# Gemfile
gem 'sqlite3'

Note that dashboard should be required only after stats_database_url provided:

# config/boot.rb
# ...

Kimurai.configuration.stats_database_url = "sqlite://db/spiders_runs_#{Kimurai.env}.sqlite3"
# Important: require dashboard ONLY after stats_database_url was provided:
require 'kimurai/dashboard'

Also, there are optional settings for a dashboard:

# config/application.rb

Kimurai.configure do |config|
  # ...

  config.dashboard = {
    bind_address: "0.0.0.0",
    port: 3001,
    basic_auth: { username: "admin", password: "123456" }
  }
end

Usage

After successful configuration, all spiders (running individually kimurai start or in queue kimurai runner) will save stats to the database.

Run $ bundle exec kimurai dashboard and navigate to a dashboard url to see the stats.

License

The gem is available as open source under the terms of the MIT License.

You can’t perform that action at this time.