Website analyzer in the same vein as BuiltWith, Datanyze and NerdyData. Ruby on Rails 4. Free.
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
app Updates to location scan and experimenting with apps Jun 29, 2014
bin
config
db Added sidekiq for background scanning plus a rake task for importing … Jun 20, 2014
lib
log Second Jun 17, 2014
public Second Jun 17, 2014
test Second Jun 17, 2014
vendor/assets
.gitignore Added sidekiq for background scanning plus a rake task for importing … Jun 20, 2014
Gemfile
Gemfile.lock Updates to location scan and experimenting with apps Jun 29, 2014
LICENSE
Procfile Updates to location scan and experimenting with apps Jun 29, 2014
README.md
Rakefile
config.ru
screen.png

README.md

Site Lab

Site Lab aims to be an open-source replacement for website analysis tools such as BuiltWith, NerdyData, and DataNyze.

Site Lab is a Ruby on Rails application. It uses PostgreSQL as its database and Redis + Sidekiq for background processing.

How Does it Work?

Right now, it's fairly simple:

  • The MetaInspector Gem retrieves some basic info about the site/URL
  • There is a "Technology" model which stores regular expressions
  • Technologies are matched against the source of the sites/URLs
  • Much of the processing now happens in the background (via Sidekiq)

More complex analysis is in the works.

Installation

It's a Rails 4.1 app, so you'll need a dev environment that supports that (prolly RVM). You'll also need Redis installed and running (probably via Homebrew)

  • Clone the repo
  • Edit the database.yml file with your info
  • Run bundle install to install gems
  • Run bundle exec rake db:create to create the DB(s)
  • Run bundle exec rake db:seed to load the seed data
  • Run foreman start -p 3000 to start the rails server & sidekiq locally on port 3000

Importing Data

While you can surely add sites/URLs one-by-one in the app, most use-cases will involve importing large sets of URLs from files or external sites. With that in mind, I've started a set of Rake tasks for importing URLs. Currently, it includes:

  • Importing all startups from AngelList for a given market
  • Importing all startup/product URLs listed on Producthunt
  • Importing URLs from a text file (placed in app/import)
  • Importing all startup URLs from VCDelta

Run a rake -T to see the tasks and required parameters. There is also a sample text file in app/import.

Screenshot

Screenshot