Home

Search Engine Scraper

A google scraper to search in google

Implementations

 Allow users to upload a csv file of keywords to search
 Parse the csv file to find the keywords
 for each keyword triggering a worker to scrape
 Worker creates an instance of the search engine scraper and scrape with the keyword
 Sidekiq worker should try repeated search in case of search failure
 Store the scrape results in database

Technologies

• Frontend HTML, CSS, Javascript, JQuery, Twitter Bootstrap (4.3.1)

• Backend Ruby on Rails ( Ruby: 2.6.0, Rails: 5.2.2.1)

• Background Job Sidekiq: 5.2.5, Redis

Pre requisite

1. Ruby - 2.6.0 ( Choose rvm or rbenv )

2. Redis brew install redis

3. Postgresql - 10.7 brew install postgresql brew services start postgresql

Setup

1. Clone the repository

git clone git@github.com:gajendrajena/search-engine-scraper.git

2. Install Dependencies

cd search-engine-scraper
bundle install

3. Create Database

rake db:create

4. Setup seed data

rake db:setup

5. Run server

rails s

redis-server

bundle exec sidekiq

6. To run testcases

bundle exec rspec

Deployment

Platform - Heroku

This application is currently deployed at https://search-engine-scraper.herokuapp.com/

Credits

Gajendra Jena