The search engine is implemented in Java. This repository contains the complete implementation of search engine except the crawler. The crawled data is available in the data subfolder. The search engine consists of the following.
- A tokenizer where the liguistic models are applied to the content of the HTML page forming the tokens. Stopwords are removed and the words are stemmed using snowball stemmer.
- Other properties of a webpage such as anchor links, anchor texts, HTML tags (Eg: title, b, i, h1, h2, p, ...) are extracted from the page and stored in the database. Used MapReduce for indexing the webpages.
- MongoDB is used for storing the inverted index.
- TF-IDF, cosine similarity, PageRank, weights for anchor text and HTML tags are the techniques used for scoring the search results.
- A webpage to enter the query and display the search results is created using JSP.
The search engine is deployed on Heroku. MongoDB is hosted using mLab's service. The repository is configured to auto build and deploy WAR on Heroku for every commit.