The aim of this project was to create a price comparison website for books where a user could input a search term and related books would be displayed. They could then select one of those books, and more detailed information about the book would be displayed along with price comparisons. This was done using web scraping to get comparison data from 5 different online book retailers (WHSmith, Blackwells, Dauntbooks, Foyles, Waterstones). This data was put into a database to be used later by Node.js to display it in a meaningful way on the website.
- Maven: Used to build this project, manage and maintain the different Java framework dependencies, run unit testing, and efficiently organise and package the project.
- Hibernate: Used to map object fields onto their respective fields in the tables in a MySQL database, for example a Book object’s “title” field would map onto the “title” column in the books table.
- Spring Beans: Manage class dependencies within the code, for example there was a “ScraperManager” class which depended on the WebScraper class, so Spring was used to inject those classes.
- Selenium: For scraping data from book retailer websites for 500+ books. Selenium was chosen over alternatives such as JSoup for its useful page interaction capabilities.
- JUnit: Used to run unit tests, testing different scraper and database functions.
- Vue: Used to dynamically display books on the front end (depending the user's query).
- Node.js
- Express
- Axios: For retrieving data from the local MySQL database through the RESTful API, and passing it to the front end.
- Mocha Chai: Used for testing JS in the front end, and to test the RESTful API endpoints.
- RESTful API: A RESTful API was created for this project, allowing users to scrape data straight from the database in JSON format.