Sentiment Analysis and Aspect classification for Hotel Reviews
This is the source code of MonkeyLearn's series of posts related to analyzing sentiment and aspects from hotel reviews using machine learning models. This code runs in python2.7.
(May 2018 update -- TripAdvisor and Booking.com have changed their sites greatly since these spiders were written, and as such, they no longer work. The blog posts and code are still very useful as an example on how to build a Scrapy spider, but sadly, the examples themselves are no longer functional. We will probably fix the spiders in the future, since it's probably enough to update all the selectors to get everything working again.)
The project itself is a Scrapy project that is used to gather training and testing data from different sites like TripAdvisor and Booking. Besides, there are a series of Python scripts and Jupyter notebooks that implement some necessary scripts.
The TripAdvisor (hotel_sentiment/spider/tripadvisor_spider.py) spider is used to gather data to train a sentiment analysis classifier in MonkeyLearn. Reviews texts are used as the sample content and reviews stars are used as the category (1 and 2 stars = Negative, 4 and 5 stars = Positive).
To crawl ~15000 items from tripadvisor use:
scrapy crawl tripadvisor -o itemsTripadvisor.csv -s CLOSESPIDER_ITEMCOUNT=15000
You can check out the generated machine learning sentiment analysis model here.
The Booking spider (hotel_sentiment/spider/booking_spider.py) is used to gather data to train an aspect classifier in MonkeyLearn. The data obtained with this spider can be manually tagged with each aspect (eg: cleanliness, comfort & facilities, food, internet, location, staff, value for money) using MonkeyLearn's Sample tab or an external crowd sourcing service like Mechanical Turk.
To crawl from booking use:
scrapy crawl booking -o itemsBooking.csv
You first have to add the url of a starting city. To crawl from a single hotel in booking use:
scrapy crawl booking_singlehotel -o <hotel name>.csv
opinionTokenizer.pyis a simple script to obtain the "opinion units" from each review.
classify_and_plot_reviews.ipynbis a simple script that uses the generated model to classify new reviews and then plot the results in a graph using Plotly.
You can check out the generated machine learning aspect classifier here.
To crawl from Tripadvisor use:
scrapy crawl tripadvisor_more -a start_url="http://some_url" -o <hotel_name>.csv -s CLOSESPIDER_ITEMCOUNT=20000
With the url of a starting city to crawl from, such as https://www.tripadvisor.com/Hotels-g186338-London_England-Hotels.html.
The scripts and notebooks necessary to replicate the post are in the
classify_elastic/generate_files_for_indexing.pywill take the csv file produced by scrapy and generate two files that other scripts will use.
classify_elastic/classify_pipe.pywill open the
opinion_unitsfile and classify it with MonkeyLearn according to topic and sentiment, and save the results to a new csv file.
classify_elastic/index_definition.jsoncontains the mapping definitions used in ElasticSearch.
classify_elastic/index_reviews.pywill index into your ElasticSearch instance the reviews generated by
classify_elastic/index_opinion_units.pywill index into your ElasticSearch instance the classified opinion units.
classify_elastic/Extract keywords.ipynbshows how to extract keywords from the indexed data.
queries folder contains some queries that were used to power the Kibana visualization.