Author: Luis Eduardo Ferro Diez luisedof10@gmail.com
This repository contains all my work for my MsC in Computer Science project.
Where To Sell Products (wtsp) is a project in which I try to solve this very same question. The idea is to characterize geographic areas in terms of its relationship with a selected set of products. The relationship is derived from geotagged texts, i.e., I gathered geotagged text data from Twitter and product reviews from Amazon. For the former I generated spatial clusters from which all the tweets are aggregated to form a single cluster-corpus. For the latter I trained a convolutional neural network to classify product categories given several review texts. Finally each cluster-corpus is submitted to the classifier to emit a relevance score for some categories. The result is displayed on a map of an area of interest, e.g., a city, in which the clusters are shown with their corresponding relationship score with certain products categories.
- Data Preparation (dataprep): It contains Apache Spark data engineering pipelines to prepare the raw sources for model training.
- Notebooks: It contains several notebooks with the plain experiments while developing the project, as well as a notebook showcasing the CLI usage.
- Runtime environments (env): It contains conda and docker environment configuration recipies and files.
- Java 1.8.x
- Apache Spark >= 2.3
- Conda >= 4.8.x
- Python 3.7.x
- CUDA 10.1
- Tensorflow 2.1.0
- Keras 2.3.1
- Data from twitter can be obtained from https://archive.org here.
- Data from Amazon product reviews can be obtained here.
Both twitter and product review data needs to be pre-processed, for this there are two spark projects under dataprep.
This job takes the twitter data, filters out the tweets that are not geotagged and sinks the result as parquet files.
This job takes the raw amazon product reviews and product metadata and converts them into 'documents' where each document has categories and either a review text or a product description.
Install the cli and follow the instructions to create the embeddings and train the classifier with the transformed product documents.
Use the cli to predict the detect and classify the geographic areas.
A more detailed process is written in jupyter notebooks here
I have created a docker image with the environment and cli pre-installed and configured to run experments starting from pre-processed data https://hub.docker.com/r/ohtar10/wtsp.
To download the image:
docker image push ohtar10/wtsp:0.1.1
GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007
See the LICENSE_ file in the root of this project for license details.