About The Project

Multiprocessing crawlers were built to crawl data from different forums with polite frequencies. Each crawler consists of classes:

1. Crawler class: stores the browser headers, interacts with our database server, sends & receives data from the corresponding forum website, and handles failed requests.
2. Refiner class: called by the Crawler class to extract useful information from raw HTML contents sent by forum websites.
3. Parser class: called by the Crawler class to lemmatize posts and replies to generate tokens.

TF-IDF was generated for each dataset and then classified by using DBSCAN and K-means.

Built With

• NLTK
• Beautiful Soup
• PyMongo
• urllib3
• sklearn

Data Storage Example

Classification Example

Link to Our Dataset

• Hosted by Zenodo

(Back to Top)