Royal road mining

Webscraper for RoyalRoad.com, custom-built with BeautifulSoup4. Current specific functions:

• Search novel
• Scrape novel data
• Scrape names and url's of multiple novels in a category

In-development:

• Storage and retrieval with MongoDB
• Extension of scraping pipeline to collect incomes on patreon if given
• Data Analysis pipeline to conduct exploratory data analysis
• NLP pipeline to summarise large amount of reviews and extract main critiques and praises

Possible improvements:

• Speed up storage and retrieval times. Make memory usage more efficient (explore async, caching requests, and PostgreSQL and MongoDB)