k-Scraping

k-Scraping is a full-stack application that scrapes Korean Drama news from the web!

About

The purpose of this project is to learn the process of web scraping and the storage of data utilizing MongoDB and Mongoose. The incorporation of Mongoose allows the establishment of associations between documents of different models, such that articles are easily displayed with their associated comments. Passport.js is included to add Google OAuth 2.0 to differentiate between individual users, supporting personalized bookmarks and comments.

Deployment

The application is deployed to Heroku: http://jl-kscraping.herokuapp.com

• There may be a small delay when you first try to access the application due to the web dyno going to sleep if it receives no traffic for 30 minutes

How To

Main Page

• The landing page displays the five latest articles in the database.

  

• Clicking on "Get Latest k-Scraps" will scrape the articles seen at Dramabeans News

• If the scrape was successful, the new articles will replace the current articles on the page.

  

• Articles cannot be repeated in the database, so if the database already contains all of the scraped articles, then a modal will pop up to alert users that the latest articles are already displayed

  

• Clicking on "Load More k-Scraps" will display five older articles on the page.

• If all articles in the database has been displayed, clicking "Load More k-Scraps" will pop up a modal to inform users that there are no more articles.

  

Articles

• Each article includes the headline, a small preview of the full article, a "Read Article" button and a "Comments" button.

  

• Clicking on the "Read Article" button will direct the user to the external webpage of the article

• Clicking on the "Comments" button will reveal the comments section of the article.

  

• All users can read the existing comments of an article, but only signed-in users can write and post a new comment.

  

• Users can only delete their own comments.

  

Bookmarks

• When a user is signed in, they can see a bookmark button on each article.

  

• Clicking on the bookmark button will add the articles to the user's bookmarks.

• Clicking on "Bookmarks" in the nav-bar will direct users to their bookmarks page.

• On the bookmarks page, a user can click on the trash can button to delete a bookmarked article.

  

Technologies Used

• Bootstrap and Handlebars.js
• MongoDB and Mongoose
• Node.js and Express.js
• Axios and Cheerio
• Passport.js, Cookie Parser, Cookie Session
• Moment
• Async