All the News That's Fit to Scrape
In this assignment, you'll create a web app that lets users view and leave comments on the latest news. But you're not going to actually write any articles; instead, you'll flex your Mongoose and Cheerio muscles to scrape news from another site.
Before You Begin
Create a GitHub repo for this assignment and clone it to your computer. Any name will do -- just make sure it's related to this project in some fashion.
npm init. When that's finished, install and save these npm packages:
npm install express express-handlebars mongoose body-parser cheerio request axios morgan
NOTE: If you want to earn complete credit for your work, you must use all six of these packages in your assignment.
In order to deploy your project to Heroku, you must set up an mLab provision. mLab is remote MongoDB database that Heroku supports natively. Follow these steps to get it running:
Create a Heroku app in your project directory.
Run this command in your Terminal/Bash window:
heroku addons:create mongolab
This command will add the free mLab provision to your project.
When you go to connect your mongo database to mongoose, do so the following way:
- This code should connect mongoose to your remote mongolab database if deployed, but otherwise will connect to the local mongoHeadlines database on your computer.
Your site doesn't need to match the demo's style, but feel free to attempt something similar if you'd like. Otherwise, just be creative!
├── controllers | ├── fetch.js | ├── headline.js | └── note.js ├── models | ├── Headline.js | ├── index.js | └── Note.js ├── public | └── assets ├── routes | ├── api | ├── index.js | └── view ├── scripts | └── scrape.js └── views | ├── home.handlebars | ├── layouts | └── saved.handlebars ├── package-lock.json ├── package.json └── server.js
Create an app that accomplishes the following:
Whenever a user visits your site, the app should scrape stories from a news outlet of your choice and display them for the user. Each scraped article should be saved to your application database. At a minimum, the app should scrape and display the following information for each article:
Headline - the title of the article
Summary - a short summary of the article
URL - the url to the original article
Feel free to add more content to your database (photos, bylines, and so on).
Users should also be able to leave comments on the articles displayed and revisit them later. The comments should be saved to the database as well and associated with their articles. Users should also be able to delete comments left on articles. All stored comments should be visible to every user.
Beyond these requirements, be creative and have fun with this!
Go back to Saturday's activities if you need a refresher on how to partner one model with another.
Whenever you scrape a site for stories, make sure an article isn't already represented in your database before saving it; we don't want duplicates.
Don't just clear out your database and populate it with scraped articles whenever a user accesses your site.
- If your app deletes stories every time someone visits, your users won't be able to see any comments except the ones that they post.
Attempt to complete homework assignment as described in instructions. If unable to complete certain portions, please pseudocode these portions to describe what remains to be completed. Hosting on Heroku and adding a README.md are required for this homework. In addition, add this homework to your portfolio, more information can be found below.
Hosting on Heroku
Now that we have a backend to our applications, we use Heroku for hosting. Please note that while Heroku is free, it will request credit card information if you have more than 5 applications at a time or are adding a database.
Please see Heroku’s Account Verification Information for more details.
Create a README.md
README.md to your repository describing the project. Here are some resources for creating your
README.md. Here are some resources to help you along the way:
Add To Your Portfolio
After completing the homework please add the piece to your portfolio. Make sure to add a link to your updated portfolio in the comments section of your homework so the TAs can easily ensure you completed this step when they are grading the assignment. To receive an 'A' on any assignment, you must link to it from your portfolio.
One Last Thing
If you have any questions about this project or the material we have covered, please post them in the community channels in slack so that your fellow developers can help you! If you're still having trouble, you can come to office hours for assistance from your instructor and TAs.
That goes threefold for this week: MongoDB and Mongoose compose a challenging data management system. If there's anything you find confusing about these technologies, don't hesitate to speak with someone from the Bootcamp team.
One Paragraph of project description goes here
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
What things you need to install the software and how to install them
A step by step series of examples that tell you have to get a development env running
Say what the step will be
Give the example
End with an example of getting some data out of the system or using it for a little demo
Running the tests
Explain how to run the automated tests for this system
Break down into end to end tests
Explain what these tests test and why
Give an example
And coding style tests
Explain what these tests test and why
Give an example
Add additional notes about how to deploy this on a live system
Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.
- Sara Bracewell - Initial work - 2crazyflowers
See also the list of contributors who participated in this project.
This project is licensed under the MIT License - see the LICENSE.md file for details
- Hat tip to anyone who's code was used