Skip to content

App for scraping and viewing job postings found on KSL's classifieds

Notifications You must be signed in to change notification settings

Hall-Erik/ksl-scraper

Repository files navigation

KSL Scraper

Build Status Coverage Status

A Django/Angular app for scraping and viewing job listings found on KSL classieds, but ordered by date posted.

This is deployed on Heroku here.

Development server

Backend

Open a terminal in the project root and run pip install -r requirements.txt to download dependencies.

Run python manage.py migrate to build a SQLite database.

Run python manage.py runserver for a dev server. The app will automatically reload if you change any of the source files.

Frontend

Open a terminal in the ng-frontend directory and run npm install

Run npm start for a dev server. Navigate to http://localhost:4200/. The app will automatically reload if you change any of the source files.

Docker Compose

Alternatively, you could just run the project in Docker.

Open a terminal in the project root and run docker-compose build to set up your containers.

Run docker-compose up to run the images.

Go to http://localhost:1337/ in your browser.

Running unit tests

Run python manage.py test to execute the tests.

Building the Frontend

The build process uses a Python script called ng2django. So, be sure to use an environment where you have installed dependencies with pip install -r requirements.txt.

Then you can cd into the ng-frontend directory. Run npm run build.

What I learned

  • Beautifulsoup for web scraping
  • Angular Material for the front end
  • Django for web development
  • Unit testing in Django
  • CI/CD with Travis-ci, Heroku and GitHub
  • Test coverage reporting with Coverage.py and Coveralls

About

App for scraping and viewing job postings found on KSL's classifieds

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published