FJob

Application to search for job advertisements around the world. By scraping multiple job portals, you won't miss anything.

Report Bug · Request Feature

About The Project

FJob is a project that will make it easier for you to look for a new job. Every day, scrapers search popular job sites, process the collected data to provide you with new offers that can change your life. In the future, it will be possible for companies to add their own offers that will be specially marked as sponsored to distinguish them from those collected from other websites. The website is translated into two languages - Polish and English. Currently, data on job offers is collected from: nofluffjobs.com, justjoin.it, olx.pl, Pracuj.pl, praca.pl I'm still working on improving the website and adding new scrapers to expand the database of job offers.

Key Features

1. Scraping and processing data from various websites
2. Authentication with JWT
3. Automated scrapers by using Celery, Redis and Django Celery Beat
4. Reporting broken offers just with a few clicks
5. Registration, login, password change and account deletion
6. Contact form

How does scrapers works

Here I used Strategy Pattern to easily add additional scrapers. The first module is GetContentStrategy which is responsible for downloading the content (html) of the page and then saving it to the database.

The second module - Process is responsible for processing data that was saved in a raw state in the first module. Here, details such as the name of the offer, salary, work mode, type of contract, location, skills and much more are extracted. Then the processed data is saved to the database and waits for approval by the administrator in order to display it to users.

List of scrapers

• JustJoin.it
• nofluffjobs.com
• OLX.pl
• Praca.pl
• Pracuj.pl
• TheProtocol.it

Built With

• Python
  • Django Rest Framework
  • Django Celery Beat
  • Celery
  • Selenium
  • Gunicorn
• PostgreSQL (production), SQLite (dev)
• Docker and Docker-compose
• React (Javascript + Vite)
• Redis
• Nginx

Installation

Development

1. Clone git repository

git clone https://github.com/DEENUU1/fjob.git

2. Create dotenv file and add required data

cp .env_example .env

3. Install all requirements

pip install -r requirements.txt

4. Run DRF application

python manage.py runserver

5. Create superuser

python manage.py createsuperuser  

6. Run React application

npm run dev

Production

1. Clone git repository

git clone https://github.com/DEENUU1/fjob.git

2. Create dotenv file and add required data

cp .env_example .env

3. Run docker-compose

docker-compose build
docker-compose up 

Tests

To run pytests use this command

pytest

Custom Django commands

1. Scraper commands

python manage.py olx
python manage.py justjoinit
python manage.py nfj
python manage.py pracujpl
python manage.py pracapl
python manage.py theprotocol

License

See LICENSE.txt for more information.