This project implements an AI-powered career recommendation system that matches job seekers with relevant job postings based on their skills. The system uses natural language processing (NLP) and cosine similarity to analyze job descriptions and find the best matches for a user's skill set. It leverages job posting data collected from multiple countries using the Adzuna Jobs API.
- Project Overview
- Features
- Data Collection
- Technical Implementation
- Skill Extraction
- Recommendation System
- Results
- Future Improvements
- Setup and Installation
- Contributing
- License
The AI-Powered Career Recommendation System is designed to help job seekers find relevant job opportunities by matching their skills with job postings. The system:
- Collects job posting data from multiple countries using the Adzuna Jobs API.
- Extracts skills from job descriptions using NLP techniques.
- Builds a skill-job matrix to represent the relationship between jobs and skills.
- Recommends jobs based on cosine similarity between a user's skill set and job requirements.
This project demonstrates expertise in data collection, NLP, machine learning, and recommendation systems.
- Data Collection: Fetches job postings from multiple countries using the Adzuna API.
- Skill Extraction: Identifies relevant skills from job descriptions using spaCy and a custom skill keyword library.
- Recommendation Engine: Uses cosine similarity to match user skills with job requirements.
- Scalable Design: Handles large datasets efficiently using sparse matrices.
- Customizable: Allows users to adjust the number of recommendations and similarity thresholds.
Job posting data is collected using the Adzuna Jobs API, which provides detailed information including:
- Job titles
- Descriptions
- Salary information
- Company details
- Location data
- Job categories
- API Configuration: The Adzuna API is accessed using an API ID and key stored in environment variables.
- Job Fetching:
- The
get_job_data()function fetches a single page of job listings from a specific country. - The
fetch_all_jobs_from_all_countries()function iterates through all countries and pages to collect job listings.
- The
- Error Handling: Includes robust error handling for failed API requests and rate limiting to avoid overloading the API.
The project uses the following Python libraries:
requests: For API calls.pandas&numpy: For data manipulation.spacy: For natural language processing.scikit-learn: For text vectorization and similarity calculations.matplotlib&seaborn: For visualization.
- A comprehensive set of skill keywords is defined across multiple domains (e.g., Business, IT, Healthcare, Marketing).
- spaCy is used for advanced text processing, including:
- Loading the English language model.
- Creating a
PhraseMatcherfor efficient skill keyword matching. - Converting skill keywords into spaCy patterns.
The extract_skills() function processes job descriptions to identify mentioned skills:
- Converts text to lowercase for consistent matching.
- Uses spaCy's
PhraseMatcherto find skill keyword matches. - Returns unique skills found in each description.
This creates a structured representation of skills for each job posting, which is used to build the skill-job matrix.
The recommendation system works in several steps:
- Takes a list of user skills.
- Creates a binary vector matching the skill matrix format.
- Uses cosine similarity to compare the user profile with job skill requirements.
- Handles both sparse and dense matrices efficiently.
- Sorts jobs by similarity score.
- Returns the top N most relevant positions, including:
- Job title
- Category
- Similarity score
The system successfully recommends jobs based on user skills, with the following outcomes:
- High accuracy in matching user skills with job requirements.
- Scalable design capable of handling large datasets.
- Customizable recommendations based on user preferences.
| Job Title | Category | Similarity Score |
|---|---|---|
| Data Scientist | IT Jobs | 0.95 |
| Marketing Manager | Marketing Jobs | 0.92 |
| Software Engineer | Engineering Jobs | 0.90 |
To enhance the system further, consider the following improvements:
- Incorporate job postings from additional platforms (e.g., LinkedIn, Indeed).
- Collect more diverse data to improve recommendation accuracy.
- Create a user-friendly interface using Streamlit or Dash.
- Allow users to input skills, view recommendations, and explore job details.
- Experiment with advanced recommendation algorithms:
- Collaborative Filtering: Recommend jobs based on what similar users have chosen.
- Hybrid Models: Combine content-based and collaborative filtering for better recommendations.
- Use deep learning models (e.g., neural networks) for skill extraction and matching.
- Include salary predictions based on job trends.
- Provide skill gap analysis to help users identify areas for improvement.
- Add geographic filters to recommend jobs in specific locations.
- Use pre-trained language models (e.g., BERT) for more accurate skill extraction.
- Expand the skill keyword library to cover more domains and emerging skills.
- Python 3.8+
- Adzuna API ID and key (stored in environment variables).
- Clone the repository:
git clone https://github.com/KitwanaSh/job_recommender.git cd job_recommender - Install dependencies:
pip install -r requirements.txt
- Download the spaCy English model:
python -m spacy download en_core_web_sm
Contributions are welcome! If you'd like to contribute, please:
- Fork the repository.
- Create a new branch for your feature or bug fix.
- Submit a pull request with a detailed description of your changes.
This project is licensed under the MIT License. See the LICENSE file for details.
- Adzuna for providing the job posting data.
- spaCy and scikit-learn for their powerful NLP and machine learning tools.
- The open-source community for their contributions to Python libraries.