This Python script scrapes job listings from Yellow Pages for a given city and job title, and updates a Google Sheets document with the scraped data. It utilizes the requests, BeautifulSoup, and gspread libraries.
- Python 3.x (Recommended: 3.6 or later)
requests([requests](URL requests library python ON Read the Docs docs.python-requests.org))beautifulsoup4(BeautifulSoup4)gspread(gspread)
-
Clone this repository to your local machine: git clone https://github.com/mujtabach2/yellowPagesScraper.git
-
Install the required Python libraries: pip install requests beautifulsoup4 gspread
-
Important: Create a file named
creds.jsonin the root directory of the project. This file will store your Google service account credentials, which are necessary for accessing Google Sheets. Keep this file confidential as it contains sensitive information. -
Change the name of the sheet within the script (
sh = gc.open("ScrapeToSheets").sheet1) to match your actual sheet name.
-
Run the
scrape.pyscript: python3 scrape.py -
Follow the prompts to enter the city you want to search in and the job title you are looking for.
-
The script will scrape job listings from Yellow Pages for the specified city and job title, and update the Google Sheets document with the scraped data.
-
The script will pause for 60 seconds between each scrape to comply with rate limits.