A Django-based web scraping project designed to extract data from online maps and location directories using automated browser control and data processing tools.
- Automated web scraping using Selenium and WebDriver for dynamic content.
- Efficient parsing and data extraction with BeautifulSoup (
bs4). - Data handling and analysis using Pandas and NumPy.
- HTTP requests management via the Requests library.
- Built on the Django framework for easy integration and extensibility.
- Clone the repository:
git clone https://github.com/your-username/map-scraper.git
cd map-scraper
- Create a Python virtual environment:
python3 -m venv venv
source venv/bin/activate # On Windows use venv\Scripts\activate
- Install dependencies:
pip install -r requirements.txt
The project requires the following Python packages:
- Django
- selenium
- webdriver-manager (to manage browser drivers easily)
- pandas
- numpy
- requests
- beautifulsoup4 (
bs4)
Make sure you have them installed either via requirements.txt or manually via pip.
- Start the Django development server:
python manage.py runserver
-
Access the scraper functionality via the configured routes/views in your browser or trigger scraping jobs as defined.
-
Customize scraping logic in the provided Django app modules as needed.
- The project uses Selenium WebDriver; make sure you have the appropriate browser driver installed (e.g., chromedriver for Chrome).
- For easier management, install and use
webdriver-managerwhich automatically downloads and manages drivers.
Django>=4.0
selenium>=4.10
webdriver-manager>=3.8
pandas>=2.0
numpy>=1.25
requests>=2.31
beautifulsoup4>=4.12
This project is licensed under the MIT License - see the LICENSE file for details.
For issues or requests, please contact:
- Email: haneishiteiru@gmail.com
- GitHub: https://github.com/easikinc/
Thank you for using Map-Scraper! Contributions and feedback are welcome.