Repository navigation
Home
Dawood Khan Masood edited this page Mar 9, 2024
·
3 revisions
This project integrates FastAPI with Scrapy to create a powerful and flexible web scraping API. Users can initiate scraping tasks via an API endpoint, and the results are stored in a PostgreSQL database, ensuring a robust and scalable solution for web scraping needs.
- Python 3.8+
- PostgreSQL
- pipenv or virtualenv
Clone the repository and install the required dependencies to get started with the project.
git clone https://github.com/ExoBinary/scrape-specter.git
cd scrape-specter
pip install -r requirements.txt
Make sure to set up your PostgreSQL database and update the .env file with your DATABASE_URL.
DATABASE_URL=postgresql://username:password@localhost/yourdatabase
Run the following Python script to create the necessary database tables:
from modules.database import create_tables
create_tables()
- main.py: Entry point for the FastAPI application.
- modules/spider.py: Contains the PyppeteerSpider class for scraping.
- modules/database.py: Database models and session management.
- .env: Environment variables for configuration.
Send a POST request to the /crawl/ endpoint with the target URL in the JSON body:
curl -X 'POST' \
'http://localhost:8000/crawl/' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"url": "https://example.com"}'
Contributions are welcome! If you have suggestions for improvements or want to contribute to the project.