Skip to content
Dawood Khan Masood edited this page Mar 9, 2024 · 3 revisions

Welcome to the Scrape Specter Project Wiki!

This project integrates FastAPI with Scrapy to create a powerful and flexible web scraping API. Users can initiate scraping tasks via an API endpoint, and the results are stored in a PostgreSQL database, ensuring a robust and scalable solution for web scraping needs.

Table of Contents

Getting Started

Prerequisites

  1. Python 3.8+
  2. PostgreSQL
  3. pipenv or virtualenv

Installation

Clone the repository and install the required dependencies to get started with the project.

git clone https://github.com/ExoBinary/scrape-specter.git
cd scrape-specter
pip install -r requirements.txt

Setting Up the Database

Make sure to set up your PostgreSQL database and update the .env file with your DATABASE_URL.

DATABASE_URL=postgresql://username:password@localhost/yourdatabase

Run the following Python script to create the necessary database tables:

from modules.database import create_tables
create_tables()

Project Structure

  • main.py: Entry point for the FastAPI application.
  • modules/spider.py: Contains the PyppeteerSpider class for scraping.
  • modules/database.py: Database models and session management.
  • .env: Environment variables for configuration.

API Usage

Initiating a Crawl

Send a POST request to the /crawl/ endpoint with the target URL in the JSON body:

curl -X 'POST' \
  'http://localhost:8000/crawl/' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://example.com"}'

Contributing

Contributions are welcome! If you have suggestions for improvements or want to contribute to the project.