Skip to content

v0.1.0-alpha

Pre-release
Pre-release

Choose a tag to compare

@DawoodKMasood DawoodKMasood released this 09 Mar 13:47
· 7 commits to main since this release

This is an alpha pre-release of the FastAPI and Scrapy integration project. This version integrates FastAPI with Scrapy to create a web scraping API, allowing users to initiate scraping tasks via an API endpoint. The results are stored in a PostgreSQL database. This release is intended for testing and feedback purposes and is not recommended for production use.

Features included in this pre-release:

  • FastAPI endpoint to initiate scraping tasks.
  • Scrapy PyppeteerSpider to scrape web pages asynchronously.
  • Concurrency control to prevent simultaneous scraping of the same domain.
  • PostgreSQL integration for storing scraped data and domain statuses.

Please note:

  • This is not a stable release, and features might change in future releases.
  • Your feedback and contributions are welcome to improve the project.

Installation and Usage:
Refer to the README.md for detailed instructions on setting up and using the project.