Repository navigation
v0.1.0-alpha
Pre-release
Pre-release
·
7 commits
to main
since this release
This is an alpha pre-release of the FastAPI and Scrapy integration project. This version integrates FastAPI with Scrapy to create a web scraping API, allowing users to initiate scraping tasks via an API endpoint. The results are stored in a PostgreSQL database. This release is intended for testing and feedback purposes and is not recommended for production use.
Features included in this pre-release:
- FastAPI endpoint to initiate scraping tasks.
- Scrapy PyppeteerSpider to scrape web pages asynchronously.
- Concurrency control to prevent simultaneous scraping of the same domain.
- PostgreSQL integration for storing scraped data and domain statuses.
Please note:
- This is not a stable release, and features might change in future releases.
- Your feedback and contributions are welcome to improve the project.
Installation and Usage:
Refer to the README.md for detailed instructions on setting up and using the project.