A simple web application to crawl webpages and check for suspicious words. Developed for fun and as a hobby project to experiment with FastAPI.
- Upload: Upload a CSV containing URLs to start the crawling process.
- Suspicious Word Detection: The crawler checks webpages for any suspicious words specified by the user.
- Real-time Feedback: The application provides real-time feedback on the webpages being scanned and the detection of suspicious words.
- Backend: FastAPI
- Web Templating: Jinja2
- Styling: TailwindCSS with DaisyUI
- Frontend Enhancements: HTMX and Hyperscript
- Deployment: Docker and docker-compose
- Crawler: Scrapy
Ensure you have Docker and docker-compose installed.
- Clone the repository:
git clone <repository-url>
- Navigate to the project directory:
cd <project-directory>
- Use docker-compose to build and run the application:
docker-compose up --build
- The application should now be running at http://localhost:8000.
- Go to the homepage at http://localhost:8000.
- Upload a .csv file containing the URLs you want to crawl.
- Input suspicious words for the crawler to detect.
- Start the web crawling process and monitor the results in real-time. (Integration with the crawler is currently pending)
Feedback is welcome! Feel free to open an issue or submit a pull request.
This project is open-source and available under the MIT License.