ScrapeGPT is a Telegram bot designed to scrape and analyze websites, then answer questions based on the scraped content. The bot utilizes advanced natural language processing techniques to provide accurate responses to user queries.
The idea came from the job interview question when I was given an assignment to "study our company website and provide new product recommendations"> I decided to automate the process of studying a whole website and made ScrapeGPT.
Note:
You don't have to use it as a Telegram bot, at the very end of the scrapeGPT.py there is a commented main() function to use it as CLI. Also, there is an experimental file called scrapeGPT_gradio_app.py, where the same functionality is implemented in the form of Gradio app and using qDrant VectorDB, Ollama and GPT4ALLEmbeddings.
- Web Scraping: Automatically scrapes text from provided URLs, including PDF files.
- Context Retrieval: Utilizes embeddings and retrieval models to extract relevant context from scraped content.
- Question Answering: Generates answers to user questions based on the retrieved context.
- Robots.txt Parsing: Respects website's robots.txt to avoid scraping restricted areas.
- Database Management: Stores scraped content in a database for future reference and quick access.
- Proxy Support: Uses rotating proxies to bypass geo-restrictions and anonymize requests.
- LLM-based: Supports both public and local LLMs.
To get started with the bot, follow these steps:
- Clone the repository to your local machine:
git clone https://github.com/LexiestLeszek/scrapeGPT.git
cd scrapeGPT- Install the required dependencies:
pip install -r requirements.txt- Go to ollama.com and download it
- download local model:
ollama pull qwen:1.8b- Set up your Telegram bot by obtaining a token from the BotFather.
- Replace
API_TOKENin the script with your actual token and don't forget bot's telegram nickname. - Run the bot server script:
python scrapeGPT.pyOnce the bot is running, interact with it on Telegram:
- Find the bot by its name and open the chat with the bot
- Send
/startcommand to initialize the bot. - Provide a website URL to begin analysis.
- Ask a question regarding the analyzed content, and the bot will respond with an answer.
Contributions are welcome! To contribute:
- Fork the repository.
- Make changes in your fork.
- Submit a pull request with a clear description of your changes.
This project is licensed under the terms of the MIT license. See the LICENSE file for details.
If you have any questions or suggestions, feel free to open an issue on GitHub.
- Usage of this software for attacking targets without prior mutual consent is illegal.
- It is the end user's responsibility to obey all applicable local, state and federal laws.
- Developers of this software assume no liability and are not responsible for any misuse or damage caused by this program by third parties using the software in violation of laws and regulations.