Meet GoScraper – the ultimate bot with no chill! 🤖 This tool not only scrapes a website's soul (a.k.a., its main content) but also uses Google's Gemini API to translate that garbled HTML mess into human-friendly lingo! If you're tired of squinting at tags and nested divs, GoScraper's got your back. It’s like magic, but with more Go code. ✨
Because sometimes you need the internet to just… make sense! 🧐
- Step 1: Point GoScraper at a website.
- Step 2: It scrapes.
- Step 3: The content is magically transformed into something you can read without a PhD in HTML hieroglyphics.
-
Clone the repo like it’s hot:
git clone https://github.com/anima-regem/GoScraper.git cd GoScraper -
Install dependencies:
go get -u github.com/google/generative-ai-go/genai go get -u github.com/gocolly/colly go get -u github.com/joho/godotenv
You'll need your secret API key to use the magic of AI 🤖🔮. Don’t worry; it’s just one tiny step!
-
Make a file called
.envin your project directory:touch .env
-
Inside
.env, add your Gemini API key:GEMINI_API_KEY=YOUR_SECRET_KEY💡 Pro Tip: Don’t commit your
.envfile to Git, unless you like exposing secrets and living dangerously!
Fire up GoScraper like this:
go run main.go <URL>-
Example: Scrape and beautify "https://example.com" 🌐
go run main.go https://example.com
-
The Result: Voilà! Human-readable content, courtesy of GoScraper and Gemini! 🌌
Too lazy to run Go every time? We got you. Just download the latest release and run it like a boss:
./goscraper <URL>- URL not starting with
https://– don’t worry; GoScraper will add it for you because it’s thoughtful like that. 💅 - Error loading
.envfile – make sure your.envfile is sitting right in the project folder, like a good dog. 🐶 - API Key Issues – if GoScraper screams, make sure you haven’t pasted your WiFi password by accident. 😉
- It's fun, fast, and mostly accurate! 😅
- Leaves your screen spotless (no HTML mess).
- Turns robot gibberish into readable content like a dream! 💤
Happy Scraping! Go forth and make the internet readable! 🌎