Skip to content

sudoAlireza/fiber-scraper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Simple Web Scraping Colly App with Fiber

This is a basic Go application using the Fiber framework to create scraping tasks in colly.

How to Run

  1. Clone the repository.
  2. Navigate to the project directory.
  3. Run docker compose up --build.
  4. Visit http://127.0.0.1:3000/api/healthchecker in a web browser or use a tool like curl to test it.
  5. Send GET request to http://127.0.0.1:3000/scrape/coursera to start scraping Coursera courses. And http://127.0.0.1:3000/scrape/quotes to scrape quotes.toscrape.com.

What It Does

  • Scrapes data from websites and stores in PostgreSQL database.

Releases

No releases published

Packages