GoScrapy: Web Scraping Framework in Go

GoScrapy aims to be a powerful web scraping framework in Go, inspired by Python's Scrapy framework. It offers an easy-to-use Scrapy-like experience for extracting data from websites, making it an ideal tool for various data collection and analysis tasks, especially for those coming from Python and wanting to try scraping in Golang..

Getting Started

Goscrapy is tested with go v1.21

1: Project Initialization

go mod init scrapejsp

2. Install goscrapy cli

go install github.com/tech-engine/goscrapy@latest

Note: make sure to always keep your goscrapy cli updated.

3. Verify Installation

goscrapy -v

4. Create a New Project

goscrapy startproject scrapejsp

This will create a new project directory with all the files necessary to begin working with GoScrapy.

\iyuioy\go\go-test-scrapy> goscrapy startproject scrapejsp

🚀 GoScrapy generating project files. Please wait!

✔️  scrapejsp\constants.go
✔️  scrapejsp\errors.go
✔️  scrapejsp\job.go
✔️  main.go
✔️  scrapejsp\record.go
✔️  scrapejsp\spider.go

✨ Congrates. scrapejsp created successfully.

main.go

In your main.go file, set up and execute your spider.

For detailed code, please refer to the sample code here.

package main

import (
	"context"
	"errors"
	"fmt"
	"os"
	"os/signal"
	"scrapejsp/scrapejsp"
	"sync"
	"syscall"
)

func main() {
	ctx, cancel := context.WithCancel(context.Background())

	var wg sync.WaitGroup
	wg.Add(1)

	spider, errCh := scrapejsp.New(ctx)
	
	go func() {
		defer wg.Done()

		err := <-errCh

		if err != nil && errors.Is(err, context.Canceled) {
			return
		}

		fmt.Printf("failed: %q", err)
	}()

	// trigger the Start Request
	spider.StartRequest(ctx, nil)

	OnTerminate(func() {
		fmt.Println("exit signal received: shutting down gracefully")
		cancel()
		wg.Wait()
	})
}

Wiki

Please follow the wiki docs for details.

Note

GoScrapy is not stable, so its API may change drastically. Please exercise caution when using it in production.

License

GoScrapy is available under BSL with additional usage grant which allows for free internal use. Please make sure that you agree with the license before contributing to GoScrapy because by contributing to goscrapy project you are agreeing on the license.

Roadmap

• Cookie management
• Builtin & Custom Middlewares support
• HTML parsing
• Triggers
• Unit Tests(work in progress)

Contact

Discord