XSpider

XSpider是一款高效的站点爬虫工具，采用深度遍历策略，持续发现并访问页面中的新链接。

Screenshots

在releases下载二进制文件 ./xspider-v0.0.1-darwin-arm64 -url https://www.apple.com

make help

make run ARGS='-file test/test_urls.txt'

make run ARGS='-url https://www.apple.com'

Name		Name	Last commit message	Last commit date
Latest commit History 30 Commits
.github/workflows		.github/workflows
middleware		middleware
png		png
scripts		scripts
test		test
Makefile		Makefile
README.md		README.md
config.go		config.go
config.yaml		config.yaml
config_test.go		config_test.go
go.mod		go.mod
go.sum		go.sum
main.go		main.go
utils.go		utils.go
utils_test.go		utils_test.go