Skip to content

Commit

Permalink
chore: rename mod, update go.mod
Browse files Browse the repository at this point in the history
  • Loading branch information
andeya committed Jun 26, 2022
1 parent fd4f526 commit 2be84c7
Show file tree
Hide file tree
Showing 125 changed files with 23,718 additions and 6,937 deletions.
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@
same "printed page" as the copyright notice for easier
identification within third-party archives.

Copyright 2015 HenryLee
Copyright 2022 AndeyaLee

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand Down
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Surfer [![GitHub release](https://img.shields.io/github/release/henrylee2cn/surfer.svg?style=flat-square)](https://github.com/henrylee2cn/surfer/releases) [![report card](https://goreportcard.com/badge/github.com/henrylee2cn/surfer?style=flat-square)](http://goreportcard.com/report/henrylee2cn/surfer) [![github issues](https://img.shields.io/github/issues/henrylee2cn/surfer.svg?style=flat-square)](https://github.com/henrylee2cn/surfer/issues?q=is%3Aopen+is%3Aissue) [![github closed issues](https://img.shields.io/github/issues-closed-raw/henrylee2cn/surfer.svg?style=flat-square)](https://github.com/henrylee2cn/surfer/issues?q=is%3Aissue+is%3Aclosed) [![GoDoc](https://img.shields.io/badge/godoc-reference-blue.svg?style=flat-square)](http://godoc.org/github.com/henrylee2cn/surfer) [![view Go大数据](https://img.shields.io/badge/官方QQ群-Go大数据(42731170)-27a5ea.svg?style=flat-square)](http://jq.qq.com/?_wv=1027&k=XnGGnc)
# Surfer [![GitHub release](https://img.shields.io/github/release/andeya/surfer.svg?style=flat-square)](https://github.com/andeya/surfer/releases) [![report card](https://goreportcard.com/badge/github.com/andeya/surfer?style=flat-square)](http://goreportcard.com/report/andeya/surfer) [![github issues](https://img.shields.io/github/issues/andeya/surfer.svg?style=flat-square)](https://github.com/andeya/surfer/issues?q=is%3Aopen+is%3Aissue) [![github closed issues](https://img.shields.io/github/issues-closed-raw/andeya/surfer.svg?style=flat-square)](https://github.com/andeya/surfer/issues?q=is%3Aissue+is%3Aclosed) [![GoDoc](https://img.shields.io/badge/godoc-reference-blue.svg?style=flat-square)](http://godoc.org/github.com/andeya/surfer) [![view Go大数据](https://img.shields.io/badge/官方QQ群-Go大数据(42731170)-27a5ea.svg?style=flat-square)](http://jq.qq.com/?_wv=1027&k=XnGGnc)

Package surfer is a high level concurrency http client.
It has `surf` and` phantom` download engines, highly simulated browser behavior, the function of analog login and so on.

[简体中文](https://github.com/henrylee2cn/surfer/blob/master/README_ZH.md)
[简体中文](https://github.com/andeya/surfer/blob/master/README_ZH.md)

## Features
- Both `surf` and `phantomjs` engines are supported
Expand All @@ -16,15 +16,15 @@ It has `surf` and` phantom` download engines, highly simulated browser behavior,
package main
import (
"github.com/henrylee2cn/surfer"
"github.com/andeya/surfer"
"io/ioutil"
"log"
)
func main() {
// Use surf engine
resp, err := surfer.Download(&surfer.Request{
Url: "http://github.com/henrylee2cn/surfer",
Url: "http://github.com/andeya/surfer",
})
if err != nil {
log.Fatal(err)
Expand All @@ -35,7 +35,7 @@ func main() {
// Use phantomjs engine
surfer.SetPhantomJsFilePath("Path to phantomjs.exe")
resp, err = surfer.Download(&surfer.Request{
Url: "http://github.com/henrylee2cn",
Url: "http://github.com/andeya",
DownloaderID: 1,
})
if err != nil {
Expand All @@ -48,8 +48,8 @@ func main() {
surfer.DestroyJsFiles()
}
```
[Full example](https://github.com/henrylee2cn/surfer/tree/master/example)
[Full example](https://github.com/andeya/surfer/tree/master/example)

## License

Surfer is under Apache v2 License. See the [LICENSE](https://github.com/henrylee2cn/surfer/raw/master/LICENSE) file for the full license text.
Surfer is under Apache v2 License. See the [LICENSE](https://github.com/andeya/surfer/raw/master/LICENSE) file for the full license text.
14 changes: 7 additions & 7 deletions README_ZH.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Surfer [![GitHub release](https://img.shields.io/github/release/henrylee2cn/surfer.svg?style=flat-square)](https://github.com/henrylee2cn/surfer/releases) [![report card](https://goreportcard.com/badge/github.com/henrylee2cn/surfer?style=flat-square)](http://goreportcard.com/report/henrylee2cn/surfer) [![github issues](https://img.shields.io/github/issues/henrylee2cn/surfer.svg?style=flat-square)](https://github.com/henrylee2cn/surfer/issues?q=is%3Aopen+is%3Aissue) [![github closed issues](https://img.shields.io/github/issues-closed-raw/henrylee2cn/surfer.svg?style=flat-square)](https://github.com/henrylee2cn/surfer/issues?q=is%3Aissue+is%3Aclosed) [![GoDoc](https://img.shields.io/badge/godoc-reference-blue.svg?style=flat-square)](http://godoc.org/github.com/henrylee2cn/surfer) [![view Go大数据](https://img.shields.io/badge/官方QQ群-Go大数据(42731170)-27a5ea.svg?style=flat-square)](http://jq.qq.com/?_wv=1027&k=XnGGnc)
# Surfer [![GitHub release](https://img.shields.io/github/release/andeya/surfer.svg?style=flat-square)](https://github.com/andeya/surfer/releases) [![report card](https://goreportcard.com/badge/github.com/andeya/surfer?style=flat-square)](http://goreportcard.com/report/andeya/surfer) [![github issues](https://img.shields.io/github/issues/andeya/surfer.svg?style=flat-square)](https://github.com/andeya/surfer/issues?q=is%3Aopen+is%3Aissue) [![github closed issues](https://img.shields.io/github/issues-closed-raw/andeya/surfer.svg?style=flat-square)](https://github.com/andeya/surfer/issues?q=is%3Aissue+is%3Aclosed) [![GoDoc](https://img.shields.io/badge/godoc-reference-blue.svg?style=flat-square)](http://godoc.org/github.com/andeya/surfer) [![view Go大数据](https://img.shields.io/badge/官方QQ群-Go大数据(42731170)-27a5ea.svg?style=flat-square)](http://jq.qq.com/?_wv=1027&k=XnGGnc)


Surfer 是一款Go语言编写的高并发 web 客户端,拥有surf与phantom两种下载内核,高度模拟浏览器行为,可实现模拟登录等功能。

高并发爬虫[Pholcus](https://github.com/henrylee2cn/pholcus)的专用下载器。
高并发爬虫[Pholcus](https://github.com/andeya/pholcus)的专用下载器。

## 特性

Expand All @@ -17,15 +17,15 @@ Surfer 是一款Go语言编写的高并发 web 客户端,拥有surf与phantom
package main
import (
"github.com/henrylee2cn/surfer"
"github.com/andeya/surfer"
"io/ioutil"
"log"
)
func main() {
// 默认使用surf内核下载
resp, err := surfer.Download(&surfer.Request{
Url: "http://github.com/henrylee2cn/surfer",
Url: "http://github.com/andeya/surfer",
})
if err != nil {
log.Fatal(err)
Expand All @@ -36,7 +36,7 @@ func main() {
// 指定使用phantomjs内核下载
surfer.SetPhantomJsFilePath("Path to phantomjs.exe")
resp, err = surfer.Download(&surfer.Request{
Url: "http://github.com/henrylee2cn",
Url: "http://github.com/andeya",
DownloaderID: 1,
})
if err != nil {
Expand All @@ -50,9 +50,9 @@ func main() {
}
```

[完整示例](https://github.com/henrylee2cn/surfer/tree/master/example)
[完整示例](https://github.com/andeya/surfer/tree/master/example)


## 开源协议

Surfer 项目采用商业应用友好的[Apache License v2](https://github.com/henrylee2cn/surfer/raw/master/LICENSE).发布
Surfer 项目采用商业应用友好的[Apache License v2](https://github.com/andeya/surfer/raw/master/LICENSE).发布
2 changes: 1 addition & 1 deletion agent.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright 2015 henrylee2cn Author. All Rights Reserved.
// Copyright 2015 andeya Author. All Rights Reserved.

// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
Expand Down
4 changes: 2 additions & 2 deletions body.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright 2015 henrylee2cn Author. All Rights Reserved.
// Copyright 2015 andeya Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
Expand All @@ -21,7 +21,7 @@ import (
"mime/multipart"
"net/url"

"github.com/henrylee2cn/goutil"
"github.com/andeya/goutil"
)

// body set request body
Expand Down
3 changes: 2 additions & 1 deletion example/example001/example.go
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
package example001

import (
"github.com/henrylee2cn/surfer"
"io/ioutil"
"log"
"net/url"
"time"

"github.com/andeya/surfer"
)

func main() {
Expand Down
9 changes: 5 additions & 4 deletions example/example002/example.go
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
package main

import (
"github.com/henrylee2cn/surfer"
"io/ioutil"
"log"
"net/url"
"time"

"github.com/andeya/surfer"
)

const (
Expand Down Expand Up @@ -58,7 +59,7 @@ func main() {
resp, err = surfer.Download(&surfer.Request{
Url: "http://httpbin.org/get",
DownloaderID: 1,
DialTimeout: time.Millisecond * 2000, //设置超时时间
DialTimeout: time.Millisecond * 2000, // 设置超时时间
})

handleError(err)
Expand All @@ -71,14 +72,14 @@ func main() {
log.Println("resp.Body=", string(b))

// 4.phantomjs内核POST下载测试开始---------------------------------------------------------------------------
surfer.SetPhantomJsFilePath("../phantomjs.exe") //相对路径
surfer.SetPhantomJsFilePath("../phantomjs.exe") // 相对路径
log.Println("phantomjs内核POST下载测试开始" + HR)
resp, err = surfer.Download(&surfer.Request{
DownloaderID: 1,
Url: "http://httpbin.org/post",
Method: "POST",
Body: form,
DialTimeout: time.Millisecond * 2000, //设置超时时间
DialTimeout: time.Millisecond * 2000, // 设置超时时间
})
handleError(err)
log.Println("resp.Status=", resp.Status)
Expand Down
7 changes: 4 additions & 3 deletions example/example003/example.go
Original file line number Diff line number Diff line change
@@ -1,17 +1,18 @@
package main

import (
"github.com/henrylee2cn/surfer"
"log"
"time"

"github.com/andeya/surfer"
)

const (
HR = "--------------------------------------------------------------------------"
)

func main() {
//默认内核
// 默认内核
log.Println("默认内核timeout" + HR)
log.Println(time.Now(), "start")
resp, err := surfer.Download(&surfer.Request{
Expand All @@ -26,7 +27,7 @@ func main() {
log.Println("resp.Status=", resp.Status)
}

//phantomjs内核
// phantomjs内核
surfer.SetPhantomJsFilePath("E:/Workspace/go-labs/src/lab089/lab003/phantomjs/phantomjs.exe")
log.Println("phantomjs内核" + HR)
log.Println(time.Now(), "start")
Expand Down
9 changes: 5 additions & 4 deletions example/example004/example.go
Original file line number Diff line number Diff line change
@@ -1,20 +1,21 @@
package main

import (
"github.com/henrylee2cn/surfer"
"io/ioutil"
"log"
"net/http"
"net/http/cookiejar"
"net/url"

"github.com/andeya/surfer"
)

const (
HR = "------------------------------------------------------------------"
)

func main() {
//默认内核
// 默认内核
jar1, _ := cookiejar.New(nil)
var cookies []*http.Cookie
cookie := &http.Cookie{
Expand All @@ -28,7 +29,7 @@ func main() {
u, _ := url.Parse("http://httpbin.org/cookies")
jar1.SetCookies(u, cookies)

//查看cookie-------------------------------------------------------------------------------------
// 查看cookie-------------------------------------------------------------------------------------
log.Println("查看Cookie" + HR)
defaultSurfer := surfer.New(jar1)
resp, err := defaultSurfer.Download(&surfer.Request{
Expand All @@ -43,7 +44,7 @@ func main() {
log.Println("body=", string(b))
log.Println("cookie=", jar1.Cookies(u))

//设置cookie-------------------------------------------------------------------------------------
// 设置cookie-------------------------------------------------------------------------------------
log.Println("设置Cookie" + HR)
resp, err = defaultSurfer.Download(&surfer.Request{
Url: "http://httpbin.org/cookies/set?k2=v2&k1=v1",
Expand Down
11 changes: 6 additions & 5 deletions example/example005/example.go
Original file line number Diff line number Diff line change
@@ -1,21 +1,22 @@
package main

import (
"github.com/henrylee2cn/surfer"
"io/ioutil"
"log"
"net/http"
"net/http/cookiejar"
"net/url"
"time"

"github.com/andeya/surfer"
)

const (
HR = "------------------------------------------------------------------"
)

func main() {
//phantomjs内核
// phantomjs内核
jar, _ := cookiejar.New(nil)
var cookies []*http.Cookie
cookie := &http.Cookie{
Expand All @@ -29,7 +30,7 @@ func main() {
u, _ := url.Parse("http://httpbin.org/cookies")
jar.SetCookies(u, cookies)

//查看cookie------------------------------------------------------------------
// 查看cookie------------------------------------------------------------------
log.Println("查看cookie" + HR)
phantomSurfer := surfer.NewPhantom("E:\\Workspace\\go-labs\\src\\lab089\\phantomjs.exe", "./tmp", jar)
resp, err := phantomSurfer.Download(&surfer.Request{
Expand All @@ -47,7 +48,7 @@ func main() {
resp.Body.Close()
log.Println("resp.Body=", string(b))

//设置cookie------------------------------------------------------------------
// 设置cookie------------------------------------------------------------------
log.Println("设置cookie" + HR)
resp, err = phantomSurfer.Download(&surfer.Request{
Url: "http://httpbin.org/cookies/set?k2=v2&k1=v1",
Expand All @@ -64,7 +65,7 @@ func main() {
resp.Body.Close()
log.Println("resp.Body=", string(b))

//删除js文件
// 删除js文件
phantomSurfer.(*surfer.Phantom).DestroyJsFiles()
time.Sleep(10e9)
}
Expand Down
14 changes: 9 additions & 5 deletions go.mod
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
// Deprecated: Use the "github.com/andeya/surfer" module instead.
module github.com/henrylee2cn/surfer
module github.com/andeya/surfer

require (
github.com/henrylee2cn/goutil v0.0.0-20190925102525-a28b3dc837cc
golang.org/x/net v0.0.0-20191014212845-da9a3fd4c582
github.com/andeya/goutil v0.0.0-20220626152529-9b7868da7b6d
golang.org/x/net v0.0.0-20220624214902-1bab6f366d9e
)

go 1.13
require (
github.com/henrylee2cn/ameda v1.4.10 // indirect
golang.org/x/text v0.3.7 // indirect
)

go 1.18
24 changes: 10 additions & 14 deletions go.sum
Original file line number Diff line number Diff line change
@@ -1,22 +1,18 @@
github.com/andeya/goutil v0.0.0-20220626152529-9b7868da7b6d h1:SFmlknB2bt6VEND93Uu55IP1HK5CXEk0sTUtxdr+CJQ=
github.com/andeya/goutil v0.0.0-20220626152529-9b7868da7b6d/go.mod h1:29HngAq7SN4CEJuycvNReDnCGYgCUzB9JzrFTj/u0xg=
github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/henrylee2cn/goutil v0.0.0-20190807075143-e8afa09140e9 h1:a2H8nokxD0y7RtMAD0c0iyRjIc6PPBopF2/8zw5kFiE=
github.com/henrylee2cn/goutil v0.0.0-20190807075143-e8afa09140e9/go.mod h1:I9qYeMYwdKC7UFXMECNzCEv0fYuolqLeBMqsmeG7IVo=
github.com/henrylee2cn/goutil v0.0.0-20190925102525-a28b3dc837cc h1:b4gkMTuTTNcGe4mXaNkqtO+pxua81JedxBy2xmAJy4M=
github.com/henrylee2cn/goutil v0.0.0-20190925102525-a28b3dc837cc/go.mod h1:81hpTd0/0IQ9/yK+HKFg489/HHZWEpslCQtbV8M0P3U=
github.com/henrylee2cn/ameda v1.4.10 h1:JdvI2Ekq7tapdPsuhrc4CaFiqw6QXFvZIULWJgQyCAk=
github.com/henrylee2cn/ameda v1.4.10/go.mod h1:liZulR8DgHxdK+MEwvZIylGnmcjzQ6N6f2PlWe7nEO4=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/net v0.0.0-20190724013045-ca1201d0de80 h1:Ao/3l156eZf2AW5wK8a7/smtodRU+gha3+BeqJ69lRk=
golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191014212845-da9a3fd4c582 h1:p9xBe/w/OzkeYVKm234g55gMdD1nSIooTir5kV11kfA=
golang.org/x/net v0.0.0-20191014212845-da9a3fd4c582/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/net v0.0.0-20220624214902-1bab6f366d9e h1:TsQ7F31D3bUCLeqPT0u+yjp1guoArKaNKmCr22PYgTQ=
golang.org/x/net v0.0.0-20220624214902-1bab6f366d9e/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

0 comments on commit 2be84c7

Please sign in to comment.