Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 58 additions & 53 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,70 @@

</div>

Welcome to GoPool, **a project where 95% of the code is generated by GPT**. You can find the corresponding list of Commit and Prompt at [pro.devchat.ai](https://pro.devchat.ai).
Welcome to GoPool, **a project with 95% of the code is generated by GPT**. You can find the corresponding list of Commit and Prompt at [pro.devchat.ai](https://pro.devchat.ai).

GoPool is a high-performance, feature-rich, and easy-to-use worker pool library for Golang. It is designed to manage and recycle a pool of goroutines to complete tasks concurrently, improving the efficiency and performance of your applications.
GoPool is a **high-performance**, **feature-rich**, and **easy-to-use** worker pool library for Golang. It is designed to manage and recycle a pool of goroutines to complete tasks concurrently, improving the efficiency and performance of your applications.

<div align="center">
<img src="./logo/gopool.png" width="750">
</div>

## Performance Testing

This table shows the performance testing results for three Go libraries: GoPool, [ants](https://github.com/panjf2000/ants), and [pond](https://github.com/alitto/pond). The table includes the time it takes for each library to process 1 million tasks and the memory consumption in MB.

| Project | Time to Process 1M Tasks (s) | Memory Consumption (MB) |
|----------------|:----------------------------:|:-----------------------:|
| GoPool | 1.13 | 1.23 |
| [ants](https://github.com/panjf2000/ants) | 1.43 | 9.49 |
| [pond](https://github.com/alitto/pond) | 3.51 | 1.88 |

You can run the following commands to test the performance of GoPool, ants, and pond on your machine:

```bash
$ go test -benchmem -run=^$ -bench ^BenchmarkGoPoolWithMutex$ github.com/devchat-ai/gopool
$ go test -benchmem -run=^$ -bench ^BenchmarkAnts$ github.com/devchat-ai/gopool
$ go test -benchmem -run=^$ -bench ^BenchmarkPond$ github.com/devchat-ai/gopool
```

The results of the performance testing on my machine are as follows:

- GoPool

```bash
$ go test -benchmem -run=^$ -bench ^BenchmarkGoPoolWithMutex$ github.com/devchat-ai/gopool
goos: darwin
goarch: arm64
pkg: github.com/devchat-ai/gopool
BenchmarkGoPoolWithMutex-10 1 1131753125 ns/op 1966192 B/op 13609 allocs/op
PASS
ok github.com/devchat-ai/gopool 1.5085s
```

- ants

```bash
$ go test -benchmem -run=^$ -bench ^BenchmarkAnts$ github.com/devchat-ai/gopool
goos: darwin
goarch: arm64
pkg: github.com/devchat-ai/gopool
BenchmarkAnts-10 1 1425282750 ns/op 9952656 B/op 74068 allocs/op
PASS
ok github.com/devchat-ai/gopool 1.730s
```

- pond

```bash
$ go test -benchmem -run=^$ -bench ^BenchmarkPond$ github.com/devchat-ai/gopool
goos: darwin
goarch: arm64
pkg: github.com/devchat-ai/gopool
BenchmarkPond-10 1 3512323792 ns/op 1288984 B/op 11106 allocs/op
PASS
ok github.com/devchat-ai/gopool 3.946s
```

## Features

- [x] **Task Queue**: GoPool uses a thread-safe task queue to store tasks waiting to be processed. Multiple workers can simultaneously fetch tasks from this queue.
Expand Down Expand Up @@ -264,54 +320,3 @@ func main() {
```

In this example, if a task fails, it will be retried up to 3 times.

## Performance Testing

We have conducted several performance tests to evaluate the efficiency and performance of GoPool. Here are the results:

- **TestGoPoolWithMutex**:

```bash
$ go test -benchmem -run=^$ -bench ^BenchmarkGoPoolWithMutex$ github.com/devchat-ai/gopool

goos: darwin
goarch: arm64
pkg: github.com/devchat-ai/gopool
=== RUN BenchmarkGoPoolWithMutex
BenchmarkGoPoolWithMutex
BenchmarkGoPoolWithMutex-10 2 803105167 ns/op 17416408 B/op 1017209 allocs/op
PASS
ok github.com/devchat-ai/gopool 2.586s
```

- **TestGoPoolWithSpinLock**:

```bash
$ go test -benchmem -run=^$ -bench ^BenchmarkGoPoolWithSpinLock$ github.com/devchat-ai/gopool

goos: darwin
goarch: arm64
pkg: github.com/devchat-ai/gopool
=== RUN BenchmarkGoPoolWithSpinLock
BenchmarkGoPoolWithSpinLock
BenchmarkGoPoolWithSpinLock-10 2 662952562 ns/op 17327176 B/op 1016087 allocs/op
PASS
ok github.com/devchat-ai/gopool 2.322s
```

- **BenchmarkGoroutines**:

```bash
$ go test -benchmem -run=^$ -bench ^BenchmarkGoroutines$ github.com/devchat-ai/gopool

goos: darwin
goarch: arm64
pkg: github.com/devchat-ai/gopool
=== RUN BenchmarkGoroutines
BenchmarkGoroutines
BenchmarkGoroutines-10 3 371622847 ns/op 96642458 B/op 2005219 allocs/op
PASS
ok github.com/devchat-ai/gopool 2.410s
```

Please note that the actual performance may vary depending on the specific use case and system environment.
109 changes: 57 additions & 52 deletions README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,68 @@

欢迎来到 GoPool,这是**一个95%的代码由GPT生成的项目**。你可以在[pro.devchat.ai](https://pro.devchat.ai)找到相应的 Commit 和 Prompt 列表。

GoPool 是一个用 Golang 实现的高性能、功能丰富、易于使用的工作池库。它会管理和回收一组 goroutine 来并发完成任务,从而提高你的应用程序的效率和性能。
GoPool 是一个用 Golang 实现的**高性能**、**功能丰富**、**简单易用**的工作池库。它会管理和回收一组 goroutine 来并发完成任务,从而提高你的应用程序的效率和性能。

<div align="center">
<img src="./logo/gopool.png" width="750">
</div>

## 性能测试

这个表格展示了三个 Go 库 GoPool、[ants](https://github.com/panjf2000/ants) 和 [pond](https://github.com/alitto/pond)的性能测试结果。表格包括每个库处理 100 万个任务所需的时间和内存消耗(以 MB 为单位)。

| 项目 | 处理一百万任务耗时 (s) | 内存消耗 (MB) |
|----------------|:----------------------------:|:-----------------------:|
| GoPool | 1.13 | 1.23 |
| [ants](https://github.com/panjf2000/ants) | 1.43 | 9.49 |
| [pond](https://github.com/alitto/pond) | 3.51 | 1.88 |

你可以通过运行下列命令在你的机器上测试 GoPool、ants 和 pond 的性能:

```bash
$ go test -benchmem -run=^$ -bench ^BenchmarkGoPoolWithMutex$ github.com/devchat-ai/gopool
$ go test -benchmem -run=^$ -bench ^BenchmarkAnts$ github.com/devchat-ai/gopool
$ go test -benchmem -run=^$ -bench ^BenchmarkPond$ github.com/devchat-ai/gopool
```

在我的电脑上,性能测试的结果如下:

- GoPool

```bash
$ go test -benchmem -run=^$ -bench ^BenchmarkGoPoolWithMutex$ github.com/devchat-ai/gopool
goos: darwin
goarch: arm64
pkg: github.com/devchat-ai/gopool
BenchmarkGoPoolWithMutex-10 1 1131753125 ns/op 1966192 B/op 13609 allocs/op
PASS
ok github.com/devchat-ai/gopool 1.5085s
```

- ants

```bash
$ go test -benchmem -run=^$ -bench ^BenchmarkAnts$ github.com/devchat-ai/gopool
goos: darwin
goarch: arm64
pkg: github.com/devchat-ai/gopool
BenchmarkAnts-10 1 1425282750 ns/op 9952656 B/op 74068 allocs/op
PASS
ok github.com/devchat-ai/gopool 1.730s
```

- pond

```bash
$ go test -benchmem -run=^$ -bench ^BenchmarkPond$ github.com/devchat-ai/gopool
goos: darwin
goarch: arm64
pkg: github.com/devchat-ai/gopool
BenchmarkPond-10 1 3512323792 ns/op 1288984 B/op 11106 allocs/op
PASS
ok github.com/devchat-ai/gopool 3.946s
```

## 特性

- [x] **任务队列**:GoPool 使用一个线程安全的任务队列来存储等待处理的任务。多个工作器可以同时从这个队列中获取任务。
Expand Down Expand Up @@ -264,54 +320,3 @@ func main() {
```

在这个示例中,如果任务失败,它将重试最多3次。

## 性能测试

我们进行了几个性能测试来评估 GoPool 的效率和性能。以下是结果:

- **TestGoPoolWithMutex**:

```bash
$ go test -benchmem -run=^$ -bench ^BenchmarkGoPoolWithMutex$ github.com/devchat-ai/gopool

goos: darwin
goarch: arm64
pkg: github.com/devchat-ai/gopool
=== RUN BenchmarkGoPoolWithMutex
BenchmarkGoPoolWithMutex
BenchmarkGoPoolWithMutex-10 2 803105167 ns/op 17416408 B/op 1017209 allocs/op
PASS
ok github.com/devchat-ai/gopool 2.586s
```

- **TestGoPoolWithSpinLock**:

```bash
$ go test -benchmem -run=^$ -bench ^BenchmarkGoPoolWithSpinLock$ github.com/devchat-ai/gopool

goos: darwin
goarch: arm64
pkg: github.com/devchat-ai/gopool
=== RUN BenchmarkGoPoolWithSpinLock
BenchmarkGoPoolWithSpinLock
BenchmarkGoPoolWithSpinLock-10 2 662952562 ns/op 17327176 B/op 1016087 allocs/op
PASS
ok github.com/devchat-ai/gopool 2.322s
```

- **BenchmarkGoroutines**:

```bash
$ go test -benchmem -run=^$ -bench ^BenchmarkGoroutines$ github.com/devchat-ai/gopool

goos: darwin
goarch: arm64
pkg: github.com/devchat-ai/gopool
=== RUN BenchmarkGoroutines
BenchmarkGoroutines
BenchmarkGoroutines-10 3 371622847 ns/op 96642458 B/op 2005219 allocs/op
PASS
ok github.com/devchat-ai/gopool 2.410s
```

请注意,实际性能可能会根据具体的使用情况和系统环境而变化。
2 changes: 2 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,11 @@ module github.com/devchat-ai/gopool
go 1.20

require (
github.com/alitto/pond v1.8.3
github.com/daniel-hutao/spinlock v0.1.0
github.com/onsi/ginkgo/v2 v2.11.0
github.com/onsi/gomega v1.27.10
github.com/panjf2000/ants/v2 v2.8.1
)

require (
Expand Down
13 changes: 12 additions & 1 deletion go.sum
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
github.com/alitto/pond v1.8.3 h1:ydIqygCLVPqIX/USe5EaV/aSRXTRXDEI9JwuDdu+/xs=
github.com/alitto/pond v1.8.3/go.mod h1:CmvIIGd5jKLasGI3D87qDkQxjzChdKMmnXMg3fG6M6Q=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
Expand All @@ -20,14 +22,23 @@ github.com/onsi/ginkgo/v2 v2.11.0 h1:WgqUCUt/lT6yXoQ8Wef0fsNn5cAuMK7+KT9UFRz2tcU
github.com/onsi/ginkgo/v2 v2.11.0/go.mod h1:ZhrRA5XmEE3x3rhlzamx/JJvujdZoJ2uvgI7kR0iZvM=
github.com/onsi/gomega v1.27.10 h1:naR28SdDFlqrG6kScpT8VWpu1xWY5nJRCF3XaYyBjhI=
github.com/onsi/gomega v1.27.10/go.mod h1:RsS8tutOdbdgzbPtzzATp12yT7kM5I5aElG3evPbQ0M=
github.com/panjf2000/ants/v2 v2.8.1 h1:C+n/f++aiW8kHCExKlpX6X+okmxKXP7DWLutxuAPuwQ=
github.com/panjf2000/ants/v2 v2.8.1/go.mod h1:KIBmYG9QQX5U2qzFP/yQJaq/nSb6rahS9iEHkrCMgM8=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
golang.org/x/mod v0.10.0 h1:lFO9qtOdlre5W1jxS3r/4szv2/6iXxScdzjoBMXNhYk=
golang.org/x/net v0.12.0 h1:cfawfvKITfUsFCeJIHJrbSxpeu/E81khclypR0GVT50=
golang.org/x/net v0.12.0/go.mod h1:zEVYFnQC7m/vmpQFELhcD1EWkZlX69l4oqgmer6hfKA=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.2.0 h1:PUR+T4wwASmuSTYdKjYHI5TD22Wy5ogLU5qZCOLxBrI=
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.10.0 h1:SqMFp9UcQJZa+pmYuAKjd9xq1f0j5rLcDIk0mj4qAsA=
golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
Expand Down
53 changes: 48 additions & 5 deletions gopool_benchmark_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,19 @@ import (
"time"

"github.com/daniel-hutao/spinlock"
"github.com/alitto/pond"
"github.com/panjf2000/ants/v2"
)

const (
PoolSize = 1e4
TaskNum = 1e6
)

func BenchmarkGoPoolWithMutex(b *testing.B) {
var wg sync.WaitGroup
var taskNum = int(1e6)
pool := NewGoPool(1e4, WithLock(new(sync.Mutex)))
var taskNum = int(TaskNum)
pool := NewGoPool(PoolSize, WithLock(new(sync.Mutex)))
defer pool.Release()

b.ResetTimer()
Expand All @@ -31,8 +38,8 @@ func BenchmarkGoPoolWithMutex(b *testing.B) {

func BenchmarkGoPoolWithSpinLock(b *testing.B) {
var wg sync.WaitGroup
var taskNum = int(1e6)
pool := NewGoPool(1e4, WithLock(new(spinlock.SpinLock)))
var taskNum = int(TaskNum)
pool := NewGoPool(PoolSize, WithLock(new(spinlock.SpinLock)))
defer pool.Release()

b.ResetTimer()
Expand All @@ -52,7 +59,7 @@ func BenchmarkGoPoolWithSpinLock(b *testing.B) {

func BenchmarkGoroutines(b *testing.B) {
var wg sync.WaitGroup
var taskNum = int(1e6)
var taskNum = int(TaskNum)

for i := 0; i < b.N; i++ {
wg.Add(taskNum)
Expand All @@ -66,3 +73,39 @@ func BenchmarkGoroutines(b *testing.B) {
wg.Wait()
}
}

func BenchmarkPond(b *testing.B) {
pool := pond.New(PoolSize, 0, pond.MinWorkers(PoolSize))

taskFunc := func() {
time.Sleep(10 * time.Millisecond)
}

b.ResetTimer()
for i := 0; i < b.N; i++ {
for i := 0; i < TaskNum; i++ {
pool.Submit(taskFunc)
}
pool.StopAndWait()
}
}

func BenchmarkAnts(b *testing.B) {
var wg sync.WaitGroup
p, _ := ants.NewPool(PoolSize)
defer p.Release()

taskFunc := func() {
time.Sleep(10 * time.Millisecond)
wg.Done()
}

b.ResetTimer()
for i := 0; i < b.N; i++ {
for i := 0; i < TaskNum; i++ {
wg.Add(1)
_ = p.Submit(taskFunc)
}
wg.Wait()
}
}