diff --git a/README.md b/README.md index 554596b..f25db56 100644 --- a/README.md +++ b/README.md @@ -15,14 +15,70 @@ -Welcome to GoPool, **a project where 95% of the code is generated by GPT**. You can find the corresponding list of Commit and Prompt at [pro.devchat.ai](https://pro.devchat.ai). +Welcome to GoPool, **a project with 95% of the code is generated by GPT**. You can find the corresponding list of Commit and Prompt at [pro.devchat.ai](https://pro.devchat.ai). -GoPool is a high-performance, feature-rich, and easy-to-use worker pool library for Golang. It is designed to manage and recycle a pool of goroutines to complete tasks concurrently, improving the efficiency and performance of your applications. +GoPool is a **high-performance**, **feature-rich**, and **easy-to-use** worker pool library for Golang. It is designed to manage and recycle a pool of goroutines to complete tasks concurrently, improving the efficiency and performance of your applications.
+## Performance Testing + +This table shows the performance testing results for three Go libraries: GoPool, [ants](https://github.com/panjf2000/ants), and [pond](https://github.com/alitto/pond). The table includes the time it takes for each library to process 1 million tasks and the memory consumption in MB. + +| Project | Time to Process 1M Tasks (s) | Memory Consumption (MB) | +|----------------|:----------------------------:|:-----------------------:| +| GoPool | 1.13 | 1.23 | +| [ants](https://github.com/panjf2000/ants) | 1.43 | 9.49 | +| [pond](https://github.com/alitto/pond) | 3.51 | 1.88 | + +You can run the following commands to test the performance of GoPool, ants, and pond on your machine: + +```bash +$ go test -benchmem -run=^$ -bench ^BenchmarkGoPoolWithMutex$ github.com/devchat-ai/gopool +$ go test -benchmem -run=^$ -bench ^BenchmarkAnts$ github.com/devchat-ai/gopool +$ go test -benchmem -run=^$ -bench ^BenchmarkPond$ github.com/devchat-ai/gopool +``` + +The results of the performance testing on my machine are as follows: + +- GoPool + +```bash +$ go test -benchmem -run=^$ -bench ^BenchmarkGoPoolWithMutex$ github.com/devchat-ai/gopool +goos: darwin +goarch: arm64 +pkg: github.com/devchat-ai/gopool +BenchmarkGoPoolWithMutex-10 1 1131753125 ns/op 1966192 B/op 13609 allocs/op +PASS +ok github.com/devchat-ai/gopool 1.5085s +``` + +- ants + +```bash +$ go test -benchmem -run=^$ -bench ^BenchmarkAnts$ github.com/devchat-ai/gopool +goos: darwin +goarch: arm64 +pkg: github.com/devchat-ai/gopool +BenchmarkAnts-10 1 1425282750 ns/op 9952656 B/op 74068 allocs/op +PASS +ok github.com/devchat-ai/gopool 1.730s +``` + +- pond + +```bash +$ go test -benchmem -run=^$ -bench ^BenchmarkPond$ github.com/devchat-ai/gopool +goos: darwin +goarch: arm64 +pkg: github.com/devchat-ai/gopool +BenchmarkPond-10 1 3512323792 ns/op 1288984 B/op 11106 allocs/op +PASS +ok github.com/devchat-ai/gopool 3.946s +``` + ## Features - [x] **Task Queue**: GoPool uses a thread-safe task queue to store tasks waiting to be processed. Multiple workers can simultaneously fetch tasks from this queue. @@ -264,54 +320,3 @@ func main() { ``` In this example, if a task fails, it will be retried up to 3 times. - -## Performance Testing - -We have conducted several performance tests to evaluate the efficiency and performance of GoPool. Here are the results: - -- **TestGoPoolWithMutex**: - -```bash -$ go test -benchmem -run=^$ -bench ^BenchmarkGoPoolWithMutex$ github.com/devchat-ai/gopool - -goos: darwin -goarch: arm64 -pkg: github.com/devchat-ai/gopool -=== RUN BenchmarkGoPoolWithMutex -BenchmarkGoPoolWithMutex -BenchmarkGoPoolWithMutex-10 2 803105167 ns/op 17416408 B/op 1017209 allocs/op -PASS -ok github.com/devchat-ai/gopool 2.586s -``` - -- **TestGoPoolWithSpinLock**: - -```bash -$ go test -benchmem -run=^$ -bench ^BenchmarkGoPoolWithSpinLock$ github.com/devchat-ai/gopool - -goos: darwin -goarch: arm64 -pkg: github.com/devchat-ai/gopool -=== RUN BenchmarkGoPoolWithSpinLock -BenchmarkGoPoolWithSpinLock -BenchmarkGoPoolWithSpinLock-10 2 662952562 ns/op 17327176 B/op 1016087 allocs/op -PASS -ok github.com/devchat-ai/gopool 2.322s -``` - -- **BenchmarkGoroutines**: - -```bash -$ go test -benchmem -run=^$ -bench ^BenchmarkGoroutines$ github.com/devchat-ai/gopool - -goos: darwin -goarch: arm64 -pkg: github.com/devchat-ai/gopool -=== RUN BenchmarkGoroutines -BenchmarkGoroutines -BenchmarkGoroutines-10 3 371622847 ns/op 96642458 B/op 2005219 allocs/op -PASS -ok github.com/devchat-ai/gopool 2.410s -``` - -Please note that the actual performance may vary depending on the specific use case and system environment. diff --git a/README_zh.md b/README_zh.md index 13cedcc..cb5ca53 100644 --- a/README_zh.md +++ b/README_zh.md @@ -17,12 +17,68 @@ 欢迎来到 GoPool,这是**一个95%的代码由GPT生成的项目**。你可以在[pro.devchat.ai](https://pro.devchat.ai)找到相应的 Commit 和 Prompt 列表。 -GoPool 是一个用 Golang 实现的高性能、功能丰富、易于使用的工作池库。它会管理和回收一组 goroutine 来并发完成任务,从而提高你的应用程序的效率和性能。 +GoPool 是一个用 Golang 实现的**高性能**、**功能丰富**、**简单易用**的工作池库。它会管理和回收一组 goroutine 来并发完成任务,从而提高你的应用程序的效率和性能。
+## 性能测试 + +这个表格展示了三个 Go 库 GoPool、[ants](https://github.com/panjf2000/ants) 和 [pond](https://github.com/alitto/pond)的性能测试结果。表格包括每个库处理 100 万个任务所需的时间和内存消耗(以 MB 为单位)。 + +| 项目 | 处理一百万任务耗时 (s) | 内存消耗 (MB) | +|----------------|:----------------------------:|:-----------------------:| +| GoPool | 1.13 | 1.23 | +| [ants](https://github.com/panjf2000/ants) | 1.43 | 9.49 | +| [pond](https://github.com/alitto/pond) | 3.51 | 1.88 | + +你可以通过运行下列命令在你的机器上测试 GoPool、ants 和 pond 的性能: + +```bash +$ go test -benchmem -run=^$ -bench ^BenchmarkGoPoolWithMutex$ github.com/devchat-ai/gopool +$ go test -benchmem -run=^$ -bench ^BenchmarkAnts$ github.com/devchat-ai/gopool +$ go test -benchmem -run=^$ -bench ^BenchmarkPond$ github.com/devchat-ai/gopool +``` + +在我的电脑上,性能测试的结果如下: + +- GoPool + +```bash +$ go test -benchmem -run=^$ -bench ^BenchmarkGoPoolWithMutex$ github.com/devchat-ai/gopool +goos: darwin +goarch: arm64 +pkg: github.com/devchat-ai/gopool +BenchmarkGoPoolWithMutex-10 1 1131753125 ns/op 1966192 B/op 13609 allocs/op +PASS +ok github.com/devchat-ai/gopool 1.5085s +``` + +- ants + +```bash +$ go test -benchmem -run=^$ -bench ^BenchmarkAnts$ github.com/devchat-ai/gopool +goos: darwin +goarch: arm64 +pkg: github.com/devchat-ai/gopool +BenchmarkAnts-10 1 1425282750 ns/op 9952656 B/op 74068 allocs/op +PASS +ok github.com/devchat-ai/gopool 1.730s +``` + +- pond + +```bash +$ go test -benchmem -run=^$ -bench ^BenchmarkPond$ github.com/devchat-ai/gopool +goos: darwin +goarch: arm64 +pkg: github.com/devchat-ai/gopool +BenchmarkPond-10 1 3512323792 ns/op 1288984 B/op 11106 allocs/op +PASS +ok github.com/devchat-ai/gopool 3.946s +``` + ## 特性 - [x] **任务队列**:GoPool 使用一个线程安全的任务队列来存储等待处理的任务。多个工作器可以同时从这个队列中获取任务。 @@ -264,54 +320,3 @@ func main() { ``` 在这个示例中,如果任务失败,它将重试最多3次。 - -## 性能测试 - -我们进行了几个性能测试来评估 GoPool 的效率和性能。以下是结果: - -- **TestGoPoolWithMutex**: - -```bash -$ go test -benchmem -run=^$ -bench ^BenchmarkGoPoolWithMutex$ github.com/devchat-ai/gopool - -goos: darwin -goarch: arm64 -pkg: github.com/devchat-ai/gopool -=== RUN BenchmarkGoPoolWithMutex -BenchmarkGoPoolWithMutex -BenchmarkGoPoolWithMutex-10 2 803105167 ns/op 17416408 B/op 1017209 allocs/op -PASS -ok github.com/devchat-ai/gopool 2.586s -``` - -- **TestGoPoolWithSpinLock**: - -```bash -$ go test -benchmem -run=^$ -bench ^BenchmarkGoPoolWithSpinLock$ github.com/devchat-ai/gopool - -goos: darwin -goarch: arm64 -pkg: github.com/devchat-ai/gopool -=== RUN BenchmarkGoPoolWithSpinLock -BenchmarkGoPoolWithSpinLock -BenchmarkGoPoolWithSpinLock-10 2 662952562 ns/op 17327176 B/op 1016087 allocs/op -PASS -ok github.com/devchat-ai/gopool 2.322s -``` - -- **BenchmarkGoroutines**: - -```bash -$ go test -benchmem -run=^$ -bench ^BenchmarkGoroutines$ github.com/devchat-ai/gopool - -goos: darwin -goarch: arm64 -pkg: github.com/devchat-ai/gopool -=== RUN BenchmarkGoroutines -BenchmarkGoroutines -BenchmarkGoroutines-10 3 371622847 ns/op 96642458 B/op 2005219 allocs/op -PASS -ok github.com/devchat-ai/gopool 2.410s -``` - -请注意,实际性能可能会根据具体的使用情况和系统环境而变化。 diff --git a/go.mod b/go.mod index 9b4fe6b..01d6252 100644 --- a/go.mod +++ b/go.mod @@ -3,9 +3,11 @@ module github.com/devchat-ai/gopool go 1.20 require ( + github.com/alitto/pond v1.8.3 github.com/daniel-hutao/spinlock v0.1.0 github.com/onsi/ginkgo/v2 v2.11.0 github.com/onsi/gomega v1.27.10 + github.com/panjf2000/ants/v2 v2.8.1 ) require ( diff --git a/go.sum b/go.sum index 688336a..0a75ab0 100644 --- a/go.sum +++ b/go.sum @@ -1,3 +1,5 @@ +github.com/alitto/pond v1.8.3 h1:ydIqygCLVPqIX/USe5EaV/aSRXTRXDEI9JwuDdu+/xs= +github.com/alitto/pond v1.8.3/go.mod h1:CmvIIGd5jKLasGI3D87qDkQxjzChdKMmnXMg3fG6M6Q= github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= @@ -20,14 +22,23 @@ github.com/onsi/ginkgo/v2 v2.11.0 h1:WgqUCUt/lT6yXoQ8Wef0fsNn5cAuMK7+KT9UFRz2tcU github.com/onsi/ginkgo/v2 v2.11.0/go.mod h1:ZhrRA5XmEE3x3rhlzamx/JJvujdZoJ2uvgI7kR0iZvM= github.com/onsi/gomega v1.27.10 h1:naR28SdDFlqrG6kScpT8VWpu1xWY5nJRCF3XaYyBjhI= github.com/onsi/gomega v1.27.10/go.mod h1:RsS8tutOdbdgzbPtzzATp12yT7kM5I5aElG3evPbQ0M= +github.com/panjf2000/ants/v2 v2.8.1 h1:C+n/f++aiW8kHCExKlpX6X+okmxKXP7DWLutxuAPuwQ= +github.com/panjf2000/ants/v2 v2.8.1/go.mod h1:KIBmYG9QQX5U2qzFP/yQJaq/nSb6rahS9iEHkrCMgM8= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= -github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0= +github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= +github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= +github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= +github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= golang.org/x/mod v0.10.0 h1:lFO9qtOdlre5W1jxS3r/4szv2/6iXxScdzjoBMXNhYk= golang.org/x/net v0.12.0 h1:cfawfvKITfUsFCeJIHJrbSxpeu/E81khclypR0GVT50= golang.org/x/net v0.12.0/go.mod h1:zEVYFnQC7m/vmpQFELhcD1EWkZlX69l4oqgmer6hfKA= +golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.2.0 h1:PUR+T4wwASmuSTYdKjYHI5TD22Wy5ogLU5qZCOLxBrI= golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.10.0 h1:SqMFp9UcQJZa+pmYuAKjd9xq1f0j5rLcDIk0mj4qAsA= golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= diff --git a/gopool_benchmark_test.go b/gopool_benchmark_test.go index 41c3a58..4eb7afe 100644 --- a/gopool_benchmark_test.go +++ b/gopool_benchmark_test.go @@ -6,12 +6,19 @@ import ( "time" "github.com/daniel-hutao/spinlock" + "github.com/alitto/pond" + "github.com/panjf2000/ants/v2" +) + +const ( + PoolSize = 1e4 + TaskNum = 1e6 ) func BenchmarkGoPoolWithMutex(b *testing.B) { var wg sync.WaitGroup - var taskNum = int(1e6) - pool := NewGoPool(1e4, WithLock(new(sync.Mutex))) + var taskNum = int(TaskNum) + pool := NewGoPool(PoolSize, WithLock(new(sync.Mutex))) defer pool.Release() b.ResetTimer() @@ -31,8 +38,8 @@ func BenchmarkGoPoolWithMutex(b *testing.B) { func BenchmarkGoPoolWithSpinLock(b *testing.B) { var wg sync.WaitGroup - var taskNum = int(1e6) - pool := NewGoPool(1e4, WithLock(new(spinlock.SpinLock))) + var taskNum = int(TaskNum) + pool := NewGoPool(PoolSize, WithLock(new(spinlock.SpinLock))) defer pool.Release() b.ResetTimer() @@ -52,7 +59,7 @@ func BenchmarkGoPoolWithSpinLock(b *testing.B) { func BenchmarkGoroutines(b *testing.B) { var wg sync.WaitGroup - var taskNum = int(1e6) + var taskNum = int(TaskNum) for i := 0; i < b.N; i++ { wg.Add(taskNum) @@ -66,3 +73,39 @@ func BenchmarkGoroutines(b *testing.B) { wg.Wait() } } + +func BenchmarkPond(b *testing.B) { + pool := pond.New(PoolSize, 0, pond.MinWorkers(PoolSize)) + + taskFunc := func() { + time.Sleep(10 * time.Millisecond) + } + + b.ResetTimer() + for i := 0; i < b.N; i++ { + for i := 0; i < TaskNum; i++ { + pool.Submit(taskFunc) + } + pool.StopAndWait() + } +} + +func BenchmarkAnts(b *testing.B) { + var wg sync.WaitGroup + p, _ := ants.NewPool(PoolSize) + defer p.Release() + + taskFunc := func() { + time.Sleep(10 * time.Millisecond) + wg.Done() + } + + b.ResetTimer() + for i := 0; i < b.N; i++ { + for i := 0; i < TaskNum; i++ { + wg.Add(1) + _ = p.Submit(taskFunc) + } + wg.Wait() + } +}