This processor will deduplicate logs based on their log body content.
The processor relies on Bloom Filters to quickly check duplication.
goos: darwin
goarch: arm64
cpu: Apple M4 Pro
BenchmarkBloomFilter
BenchmarkBloomFilter/benchmark-count=10-dupes=10
BenchmarkBloomFilter/benchmark-count=10-dupes=10-12 6052 193368 ns/op
BenchmarkBloomFilter/benchmark-count=10-dupes=100
BenchmarkBloomFilter/benchmark-count=10-dupes=100-12 4854 246983 ns/op
BenchmarkBloomFilter/benchmark-count=10-dupes=1000
BenchmarkBloomFilter/benchmark-count=10-dupes=1000-12 877 1288192 ns/op
BenchmarkBloomFilter/benchmark-count=100-dupes=2
BenchmarkBloomFilter/benchmark-count=100-dupes=2-12 5498 225383 ns/op
BenchmarkBloomFilter/benchmark-count=100-dupes=5
BenchmarkBloomFilter/benchmark-count=100-dupes=5-12 5803 215501 ns/op
BenchmarkBloomFilter/benchmark-count=100-dupes=10
BenchmarkBloomFilter/benchmark-count=100-dupes=10-12 4696 235548 ns/op
BenchmarkBloomFilter/benchmark-count=100000-dupes=2
BenchmarkBloomFilter/benchmark-count=100000-dupes=2-12 34 31368435 ns/op
BenchmarkBloomFilter/benchmark-count=100000-dupes=5
BenchmarkBloomFilter/benchmark-count=100000-dupes=5-12 18 74975183 ns/op
BenchmarkBloomFilter/benchmark-count=100000-dupes=10
BenchmarkBloomFilter/benchmark-count=100000-dupes=10-12 7 144512369 ns/op
BenchmarkBloomFilter/benchmark-count=1000000-dupes=2
BenchmarkBloomFilter/benchmark-count=1000000-dupes=2-12 4 297750573 ns/op
BenchmarkBloomFilter/benchmark-count=1000000-dupes=5
BenchmarkBloomFilter/benchmark-count=1000000-dupes=5-12 2 637263875 ns/op
BenchmarkBloomFilter/benchmark-count=1000000-dupes=10
BenchmarkBloomFilter/benchmark-count=1000000-dupes=10-12 1 1525182083 ns/op
BenchmarkBloomFilterCap10000000Dup5
BenchmarkBloomFilterCap10000000Dup5-12 18 73249340 ns/op
BenchmarkBloomFilterCap100Dup100
BenchmarkBloomFilterCap100Dup100-12 859 1394728 ns/op
PASS
ok github.com/atoulme/bloomfilterprocessor 22.461s
PASS
ok github.com/atoulme/bloomfilterprocessor/internal/metadata 0.191s
PASS
ok github.com/atoulme/bloomfilterprocessor/internal/metadatatest 0.173s
Process finished with the exit code 0