unique: Handle memory leaks; cleanup goroutine can't keep up

@mknyszek

Go version

go version go1.24.0 darwin/arm64
go version go1.24.0 linux/amd64
go version go1.23.6 darwin/arm64
go version go1.23.6 linux/amd64

What did you do?

We have a server that processes a stream of user connect & disconnect events from our edge servers. Users are identified by a [32]byte public key, wrapped up like:

type NodePublic struct {
   k [32]byte
}

This server maintains ~4 maps keyed by NodePublic. Out of curiosity and as an experiment, we decided to kick the tires with unique.Handle to make those maps instead be keyed by unique.Handle[NodePublic] to make them 8 bytes instead of 32. It saves about 96 bytes per connection. Not a ton, but ... an experiment.

The experiment didn't go as expected.

What did you see happen?

What we saw happen was unbounded memory growth and kernel OOMs. These are the two instances of this server we run. (it's an internal debug tool only, so we only run two for HA reasons)

It previously took about ~1GB of memory at steady state. With unique.Handle, it just keeps growing until the kernel OOM kills it.

The flat parts in the graph are where we changed the map key types from unique.Handle[NodePublic] back to just NodePublic, changing nothing else.

To debug, I hacked up the unique package (patch: https://gist.github.com/bradfitz/06f57b1515d58fb4d8b9b4eb5a84262e) to try out two things:

see if the weak pointers were getting tinyalloc'ed and preventing GC. To counter this, I tried making the unique package use 24 byte value types in its uniqueMap. No change.
to log stats when the cleaners ran to see how much it tried to delete
Oh, and I also tried both Go 1.23.6 and Go 1.24.0

The result is that never frees anything. The weak pointer is never nil. It takes ever longer to iterate over the map (up to 2.7 seconds now), never finding anything to delete:

Feb 16 00:30:44 derptrackd2 derptrackd[771925]: XXX did unique cleanup loop: items= 2 ; deleted= 0 ; kept= 2 ; took  16µs
Feb 16 00:30:46 derptrackd2 derptrackd[771925]: XXX did unique cleanup loop: items= 2829766 ; deleted= 0 ; kept= 2829766 ; took  2.585244s
Feb 16 00:32:51 derptrackd2 derptrackd[771925]: XXX did unique cleanup loop: items= 2 ; deleted= 0 ; kept= 2 ; took  14µs
Feb 16 00:32:53 derptrackd2 derptrackd[771925]: XXX did unique cleanup loop: items= 2850211 ; deleted= 0 ; kept= 2850211 ; took  2.692595s
Feb 16 00:34:58 derptrackd2 derptrackd[771925]: XXX did unique cleanup loop: items= 2 ; deleted= 0 ; kept= 2 ; took  19µs
Feb 16 00:35:01 derptrackd2 derptrackd[771925]: XXX did unique cleanup loop: items= 2870410 ; deleted= 0 ; kept= 2870410 ; took  2.728681s
Feb 16 00:37:06 derptrackd2 derptrackd[771925]: XXX did unique cleanup loop: items= 2 ; deleted= 0 ; kept= 2 ; took  19µs
Feb 16 00:37:08 derptrackd2 derptrackd[771925]: XXX did unique cleanup loop: items= 2890817 ; deleted= 0 ; kept= 2890817 ; took  2.635071s
Feb 16 00:39:13 derptrackd2 derptrackd[771925]: XXX did unique cleanup loop: items= 2 ; deleted= 0 ; kept= 2 ; took  18µs
Feb 16 00:39:16 derptrackd2 derptrackd[771925]: XXX did unique cleanup loop: items= 2911098 ; deleted= 0 ; kept= 2911098 ; took  2.600478s
Feb 16 00:41:20 derptrackd2 derptrackd[771925]: XXX did unique cleanup loop: items= 2 ; deleted= 0 ; kept= 2 ; took  19µs
Feb 16 00:41:22 derptrackd2 derptrackd[771925]: XXX did unique cleanup loop: items= 2931399 ; deleted= 0 ; kept= 2931399 ; took  2.606853s
Feb 16 00:43:27 derptrackd2 derptrackd[771925]: XXX did unique cleanup loop: items= 2 ; deleted= 0 ; kept= 2 ; took  20µs
Feb 16 00:43:29 derptrackd2 derptrackd[771925]: XXX did unique cleanup loop: items= 2951574 ; deleted= 0 ; kept= 2951574 ; took  2.701304s

Looking at historical logs, it never deletes anything on Linux.

I tried to write a repro (on my Mac) and I failed, despite a bunch of efforts.

I ended up doing too much debugging on a remote AWS VM, with a hacked up Go toolchain.

Eventually I ran the same code on my Mac (same server + same hacked up Go toolchain), pulling from the same data sources, and surprisingly: it worked! unique finds weak things to delete in its cleaner and it doesn't grow.

In summary:

unique leaks on linux/amd64 (prod cloud VM)
same code on same data doesn't leak on darwin/arm64 (my dev laptop)

I'll now try revisiting my standalone repro but running it on Linux instead of macOS.

I had hoped to deliver @mknyszek a bug on a silver platter before filing, but I'm going a little crazy at this pointed and figured it was time to start sharing.

What did you expect to see?

No memory leaks on Linux also.

Related Issues

unique: Fatal errors (found bad pointer in Go heap, found pointer to free object) and memory corruption #69643 (closed)
unique: large string still referenced, after interning only a small substring #69370 (closed)
unique: fatal error: found pointer to free object #69210 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

Actually, I'm going to close this for now until there's something actionable. I'll reopen if/when I have a repro.

up to 2.7 seconds now

Tangentially related, but we should probably remove the loop and switch to runtime.AddCleanup. I don't think that'll fix this issue though.

I played with hacking away at the server, trying to minimize the real code down to minimal repro instead of building up a minimal repro from scratch, with the goal to eventually plug in a in-memory generated random data source rather than reading from the network.

That seemed to have worked.... at a certain point, Linux started reclaiming a small amount of memory on a test Linux box I was running. And over night it did one huge unique.Handle reclaim during one of its GCs, but zero for all the others throughout the night.

Now I need to binary search my deletions and see what the key deletion was.

In any case, it still reclaims super slowly on Linux even after my minimization.

Will report back when I have more.

@mknyszek

Okay, I think I've narrowed it down and figured out the macOS-vs-Linux difference: this is about the number goroutines producing new unique.Handles overwhelming the sole cleanup goroutine. My Mac laptop (M4 Max) is just significantly beefier than my dinky Linux VMs (both prod & dev).

Now I understand why my ground-up repro didn't work before: it was using a single goroutine.

I have a standalone repro now from the full server that doesn't depend on the network (using 100 goroutines adding+removing keys) but now that I think I know what's happening, I think I'll switch back to my bottom-up repro, slapping on some goroutines.

So @mknyszek, maybe runtime.AddCleanup is indeed the answer here.

Here ya go: https://go.dev/play/p/rrNRBh6g-WF

Ignore that that's a playground link and also that it's written as a test function. Just run it locally (on either macOS or Linux) and watch it leak.

Running WithoutUnique, it doesn't grow:

% GODEBUG=gctrace=1 go test -v -run=WithoutUnique
...
gc 35 @27.366s 0%: 0.13+54+0.10 ms clock, 2.2+1.7/134/0+1.7 ms cpu, 3589->3667->1814 MB, 3715 MB goal, 0 MB stacks, 0 MB globals, 16 P
    uniqueleak_test.go:124: # items = 1500000; 3660669760 HeapAlloc, 2440.45 per item
gc 36 @28.590s 0%: 0.20+31+0.10 ms clock, 3.2+3.3/121/0+1.7 ms cpu, 3501->3544->1780 MB, 3629 MB goal, 0 MB stacks, 0 MB globals, 16 P
    uniqueleak_test.go:124: # items = 1499999; 3702783968 HeapAlloc, 2468.52 per item
gc 37 @29.773s 0%: 0.12+36+0.038 ms clock, 1.9+2.8/135/0+0.61 ms cpu, 3433->3483->1790 MB, 3562 MB goal, 0 MB stacks, 0 MB globals, 16 P
    uniqueleak_test.go:124: # items = 1499999; 3750955328 HeapAlloc, 2500.64 per item
    uniqueleak_test.go:124: # items = 1499984; 3678439048 HeapAlloc, 2452.32 per item
gc 38 @30.970s 0%: 0.18+50+0.11 ms clock, 2.8+3.2/128/0+1.8 ms cpu, 3451->3520->1810 MB, 3581 MB goal, 0 MB stacks, 0 MB globals, 16 P
    uniqueleak_test.go:124: # items = 1499999; 3622019400 HeapAlloc, 2414.68 per item
gc 39 @32.194s 0%: 0.21+52+0.17 ms clock, 3.4+2.1/144/0+2.8 ms cpu, 3491->3563->1816 MB, 3622 MB goal, 0 MB stacks, 0 MB globals, 16 P
    uniqueleak_test.go:124: # items = 1500000; 3665755200 HeapAlloc, 2443.84 per item
gc 40 @33.458s 0%: 0.10+64+0.060 ms clock, 1.6+3.2/128/0+0.97 ms cpu, 3551->3642->1837 MB, 3633 MB goal, 0 MB stacks, 0 MB globals, 16 P
    uniqueleak_test.go:124: # items = 1500000; 3755686688 HeapAlloc, 2503.79 per item
gc 41 @34.742s 0%: 0.20+52+0.039 ms clock, 3.2+3.0/135/0+0.62 ms cpu, 3582->3655->1821 MB, 3675 MB goal, 0 MB stacks, 0 MB globals, 16 P
    uniqueleak_test.go:124: # items = 1500000; 3832429984 HeapAlloc, 2554.95 per item

But using unique, it leaks (with my hacked up unique package doing extra logging, as linked above)

% GODEBUG=gctrace=1 go test -v -run=WithUnique
gc 20 @17.533s 5%: 0.16+1023+0.010 ms clock, 2.6+501/4094/8.2+0.16 ms cpu, 6973->8495->4847 MB, 8189 MB goal, 0 MB stacks, 0 MB globals, 16 P
    uniqueleak_test.go:124: # items = 1499986; 8882974072 HeapAlloc, 5922.04 per item
    uniqueleak_test.go:124: # items = 1499999; 8880822872 HeapAlloc, 5920.55 per item
    uniqueleak_test.go:124: # items = 1499996; 8541901672 HeapAlloc, 5694.62 per item
    uniqueleak_test.go:124: # items = 1499983; 8328100968 HeapAlloc, 5552.13 per item
    uniqueleak_test.go:124: # items = 1499986; 9557681720 HeapAlloc, 6371.85 per item
gc 21 @22.366s 5%: 0.16+1267+0.010 ms clock, 2.5+802/5066/14+0.17 ms cpu, 8255->10094->5671 MB, 9694 MB goal, 0 MB stacks, 0 MB globals, 16 P
    uniqueleak_test.go:124: # items = 1499980; 10552296616 HeapAlloc, 7034.96 per item
    uniqueleak_test.go:124: # items = 1499996; 10549919064 HeapAlloc, 7033.30 per item
    uniqueleak_test.go:124: # items = 1499987; 10354593096 HeapAlloc, 6903.12 per item
    uniqueleak_test.go:124: # items = 1499997; 9738898648 HeapAlloc, 6492.61 per item
    uniqueleak_test.go:124: # items = 1499993; 10020571944 HeapAlloc, 6680.41 per item
    uniqueleak_test.go:124: # items = 1499984; 11420736632 HeapAlloc, 7613.91 per item
XXX did unique cleanup loop:  7469606 items 3946215 deleted 3523391 kept; took  26.446266s
XXX starting unique cleanup loop
gc 22 @28.087s 6%: 0.14+1478+0.010 ms clock, 2.3+1673/5913/9.8+0.17 ms cpu, 9659->11751->6463 MB, 11343 MB goal, 0 MB stacks, 0 MB globals, 16 P
    uniqueleak_test.go:124: # items = 1499988; 12283776280 HeapAlloc, 8189.25 per item
    uniqueleak_test.go:124: # items = 1499997; 12282524312 HeapAlloc, 8188.37 per item
    uniqueleak_test.go:124: # items = 1499818; 12138670280 HeapAlloc, 8093.43 per item
    uniqueleak_test.go:124: # items = 1499987; 11612830136 HeapAlloc, 7741.95 per item
    uniqueleak_test.go:124: # items = 1499982; 10844047640 HeapAlloc, 7229.45 per item
    uniqueleak_test.go:124: # items = 1499975; 11746102088 HeapAlloc, 7830.87 per item
    uniqueleak_test.go:124: # items = 1499997; 13196850496 HeapAlloc, 8797.92 per item
gc 23 @34.794s 6%: 0.11+1706+0.013 ms clock, 1.8+2438/6823/4.2+0.20 ms cpu, 11008->13350->7317 MB, 12927 MB goal, 0 MB stacks, 0 MB globals, 16 P
    uniqueleak_test.go:124: # items = 1499989; 13948964920 HeapAlloc, 9299.38 per item
    uniqueleak_test.go:124: # items = 1499989; 13947515720 HeapAlloc, 9298.41 per item
    uniqueleak_test.go:124: # items = 1499946; 13924360552 HeapAlloc, 9283.24 per item
    uniqueleak_test.go:124: # items = 1499997; 13429541096 HeapAlloc, 8953.05 per item
    uniqueleak_test.go:124: # items = 1499987; 12570533544 HeapAlloc, 8380.43 per item
    uniqueleak_test.go:124: # items = 1499985; 12730948488 HeapAlloc, 8487.38 per item
    uniqueleak_test.go:124: # items = 1499980; 13995042328 HeapAlloc, 9330.15 per item
    uniqueleak_test.go:124: # items = 1499991; 15426806832 HeapAlloc, 10284.60 per item
gc 24 @42.260s 7%: 0.14+1943+0.013 ms clock, 2.2+3785/7772/4.1+0.21 ms cpu, 12462->15056->8208 MB, 14635 MB goal, 0 MB stacks, 0 MB globals, 16 P
    uniqueleak_test.go:124: # items = 1499988; 15727126728 HeapAlloc, 10484.84 per item
    uniqueleak_test.go:124: # items = 1499995; 15726441208 HeapAlloc, 10484.33 per item
    uniqueleak_test.go:124: # items = 1499995; 15724350536 HeapAlloc, 10482.94 per item

Note that there are multiples GCs happening while the unique cleanup func is still running.

Yeah, AddCleanup seems to do the trick. But it's not enough by itself. It also needs a bit of back pressure. Here's a trashy fix:

https://gist.github.com/bradfitz/a136006f112fee47ce1d87232e779d24

(I've signed the CLA; consider that trashy patch safe to look at 😅)

A potential fix for Go 1.23 (without using Go 1.24's AddCleanup) is https://gist.github.com/bradfitz/901f14f9c6dfd5c9344673a2c9478932 , making new handles wait for any ongoing cleanup to finish, but it's a bit too aggressive: it makes all goroutines wait.

But it works. 🤷‍♂

@bradfitz

Hm... We should make the unique cleanup more efficient by using runtime.AddCleanup (we can use finalizers in Go 1.23, maybe), but (just musing here) the fact that that's not enough and it still needs some kind of back-pressure mechanism makes me wonder if maybe any kind of canonicalization map just isn't worth it for this workload?

In other words, if there are so many new entries being created that deletion can't keep up, that suggests there isn't really all that much deduplication happening; that there are more inserts than lookups.

... But also maybe I'm wrong about that. Maybe there's enough of a benefit even if you only get, for example, 1 lookup on average after an insert, before a deletion. I'd love to understand this better if you can collect any metrics from production in an ad-hoc way, @bradfitz. And perhaps we need diagnostics (runtime/metrics, probably) to help people understand whether unique is actually working for them.

Off the top of my head, I'm not sure what the right back-pressure policy would be here. An alternative would be is that we could scale up deletions instead. Deletions on independent parts of the tree (especially for a big tree) should scale well in my testing. The runtime.AddCleanup cleanup function could put work on a queue instead of doing the deletion, and a pool of goroutines could consume this queue instead of just a single goroutine. We can scale this pool according to the rate of insertions (or even just always have enough goroutines on-hand to handle the load). There's a sort of natural back-pressure built-in as you start spending CPU time on these background cleaners, and the scheduler handles distribution of resources.

@bradfitz

Thank you for the thorough investigation @bradfitz!!

I have a hunch why there was still an apparent memory leak. In my haste, my patches removed the deletion of interior nodes in the canonicalization map when deleting entries. It's possible the memory growth is actually just the result of the map getting bigger over time. It occurs to me that lots of unique hashes will keep growing the map in new ways, so interior node deletion is vital.

I've updated https://go.dev/cl/650256 to avoid the potential problem that I mentioned, so https://go.dev/cl/650697 at PS 16 (please do not use PS 17+, too many other things also change) will contain the fix. If you're still up for trying it out, that would be super helpful. (Sorry for the long delay; this only occurred to me today...)

unique: use a bespoke canonicalization map and runtime.AddCleanup

runtime: schedule cleanups across multiple goroutines

@bradfitz

Does this remain open to wait for @bradfitz to reply, or is there anything else that we know should be done?

Hello - I think we've run into the same issue in go 1.25 - was this meant to be fixed already or should we be waiting to 1.26?

I think you're going to need to provide more details.

I did try to fix it, but as of #71772 (comment), the fix was never validated, and I don't have another (useful, real-world) reproducer at the moment. (It's still possible to out-run the cleanup goroutines if you really try, but it should be much harder to do it than it used to be. The situation with unique specifically should be orders of magnitude better.) I think I'm going to need more data to keep iterating on this.

(For example, it's possible that the reason this isn't fixed is because in my fix I was conservative with how many cleanup goroutines we're willing to spin up, so that might be something to try. Change gomaxprocs/4 to gomaxprocs here if you want to try something: https://cs.opensource.google/go/go/+/master:src/runtime/mcleanup.go;l=709;drc=eb63ef9d6676dc0edb112cd297820306a327017a)

Hi

our usecase is an app that processes a lot of spans and traces, the IDs of which are relatively short-lived (maybe 30 seconds) but appear many times during that window. I figured I could limit memory usage by using unique.

I've created what I think is a demo of this below - you might need to tweak it to become aware of exactly how well the unique map is performing (and to be sure that GC is triggering as you expect) - you can use the useUnique flag at the top to observe memory usage with/without unique turned on. I think this demos a leak but I'm fairly new to golang, haven't yet delved into the joys of garbage collection so I'm not certain - our main apps definitely OOM.

On my machine, with unique turned on the final memory stats look something like this

[81] Time: 81.65s | Goroutines: 2
  Alloc: 102 MB | TotalAlloc: 1942 MB | Sys: 164 MB
  HeapAlloc: 102 MB | HeapSys: 117 MB | HeapObjects: 2576846
  NumGC: 3774 | PauseTotalNs: 362 ms

and with unique turned off look like this (note the overall execution is ~2.5x quicker, I guess some of that's to be expected) - the GC does seem to be executing after the map is empty based on the numGC stat

[30] Time: 30.38s | Goroutines: 2
  Alloc: 12 MB | TotalAlloc: 1772 MB | Sys: 53 MB
  HeapAlloc: 12 MB | HeapSys: 24 MB | HeapObjects: 249954
  NumGC: 240 | PauseTotalNs: 28 ms

package main

import (
	"crypto/rand"
	"fmt"
	"math/big"
	"runtime"
	"runtime/debug"
	"sync"
	"time"
	"unique"
)

var spanMap sync.Map

// Toggle this to test with and without unique.Handle wrapping the key
var useUnique = true

func main() {
	// Set memory limit to 50MB
	memLimit := int64(50 * 1024 * 1024) // 50 MiB
	oldLimit := debug.SetMemoryLimit(memLimit)
	fmt.Printf("Memory limit set to: %d MB (previous: %d MB)\n",
		memLimit/1024/1024, oldLimit/1024/1024)

	// Start memory monitoring
	done := make(chan bool)
	go monitorMemory(done, 1*time.Second)

	// Print memory usage estimate
	numSpans := 1_000_000
	spawnInterval := 10 * time.Microsecond
	printMemoryEstimate(numSpans, spawnInterval)

	// Run the span processing
	processSpans(numSpans, spawnInterval)

	count := 0
	spanMap.Range(func(key, value any) bool {
		count++
		return true
	})
	//check the map is empty at the end
	fmt.Printf("Remaining spans in map: %d\n", count)

	time.Sleep(10 * time.Second) // Give all gc a chance to run and monitor to print stats
	// Stop monitoring
	done <- true
	time.Sleep(100 * time.Millisecond) // Give monitor time to print final stats
}

func printMemoryEstimate(numSpans int, spawnInterval time.Duration) {
	// Each span lives for approximately 200ms (100ms + 100ms sleep)
	spanLifetime := 200 * time.Millisecond

	// Calculate max concurrent spans at steady state
	// = spanLifetime / spawnInterval
	maxConcurrentSpans := int(spanLifetime / spawnInterval)

	// Memory per span in map:
	// - SpanId: 8 bytes (the key, wrapped in unique.Handle)
	// - String value: 16 bytes + string header (~24 bytes overhead) = ~40 bytes
	// - unique.Handle overhead: ~16 bytes (pointer + internal structure)
	// - sync.Map overhead: ~48 bytes per entry
	// Total per span: ~112 bytes (conservative estimate)
	bytesPerSpan := 112

	estimatedMemory := maxConcurrentSpans * bytesPerSpan

	fmt.Println("\n=== MEMORY USAGE ESTIMATE ===")
	fmt.Printf("Total spans to process: %d\n", numSpans)
	fmt.Printf("Spawn interval: %v\n", spawnInterval)
	fmt.Printf("Span lifetime: %v\n", spanLifetime)
	fmt.Printf("Max concurrent spans (steady state): %d\n", maxConcurrentSpans)
	fmt.Printf("Estimated bytes per span entry: %d\n", bytesPerSpan)
	fmt.Printf("Estimated peak memory: %.2f MB\n", float64(estimatedMemory)/1024/1024)
	fmt.Printf("Expected: Memory should stabilize around %.2f MB\n", float64(estimatedMemory)/1024/1024)
	fmt.Println("If memory grows beyond this, there's likely a leak!")
	fmt.Println("=============================\n")
}

// monitorMemory periodically prints memory usage stats until signaled to stop
func monitorMemory(done chan bool, interval time.Duration) {
	var m runtime.MemStats
	ticker := time.NewTicker(interval)
	defer ticker.Stop()

	startTime := time.Now()
	iteration := 0

	for {
		select {
		case <-done:
			// Print final stats
			runtime.ReadMemStats(&m)
			fmt.Printf("\n=== FINAL MEMORY STATS ===\n")
			printMemStats(&m, iteration, startTime)
			return
		case <-ticker.C:
			runtime.ReadMemStats(&m)
			iteration++
			printMemStats(&m, iteration, startTime)
		}
	}
}

// printMemStats prints the current memory usage stats along with the number of goroutines and elapsed time since the start of monitoring. It formats the output to show key memory metrics in megabytes and includes the number of garbage collection cycles and total GC pause time.
func printMemStats(m *runtime.MemStats, iteration int, startTime time.Time) {
	elapsed := time.Since(startTime)
	fmt.Printf("\n[%d] Time: %.2fs | Goroutines: %d\n",
		iteration, elapsed.Seconds(), runtime.NumGoroutine())
	fmt.Printf("  Alloc: %d MB | TotalAlloc: %d MB | Sys: %d MB\n",
		m.Alloc/1024/1024, m.TotalAlloc/1024/1024, m.Sys/1024/1024)
	fmt.Printf("  HeapAlloc: %d MB | HeapSys: %d MB | HeapObjects: %d\n",
		m.HeapAlloc/1024/1024, m.HeapSys/1024/1024, m.HeapObjects)
	fmt.Printf("  NumGC: %d | PauseTotalNs: %d ms\n",
		m.NumGC, m.PauseTotalNs/1000000)
}

// processSpans simulates processing a number of spans with a specified interval between starting each span. Each span is processed in its own goroutine, and the function waits for all spans to complete before returning.
func processSpans(numSpans int, waitBetweenStarts time.Duration) {
	var wg sync.WaitGroup
	for i := 0; i < numSpans; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			processSpan()
		}()
		time.Sleep(waitBetweenStarts)
	}
	wg.Wait()
}

// processSpan simulates the lifecycle of a span by generating a random span ID, storing it in a sync.Map with a random string value, sleeping for a short duration, updating the value, sleeping again, and finally deleting the entry from the map. The key can be wrapped in unique.Handle based on the useUnique flag.
func processSpan() {
	spanId := generateRandomSpanId()
	var handle any
	if useUnique {
		handle = unique.Make(spanId)
	} else {
		handle = spanId
	}
	spanMap.Store(handle, getRandomString())
	time.Sleep(100 * time.Millisecond)
	spanMap.Store(handle, getRandomString())
	time.Sleep(100 * time.Millisecond)
	spanMap.Delete(handle)
}

// getRandomString generates a random string of 16 characters using a predefined character set. It uses crypto/rand for secure random number generation and returns the generated string.
func getRandomString() string {
	const charset = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
	b := make([]byte, 16)
	for i := range b {
		randomIndex, err := rand.Int(rand.Reader, big.NewInt(int64(len(charset))))
		if err != nil {
			panic(err)
		}
		b[i] = charset[randomIndex.Int64()]
	}
	return string(b)
}

// generateRandomSpanId creates a random 8-byte array to be used as a span ID. It uses crypto/rand to fill the byte array with random data and returns it. If there is an error during random generation, it panics.
func generateRandomSpanId() [8]byte {
	var spanID [8]byte
	_, err := rand.Read(spanID[:])
	if err != nil {
		panic(err)
	}
	return spanID
}

ok I found something interesting - on a hunch I changed generateRandomSpanId to this

// generateRandomSpanId creates a random 8-byte span ID and returns it as a hex-encoded string. It uses crypto/rand to fill the byte array with random data and returns it. If there is an error during random generation, it panics.
func generateRandomSpanId() string {
	var spanID [8]byte
	_, err := rand.Read(spanID[:])
	if err != nil {
		panic(err)
	}
	return hex.EncodeToString(spanID[:])
}

and ran it again, this time the performance looks like this

unique off :

[30] Time: 30.56s | Goroutines: 2
  Alloc: 8 MB | TotalAlloc: 1795 MB | Sys: 53 MB
  HeapAlloc: 8 MB | HeapSys: 24 MB | HeapObjects: 98159
  NumGC: 246 | PauseTotalNs: 29 ms

unique on:

[33] Time: 33.96s | Goroutines: 2
  Alloc: 14 MB | TotalAlloc: 1964 MB | Sys: 54 MB
  HeapAlloc: 14 MB | HeapSys: 26 MB | HeapObjects: 324405
  NumGC: 494 | PauseTotalNs: 48 ms

compared to using an array of bytes for the span ID :

[81] Time: 81.65s | Goroutines: 2
  Alloc: 102 MB | TotalAlloc: 1942 MB | Sys: 164 MB
  HeapAlloc: 102 MB | HeapSys: 117 MB | HeapObjects: 2576846
  NumGC: 3774 | PauseTotalNs: 362 ms

so maybe this issue isn't related to the above issue at all, but is instead due to something around using unique with a byte array instead of a string?

Navigation Menu

Repository navigation

unique: Handle memory leaks; cleanup goroutine can't keep up #71772

Go version

What did you do?

What did you see happen?

What did you expect to see?

17 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

unique: Handle memory leaks; cleanup goroutine can't keep up #71772

Description

Go version

What did you do?

What did you see happen?

What did you expect to see?

Activity

gabyhelp commented on Feb 16, 2025

bradfitz commented on Feb 16, 2025

mknyszek commented on Feb 16, 2025

bradfitz commented on Feb 16, 2025

bradfitz commented on Feb 16, 2025

bradfitz commented on Feb 16, 2025

bradfitz commented on Feb 17, 2025

bradfitz commented on Feb 17, 2025

mknyszek commented on Feb 17, 2025

mknyszek commented on Feb 17, 2025

17 remaining items

mknyszek commented on Apr 23, 2025

mknyszek commented on Apr 23, 2025

mvdan commented on Aug 13, 2025

hughlunnon commented on Feb 2, 2026

mknyszek commented on Feb 2, 2026

hughlunnon commented on Feb 11, 2026

hughlunnon commented on Feb 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions