Repository navigation
cmd/compile: removal of duffcopy reduced visibility of memory copying cost in profiles #81392
Description
We recently switched from Go 1.25 to 1.26. This new version included a change which removed duffcopy in favour of generated loops (see CL 696016 for example on amd64).
This came with an unfortunate side effect. It was previously possible to easily measure the CPU time spent copying structs (by looking at the time spent in duffcopy in profiles); this is not possible anymore (as struct copies don't happen in a separate function).
The CPU time spent copying structs is now attributed to the line that requires copying (depending on the exact code: a function call with a value receiver, a variable assignment, etc.)
The reason I think this is worth reporting is that we were able until recently to get significant CPU footprint wins by monitoring the time spent in duffcopy and eliminating redundant copies where safe; we do not really have a way to do that anymore.
As far as I can tell, the only way to know currently would be to inspect each "hot line" individually and check whether its "hotness" may be attributed to copies. This is cumbersome and also less accurate than having CPU time properly attributed to copying in a dedicated frame.
The compiler/optimiser already has a diagnostic that helps finding those issues (reported through LogLargeCopyValue defined here) but it only partially helps, as it only reports when large struct copies are detected, and thus:
- Doesn't give an indication on how often those large copies happen in practice (if they rarely do, it might not be a problem);
- Cannot detect the smaller copies that happen on the hot path, and that can in practice incur just as much overhead as less frequent bigger copies.
Activity
gopherbot commentedon Sep 8, 2026
Change https://go.dev/cl/829146 mentions this issue:
cmd/compile: add frames for memory copying and zeroinggabyhelp commentedon Sep 8, 2026
Related Issues
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
randall77 commentedon Sep 8, 2026
I'm not sure I agree that this is a bug. If anything, the existence of
duffcopyin a profile was the bug.It was already the case that many struct->struct copies did not use
duffcopy. On amd64, for example, small (<=64 bytes) and very large (> 1024 bytes) copies did not get a duffcopy. The non-existence ofduffcopyis not a reliable indicator of no copying. (Although I agree, the existence ofduffcopydoes indicate copy cost.)I don't think introducing inlining marks is the answer. There is no function call in the user's source code at that location. We shouldn't be inventing callees in the user's code. We've been moving the other direction, making profiling and stack traces correspond more faithfully to the user's program (for instance, suppressing wrappers and other compiler/runtime introduced calls).
Profiles will still show hot marks on places that use a lot of CPU time. Sure, it is less obviously a big copy, but it still points to the exact line that is using CPU.
Jorropo commentedon Sep 8, 2026
You should take a look at
perf statand the counters it exposes.prattmic commentedon Sep 8, 2026
I have frequently found myself wanting some kind of "tagging" on different kinds of operations like this. For example, mapaccess returns the address of the element and the caller is responsible for dereferencing to get the actual value. I've found myself wanting to measure those loads in the callers as part of the "map cost", but they are hard to pinpoint.
But I agree with @randall77 that fake stack frames are not the answer here. For the map problem, I've wondered about marking PC ranges with various DWARF metadata that tooling could them group by.
felixge commentedon Sep 9, 2026
Thanks for the feedback. I'd like to invite @KowalskiThomas to continue the discussion at tomorrow's runtime diagnostics sync. Maybe you could join as well, @randall77? No worries if not.
This report should be seen as a problem statement. We're not trying to imply that the new behavior is a bug. The fake stack frames are just an idea that we're not married to. But we recently used duffcopy to locate and save almost seven figures in annual compute costs, so it'd be nice to find a new way to observe these copies. The (MC)2 paper suggests that 5% of Google's datacenter CPU goes into memory copy operations as well (probably not all struct copies tho).
We'll take a look. Do you have any particular counters in mind? That being said, even if perf can do the job, it might still be nice if Go's built-in diagnostics could do it as well.
JunyangShao commentedon Sep 11, 2026
@golang/compiler
KowalskiThomas commentedon Sep 23, 2026
Following what was discussed during the performance and diagnostics meeting, I tried to look at whether finalising support for column numbers in the profiler would help. I think in a way it could (assuming some conventions are established, such as attributing CPU time to the
:=in e.g.a := myMap[key]to indicate time spent copying memory, as opposed to time looking upkeyinmyMap)... but to be honest I still find it inferior to theduffcopyframes in terms of user experience.The problem is that the profile on its own no longer conveys what the time is spent doing, and telling whether it's memory copies requires correlating line/column numbers from the profile with the actual code.
An alternative approach I've thought of since then would be to use pprof labels to indicate whether a stack sample was captured while copying memory. With labels, the stacks reported in profiles would still be correct (they would not include a frame for a function that doesn't exist), but it would be possible with the profile alone to tell how much time was spent copying memory (including in the context of a certain stack/function).
I made a proof of concept to validate the idea. The way I did it was by adding a new PC data table mapping PC ranges to "inlined work type labels" (
None/Copy/Zero). When sampling, the profiler checks the PC against that table and pushes the right pprof label if the "inlined work type" for the current PC isn'tNone.Does that solution sound more acceptable?
randall77 commentedon Sep 23, 2026
I'm lukewarm to the idea. I worry about the size of this table, although it probably wouldn't be huge. But there are probably very few people who care about it. Maybe hidden behind a GOEXPERIMENT it would be ok?
As we discussed in the meeting, I'd rather keep this additional information in DWARF and post-process to get it. That way it can be thrown away with
-ldflags=-wby people who care about non-code size, and it doesn't cost anything at sampling time.Part of a good design is that it could be used for other things than just memcopy tracking. What else could we use this for? I was thinking about other states and "InWriteBarrier" came to mind as potentially interesting. Any other ideas?
prattmic commentedon Sep 24, 2026
I've long wondered how much time is spent on spills (both spilling and reloading the values). But this feels like the kind of thing that is difficult to measure, even with the metadata, because I suspect that a big chunk of the "cost" you'd measure on spills would actually just move around to other instructions if the spills were eliminated.
Jorropo commentedon Sep 24, 2026
I am not sure what is the value of getting pprof data for whole class of stuff that is everywhere (
Move,Zero, spills / restores).Assuming it's not a hot loop a sampling profiler has an uncertainty window that is much bigger than a single instruction.
Something like a stack Zero can often dispatch very early, and is unlikely to be caught by pprof, even if it has real cost however that cost shows up in other instructions who were slowed down by it.
felixge commentedon Sep 24, 2026
@randall77 the issue with DWARF is that it's not a good fit for our use case of continuous profiling. We just collect pprofs, we don't get the DWARF section of the binary. Arguably the Go runtime could try to access the DWARF info when building the pprofs, but I don't imagine this to be palatable in terms of performance and complexity. Or we could upload DWARF, but that may make some users uncomfortable and needs a lot of bandwidth.
Regarding write barriers: CPU profiles already contain
runtime.gcWriteBarrierstack frames (but it's not documented in the GC Guide)@Jorropo the motivation is cost savings. We were able to eliminate a high six figure amount of annual compute spend from our bill by eliminating struct copies from our code (discovered via duffcopy). But ACK on the skid issues with single instructions. IIRC we think it wouldn't be a major issue for struct copies, but it's gonna be a deal breaker for other types of use cases.
prattmic commentedon Sep 24, 2026
Indeed, I imagine using this data with precise profiling (Linux perf + PEBS), which also votes in favor of using DWARF vs internal Go metadata.
prattmic commentedon Sep 24, 2026
Further discussion today in #57175 (comment), where a lot of the discussion focused on what metadata we can/should include in pprof profiles to link them back to the exact binary they are from (
go version -m?)