Repository navigation
cmd/compile: omit or normalize line numbers in export data (.x) for incremental build pruning #81954
Description
Background & Motivation
Build systems that split cmd/compile outputs into a small export data file (-o pkg.x, containing __.PKGDEF) for downstream compile actions and a full linker object (-linkobj pkg.a, containing _go_.o) for link actions can skip recompiling reverse dependencies when pkg.x is unchanged.
However, because Unified IR encodes source line/column numbers (src.Pos) and internal linker symbol indices into __.PKGDEF, internal edits to a source file—such as adding comments, blank lines, or unexported functions above an exported declaration—shift line numbers and change pkg.x. When a core library with many reverse dependencies is edited, this forces all downstream packages to recompile even when the exported API and inlineable function bodies are unchanged.
Discussion Context from CL 837805
In CL 837805, we proposed a -nolinenum flag in cmd/compile that omits line numbers (and unexported symbol indices) from .x export data while preserving full source positions in the package's own _go_.o (.a) archive.
In the review discussion on CL 837805, it was noted that because export data includes the bodies of functions that can be inlined across packages, omitting line numbers from .x causes cross-package inlined frames in stack backtraces to lose their inner line numbers.
We'd like to discuss whether there is an approach here that would be acceptable to the Go toolchain—for example, explicitly indicating in stack traces/diagnostics when a line number was omitted due to -nolinenum, or another way to stabilize .x export data across line shifts without degrading stack backtraces.
Activity
ianlancetaylor commentedon Oct 2, 2026
CC @golang/compiler
adonovan commentedon Oct 2, 2026
You're right that line numbers attached to symbols are part of the compiler output, and consequently, changes that otherwise have no effect--such as adding a comment--can cause spurious rebuilds. But I question whether this is a problem worth optimizing. In effect, you want to add a flag to the compiler to make some incremental builds slightly faster and the cost of making the resulting program harder to debug. That seems like a serious mistake.
I expect the pathological scenario is when you are making a sequence of small changes (e.g. tweaking log statements) in a low-level dependency of a large application, and methods or fields defined in the dependency package are exported all the way up the application dependency graph. But how common is this? Most small changes have a limited blast radius: in other words, assuming an import graph A-B-C-D, if you edit package D, you may also have to compile package C that imports it directly, and package B that indirectly uses methods or fields of D exported via C, but perhaps not package A if B does not reexport any information from D, which is the common case.
It would be interesting to make a distribution measuring, for each package in a large module, what fraction of its upward dependencies are potentially affected by the insertion of a comment at the top of each file in that package. My guess is that the numbers are small.
prattmic commentedon Oct 2, 2026
I haven't thought about this too deeply, but at a high level I think it is possible for export data to only use relative line numbers in function bodies. The compiler would leave some sort of relocation in InlinedCall objects for the linker to come along and adjust the line numbers.
I think that could work fine for traceback at runtime.
If this could be done universally, every build would do this and reduce the number of rebuilds.
Compiler error messages may be a problem, as the compiler wouldn't know the absolute line number when mentioning problems. I'm unsure how much of a problem this would be. Dependent packages shouldn't ever produce type errors in inlined functions because the original package would have produced the type error.
Maybe there are no such problematic error messages? I could certainly imagine it becoming a problem though.
Consider an error like:
This error message is fine, but I could imagine us wanting to add a hint like
note: sort.Slice defined at /blah/src/sort/slice.go:123. We wouldn't be able to add that message without the absolute position of the sort.Slice definition.adonovan commentedon Oct 2, 2026
No need to imagine: we already emit errors of the form "cannot infer E" or "field and method with same name" that cite locations in imported packages. There may be others.
Colecf commentedon Oct 2, 2026
Hi, I work with @sgrover6 and proposed the project originally.
We work on soong, the android build system. In soong, there is one core package called that almost all the other packages depend on, so it's like an
A -> {B,C,D,E} -> Fdependency graph, so changing F requires rebuilding all of B,C,D,E. I just benchmarked it, and changing a comment on 1 line takes 6 seconds, adding a line of comments takes 30 seconds. So it is a noticeable improvement on a package that is changed very frequently.Yes, we're aware that this will cause worse error messages and stack traces. I considered this an acceptable tradeoff, though if there's anything we can do to improve the situation like relative line numbers that would also be interesting.
cherrymui commentedon Oct 2, 2026
Using relative line number is probably a good idea in general. It might still be good to have the absolute line number (for function start) somewhere around.
That probably would work. But I'm concerned on putting more work to the linker. I'd hope the linker not to rewrite pcln data as much as possible.
Besides inline function bodies, I think the export data also include generic functions/methods that could be instantiated in other packages. The instantiation would also need the original line numbers. It might also be possible to use relative line numbers. But we also need a mechanism to get the start line right.
It might be possible to design a format that all the pcln tables are relative, and the linker put together a table for function -> (file name, start line) mapping.
As others already commented, I don't think a flag like this is sound. It will cause the resulting binary have different line numbers for the same function depending on whether it is inlined/instantiated and where it is inlined/instantiated (in the same package or different package), which would cause a lot confusion.
If we could have the export data not emit absolute line numbers without affecting build correctness (as discussed above), we should do that without a flag. Otherwise we should never do that.
It might be slightly more acceptable if the flag applies to both the export data and the linker object, which means the binary will not have line numbers at all. At least this is consistent, and not depending on implementation choices like inlining. Not sure we want to have that flag, but I think it is more sound than an export-data-only flag.
May I suggest a quick-and-dirty workaround: format the source code to one line (or as few lines as possible, given multi-line string literals etc.). You don't have to actually change the code in your source tree. But you can run the tool to do the reformat before feeding the files to the compiler. If you use
go build, you can use the-toolexecfeature (see https://pkg.go.dev/cmd/go#hdr-Compile_packages_and_dependencies), which allows wrapping the compiler (and assembler, linker, etc.) with a custom tool, where you can do the reformat.Thanks.
prattmic commentedon Oct 2, 2026
Perhaps you've already done this, but it may be worthwhile to dig into the build performance to see if there are opportunities to speed up the build regardless of incremental vs complete build (e.g., certain pathological packages that take abnormally long to build but could be split up).
I mention this because 30s already sounds like pretty slow build for Go. I can build Go's
make.bashin 25s, and that bootstraps the toolchain, building all of the toolchain binaries from scratch 3 times, and building all of the standard library.