cmd/go: (spurious?) rebuild due to compiler mtime

[oerdnj]

What steps will reproduce the problem?
1. Go 1 Debian packages (2:1-2 version)
2. Get helloworld.go example from front page
3. Run go build -v -x helloworld.go

What is the expected output?
Compiled helloworld

What do you see instead?
morestack trampoline not defined - runtime.morestack00
morestack trampoline not defined - runtime.morestack10
morestack trampoline not defined - runtime.morestack01
morestack trampoline not defined - runtime.morestack11
morestack trampoline not defined - runtime.morestack8
morestack trampoline not defined - runtime.morestack16
morestack trampoline not defined - runtime.morestack24
morestack trampoline not defined - runtime.morestack32
morestack trampoline not defined - runtime.morestack40
morestack trampoline not defined - runtime.morestack48
Segmentation fault

(The segfault is normally hidden by go build, but it appears when 6l or 8l invoked by
hand.)

Which compiler are you using (5g, 6g, 8g, gccgo)?
export WORK=/tmp/go-build622447413;
/usr/lib/go/pkg/tool/linux_amd64/6l -o helloworld -L $WORK $WORK/command-line-arguments.a

ondrej@howl:~$ export WORK=/tmp/go-build587634510
ondrej@howl:~$ /usr/lib/go/pkg/tool/linux_386/8l -o fibonacci -L $WORK
$WORK/command-line-arguments.a
runtime.morestack not defined
Segmentation fault


Which operating system are you using?
Debian Linux amd64 (unstable) and i386 (unstable)

Which version are you using?  (run 'go version')
go1

Please provide any additional information below.
The unfortunate thing is that output binary is produce even though linking ended in
segfault, that also points to the fact, that the output of resulting binary is not
atomic (and I think it should be, e.g. generate temp file in $WORK and move binary into
the place only if the linking was successful).

Attaching $WORK environment and strace output.

I do some packaging mumbojumbo, but in go1 all patches are related to GNU/KFreeBSD
compatibility and not distributing non *.go source files, which _should_ not really
matter (or I see no error related to that fact).

Attachments:

1. go1-amd64-sigsegv.tar.gz (2403076 bytes)
2. go1-i386-sigsegv.tar.gz (204314 bytes)

[rsc]

Comment 1:

The go command is rebuilding package runtime - you can see the built objects in the
.tar.gz files you attached - but I don't understand why it is doing that.
Worse, it is rebuilding package runtime using only the .go files.  It omits all the C
and assembly implementation objects.  (The runtime.a that is built contains only _go_.8,
no other .8 files.)  Does /usr/lib/go/src/pkg/runtime have any .c or .s files in it?
Not having the C or assembly objects in runtime.a is what causes the morestack undefined
errors.
Did you unpack the tree into /usr/lib/go or did you only copy certain files?

Labels changed: added priority-later, removed priority-triage.

Status changed to WaitingForReply.

[oerdnj]

Comment 2:

Hmm, I think I have found a culprit...
In strace there is:
12188 stat64("/usr/lib/go/src/pkg/fmt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
12188 open("/usr/lib/go/src/pkg/fmt", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
12188 getdents64(3, /* 10 entries */, 4096) = 320
12188 getdents64(3, /* 0 entries */, 4096) = 0
12188 lstat64("/usr/lib/go/src/pkg/fmt/scan_test.go", {st_mode=S_IFREG|0644,
st_size=25834, ...}) = 0
12188 lstat64("/usr/lib/go/src/pkg/fmt/scan.go", {st_mode=S_IFREG|0644, st_size=30770,
...}) = 0
12188 lstat64("/usr/lib/go/src/pkg/fmt/stringer_test.go", {st_mode=S_IFREG|0644,
st_size=2156, ...}) = 0
12188 lstat64("/usr/lib/go/src/pkg/fmt/format.go", {st_mode=S_IFREG|0644, st_size=10125,
...}) = 0
12188 lstat64("/usr/lib/go/src/pkg/fmt/print.go", {st_mode=S_IFREG|0644, st_size=27721,
...}) = 0
12188 lstat64("/usr/lib/go/src/pkg/fmt/doc.go", {st_mode=S_IFREG|0644, st_size=7793,
...}) = 0
12188 lstat64("/usr/lib/go/src/pkg/fmt/export_test.go", {st_mode=S_IFREG|0644,
st_size=196, ...}) = 0
12188 lstat64("/usr/lib/go/src/pkg/fmt/fmt_test.go", {st_mode=S_IFREG|0644,
st_size=22484, ...}) = 0
12188 close(3)                          = 0
I guess this is the code you use to check if the package needs to be recompiled.
However even if I set timestamps on all directories and all source files, the fmt still
gets recompiled, which I think is wrong (or at least it doesn't really make sense to
have compiled objects in the code).
root@howl:~# ls -ld /usr/lib/go/pkg/linux_386/fmt.a 
-rw-r--r-- 1 root root 817678 Apr  6 15:18 /usr/lib/go/pkg/linux_386/fmt.a
root@howl:~# ls -ld /usr/lib/go/src/pkg/fmt
drwxr-xr-x 2 root root 4096 Jan  1 01:01 /usr/lib/go/src/pkg/fmt
root@howl:~# ls -ld /usr/lib/go/src/pkg/fmt/*
-rw-r--r-- 1 root root  7793 Jan  1 01:01 /usr/lib/go/src/pkg/fmt/doc.go
-rw-r--r-- 1 root root   196 Jan  1 01:01 /usr/lib/go/src/pkg/fmt/export_test.go
-rw-r--r-- 1 root root 22484 Jan  1 01:01 /usr/lib/go/src/pkg/fmt/fmt_test.go
-rw-r--r-- 1 root root 10125 Jan  1 01:01 /usr/lib/go/src/pkg/fmt/format.go
-rw-r--r-- 1 root root 27721 Jan  1 01:01 /usr/lib/go/src/pkg/fmt/print.go
-rw-r--r-- 1 root root 30770 Jan  1 01:01 /usr/lib/go/src/pkg/fmt/scan.go
-rw-r--r-- 1 root root 25834 Jan  1 01:01 /usr/lib/go/src/pkg/fmt/scan_test.go
-rw-r--r-- 1 root root  2156 Jan  1 01:01 /usr/lib/go/src/pkg/fmt/stringer_test.go
This is a clearly regression from 0.60.x, because it worked before (but it didn't have
to 'go' command at all).
And one more observation:
If I remove src tree altogether, the go command don't even check if there are binary
libraries available and bail out:
ondrej@howl:~$ go build helloworld.go
helloworld.go:5:8: import "fmt": cannot find package
package runtime: import "runtime": cannot find package
But I can still compile the helloworld using 8g/8l combo just fine:
ondrej@howl:~$ /usr/lib/go/pkg/tool/linux_386/8g helloworld.go 
ondrej@howl:~$ /usr/lib/go/pkg/tool/linux_386/8l helloworld.8  
ondrej@howl:~$ ./8.out 
Hello, world!
From my viewpoint there's something twisted in go command which needs to have sources
available at every possible moment.  If that's your design decision I can live with that
(and copy the rest of the source tree to the package), but I little bit doubt about the
fact that you always recompile all needed libraries every time.

[minux]

Comment 3:

What does 'go env' say? Did you have $GOROOT defined?

[oerdnj]

Comment 4:

No special settings, I am testing the package inside clean chrooted Debian unstable.
$ go env
GOROOT="/usr/lib/go"
GOBIN=""
GOARCH="amd64"
GOCHAR="6"
GOOS="linux"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOGCCFLAGS="-g -O2 -fPIC -m64 -pthread"
CGO_ENABLED="1"

[rsc]

Comment 5:

If GOARCH is amd64, then the go command needs /usr/lib/go/pkg/linux_amd64/fmt.a.
The existence of /usr/lib/go/pkg/linux_386/fmt.a doesn't help.
When you are compiling code from a directory outside /usr/lib/go,
the go command will not rebuild code from /usr/lib/go unless there
are no binaries at all.
Russ

[oerdnj]

Comment 6:

{6,8}{g,l} is able to produce binary even if the source tree is not installed at all:
ondrej@howl:~$ ls
fibonacci.6  fibonacci.go
ondrej@howl:~$ find -L /usr/lib/go/src
/usr/lib/go/src
/usr/lib/go/src/pkg
/usr/lib/go/src/pkg/runtime
/usr/lib/go/src/pkg/runtime/cgocall.h
/usr/lib/go/src/pkg/runtime/runtime.h
/usr/lib/go/src/Make.dist
ondrej@howl:~$ /usr/lib/go/pkg/tool/linux_amd64/6g fibonacci.go 
ondrej@howl:~$ /usr/lib/go/pkg/tool/linux_amd64/6l fibonacci.6
ondrej@howl:~$ ls
6.out  fibonacci.6  fibonacci.go
ondrej@howl:~$ ./6.out 
1 1 2 3 5
It is just a problem of 'go' command not checking if it has binary libraries already
available.

[rsc]

Comment 7:

The go command requires source code to be present.
Eventually we will lift that restriction, but we did not lift it
for Go 1.0.  Note that even though 6g works, tools like
godoc still need source code to tell people how to use fmt,
so it does serve other purposes too.

[oerdnj]

Comment 8:

> The go command requires source code to be present.
> Eventually we will lift that restriction, but we did not lift it
> for Go 1.0.  Note that even though 6g works, tools like
Ok, even though this is a change from previous stable release, I'll handle that somehow.
> godoc still need source code to tell people how to use fmt,
> so it does serve other purposes too.
Yes, I know and this is handled.
I will follow-up in next comment...

[rsc]

Comment 9:

You can create a dummy file in each directory
echo package fmt >src/pkg/fmt/dummy.go
and so on.  That will be enough to convince the
go command that "fmt" is a real import and not
just a stale .a file that happens to be lying around.
Russ

[oerdnj]

Comment 10:

I have tried one more thing...  with binaries from Debian package I have replaced
/usr/lib/go with content from binary distribution. It works. Removed everything with
exception of /usr/lib/go/{src,pkg}. It works.
When I have a /usr/lib/go/pkg/ from package and /usr/lib/go/src/ from upstream binary
distribution. The list of files is exactly same. And it tries to recompile the libraries.
So, the question is what is the heuristics to detect that the file needs to be
recompiled?

[rsc]

Comment 11:

1. If there's no source at all, it's not a real package.
2. If the source exists and the binary does not, compile the source.
3. If the source exists and is newer than the binary _and_ you are
working in a directory inside the GOROOT, recompile the source.
I believe you are working outside the GOROOT, so #3 does not apply.
What does 'go env' print in both cases?
In your earlier mail you were trying to run a 64-bit (amd64) Go with
32-bit (386) pre-compiled packages.  My guess is that the same thing
is happening here.
If 'go env' says GOARCH="amd64" then you need pkg/linux_amd64.  If 'go
env' says GOARCH="386" then you need pkg/linux_386.
To understand more about what the go command thinks of a given package
(say, fmt), run 'go list -json fmt'.  Does the "Target" exist?

[oerdnj]

Comment 12:

> You can create a dummy file in each directory
I don't really need to do that, since I am already distribution "find src -name '*.go'"
in the golang-src package, so it should work.
The problem here is that it keeps recompiling stuff even if it doesn't need to and I am
unable to understand why.

[oerdnj]

Comment 13:

> To understand more about what the go command thinks of a given package
> (say, fmt), run 'go list -json fmt'.  Does the "Target" exist?
ondrej@howl:~$ go list -json runtime | grep Target
    "Target": "/usr/lib/go/pkg/linux_amd64/runtime.a",
ondrej@howl:~$ ls -l /usr/lib/go/pkg/linux_amd64/runtime.a 
-rw-r--r-- 1 root root 883732 Apr 11 14:35 /usr/lib/go/pkg/linux_amd64/runtime.a
So, it does exist, also all sources are in place:
$ find /usr/lib/go/src/pkg/runtime/ -type f | wc -l
220
But still, when I try to compile fibonacci.go:
WORK=/tmp/go-build260021805
runtime
mkdir -p $WORK/runtime/_obj/
cd /usr/lib/go/src/pkg/runtime
/usr/lib/go/pkg/tool/linux_amd64/6g -o $WORK/runtime/_obj/_go_.6 -p runtime -+ -D
_/usr/lib/go/src/pkg/runtime -I $WORK ./compiler.go ./debug.go ./error.go ./extern.go
./mem.go ./softfloat64.go ./type.go ./zgoarch_amd64.go ./zgoos_linux.go
./zruntime_defs_linux_amd64.go ./zversion.go
[...more files compiling...]
/usr/lib/go/pkg/tool/linux_amd64/pack grc $WORK/runtime.a $WORK/runtime/_obj/_go_.6
$WORK/runtime/_obj/alg.6 $WORK/runtime/_obj/atomic_amd64.6 $WORK/runtime/_obj/cgocall.6
$WORK/runtime/_obj/chan.6 $WORK/runtime/_obj/closure_amd64.6
$WORK/runtime/_obj/complex.6 $WORK/runtime/_obj/cpuprof.6 $WORK/runtime/_obj/float.6
$WORK/runtime/_obj/hashmap.6 $WORK/runtime/_obj/iface.6 $WORK/runtime/_obj/lock_futex.6
$WORK/runtime/_obj/mcache.6 $WORK/runtime/_obj/mcentral.6 $WORK/runtime/_obj/mem_linux.6
$WORK/runtime/_obj/mfinal.6 $WORK/runtime/_obj/mfixalloc.6 $WORK/runtime/_obj/mgc0.6
$WORK/runtime/_obj/mheap.6 $WORK/runtime/_obj/msize.6 $WORK/runtime/_obj/print.6
$WORK/runtime/_obj/proc.6 $WORK/runtime/_obj/rune.6 $WORK/runtime/_obj/runtime.6
$WORK/runtime/_obj/signal_linux_amd64.6 $WORK/runtime/_obj/signal_unix.6
$WORK/runtime/_obj/slice.6 $WORK/runtime/_obj/symtab.6 $WORK/runtime/_obj/thread_linux.6
$WORK/runtime/_obj/traceback_x86.6 $WORK/runtime/_obj/zmalloc_amd64.6
$WORK/runtime/_obj/zmprof_amd64.6 $WORK/runtime/_obj/zruntime1_amd64.6
$WORK/runtime/_obj/zsema_amd64.6 $WORK/runtime/_obj/zsigqueue_amd64.6
$WORK/runtime/_obj/zstring_amd64.6 $WORK/runtime/_obj/ztime_amd64.6
$WORK/runtime/_obj/asm_amd64.6 $WORK/runtime/_obj/memmove_amd64.6
$WORK/runtime/_obj/rt0_linux_amd64.6 $WORK/runtime/_obj/sys_linux_amd64.6
command-line-arguments
mkdir -p $WORK/command-line-arguments/_obj/
cd /home/ondrej
/usr/lib/go/pkg/tool/linux_amd64/6g -o $WORK/command-line-arguments/_obj/_go_.6 -p
command-line-arguments -D _/home/ondrej -I $WORK ./fibonacci.go
/usr/lib/go/pkg/tool/linux_amd64/pack grc $WORK/command-line-arguments.a
$WORK/command-line-arguments/_obj/_go_.6
cd .
/usr/lib/go/pkg/tool/linux_amd64/6l -o fibonacci -L $WORK $WORK/command-line-arguments.a
It should not really happen according to your description, but it does.

[rsc]

Comment 14:

Can you attach a .tar.gz of the entire /usr/lib/go you are using,
along with the 'go' binary itself (if it lives outside that tree)?
Thanks.

[oerdnj]

Comment 15:

Attached in 10MB parts.
And thank you that you care.

Attachments:

1. gotree.tar.gz.aa (10240000 bytes)