Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile, cmd/link: can't build large arm binaries with external linking, Kubernetes now too big #17028

Closed
luxas opened this issue Sep 8, 2016 · 62 comments

Comments

Projects
None yet
@luxas
Copy link

commented Sep 8, 2016

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go1.6.3

What operating system and processor architecture are you using (go env)?

linux/arm

What did you do?

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.
Just run
docker run -it gcr.io/google_containers/kube-apiserver-arm:v1.4.0-alpha.3 /usr/local/bin/kube-apiserver

What did you expect to see?

kube-apiserver starting

What did you see instead?

unexpected fault address 0x40d62ec
fatal error: fault
[signal 0xb code=0x2 addr=0x40d62ec pc=0x40d62ec]

goroutine 1 [running, locked to thread]:
runtime.throw(0x2cc9010, 0x5)
    /usr/local/go/src/runtime/panic.go:547 +0x78 fp=0x1482beec sp=0x1482bee0
runtime.sigpanic()
    /usr/local/go/src/runtime/sigpanic_unix.go:27 +0x280 fp=0x1482bf18 sp=0x1482beec
k8s.io/kubernetes/vendor/github.com/docker/engine-api/types/versions.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/engine-api/types/versions/compare.go:62 +0x4c fp=0x1482bf20 sp=0x1482bf1c
k8s.io/kubernetes/vendor/github.com/docker/engine-api/types/filters.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/engine-api/types/filters/parse.go:295 +0x5c fp=0x1482bf34 sp=0x1482bf20
k8s.io/kubernetes/vendor/github.com/docker/engine-api/types.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/engine-api/types/types.go:473 +0x5c fp=0x1482bf38 sp=0x1482bf34
k8s.io/kubernetes/pkg/credentialprovider.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/credentialprovider/provider.go:123 +0x7c fp=0x1482bf6c sp=0x1482bf38
k8s.io/kubernetes/pkg/credentialprovider/aws.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/credentialprovider/aws/aws_credentials.go:232 +0x70 fp=0x1482bf70 sp=0x1482bf6c
k8s.io/kubernetes/pkg/cloudprovider/providers/aws.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/aws/sets_ippermissions.go:146 +0xb0 fp=0x1482bf90 sp=0x1482bf70
k8s.io/kubernetes/pkg/cloudprovider/providers.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/providers.go:30 +0x4c fp=0x1482bf94 sp=0x1482bf90
k8s.io/kubernetes/cmd/kube-apiserver/app.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:310 +0x4c fp=0x1482bf98 sp=0x1482bf94
main.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/apiserver.go:53 +0x5c fp=0x1482bf9c sp=0x1482bf98
runtime.main()
    /usr/local/go/src/runtime/proc.go:177 +0x274 fp=0x1482bfc4 sp=0x1482bf9c
runtime.goexit()
    /usr/local/go/src/runtime/asm_arm.s:990 +0x4 fp=0x1482bfc4 sp=0x1482bfc4

goroutine 5 [chan receive]:
k8s.io/kubernetes/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x462d1f8)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x60
created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.1
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x2cc

Seems like Kubernetes with it's deps just grew too big for arm :(
Any help here would be really appreciated. We're releasing Kubernetes in about 10 days

I need help from some Go guru here that knows how the internals work!
I assume this is a Go issue rather than a Kubernetes issue, since the file that's segfaulting doesn't have an init()

Please take a look as quickly as possible!
-> @lavalamp @smarterclayton @ixdy @rsc @davecheney @wojtek-t @jfrazelle @bradfitz

@lavalamp

This comment has been minimized.

Copy link

commented Sep 8, 2016

The file in question is: https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/docker/engine-api/types/versions/compare.go

@luxas have you done a git bisect to try and figure out what introduced this? I am guessing that the go team would probably appreciate a more specific problem to look at.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Sep 8, 2016

There's always at least an implicit init function in packages to run all the assignments and function calls at the package (global) scope, and to call the init functions of dependent packages.

Does the problem reproduce with Go 1.7.1?

Seems like Kubernetes with it's deps just grew too big for arm :(

What makes you say that?

@luxas

This comment has been minimized.

Copy link
Author

commented Sep 8, 2016

@lavalamp
hyperkube was working at v1.4.0-alpha.1, broke at v1.4.0-alpha.2
kube-apiserver was working at v1.4.0-alpha.2, broke at v1.4.0-alpha.3

breakage at current HEAD with go 1.7.1:

unexpected fault address 0x408cce4
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0x408cce4 pc=0x408cce4]

goroutine 1 [running, locked to thread]:
runtime.throw(0x275f8c2, 0x5)
        /usr/local/go/src/runtime/panic.go:566 +0x78 fp=0x13f1b6cc sp=0x13f1b6c0
runtime.sigpanic()
        /usr/local/go/src/runtime/sigpanic_unix.go:27 +0x280 fp=0x13f1b6f8 sp=0x13f1b6cc
k8s.io/kubernetes/vendor/golang.org/x/net/internal/timeseries.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/internal/timeseries/timeseries.go:526 +0x4c fp=0x13f1b700 sp=0x13f1b6fc
k8s.io/kubernetes/vendor/golang.org/x/net/trace.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/trace/trace.go:1064 +0x80 fp=0x13f1b748 sp=0x13f1b700
k8s.io/kubernetes/vendor/google.golang.org/grpc.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/trace.go:120 +0x6c fp=0x13f1b75c sp=0x13f1b748
k8s.io/kubernetes/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:1867 +0x70 fp=0x13f1bf18 sp=0x13f1b75c
k8s.io/kubernetes/vendor/github.com/coreos/etcd/clientv3.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/clientv3/watch.go:715 +0x58 fp=0x13f1bf3c sp=0x13f1bf18
k8s.io/kubernetes/pkg/storage/etcd3.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/storage/etcd3/watcher.go:352 +0x58 fp=0x13f1bf40 sp=0x13f1bf3c
k8s.io/kubernetes/pkg/storage/storagebackend/factory.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/storage/storagebackend/factory/factory.go:44 +0x70 fp=0x13f1bf44 sp=0x13f1bf40
k8s.io/kubernetes/pkg/registry/generic.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/generic/storage_decorator.go:61 +0x6c fp=0x13f1bf48 sp=0x13f1bf44
k8s.io/kubernetes/pkg/registry/generic/registry.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/generic/registry/store.go:937 +0x60 fp=0x13f1bf68 sp=0x13f1bf48
k8s.io/kubernetes/cmd/kube-apiserver/app/options.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/options/options.go:131 +0x5c fp=0x13f1bf6c sp=0x13f1bf68
k8s.io/kubernetes/cmd/kube-apiserver/app.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:311 +0xb8 fp=0x13f1bf70 sp=0x13f1bf6c
main.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/apiserver.go:55 +0x5c fp=0x13f1bf74 sp=0x13f1bf70
runtime.main()
        /usr/local/go/src/runtime/proc.go:172 +0x224 fp=0x13f1bf9c sp=0x13f1bf74
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x13f1bf9c sp=0x13f1bf9c

goroutine 5 [chan receive]:
k8s.io/kubernetes/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x3a05e90)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x60
created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.1
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x1ec

What makes you say that?

It guess it's hitting some kind of limit in the golang arm implementation right now
The panic error seems to change with every commit.

I don't know much about the lower-level Golang ARM implementation, can you think about something that would panic there?

@jessfraz

This comment has been minimized.

Copy link
Contributor

commented Sep 8, 2016

can you bisect for the exact sha that started the breakage, not the tag

@luxas

This comment has been minimized.

Copy link
Author

commented Sep 8, 2016

Trying to find the commit kube-apiserver broke on

@laboger

This comment has been minimized.

Copy link
Contributor

commented Sep 8, 2016

Your faulting address is the pc (program counter) and it is greater than 2**26. I'm not familiar with arm, but could it be that the 32 bit arm platform has a limit on the size of the programs it can run and these programs are exceeding it? These are the same programs we have issues with on ppc64le due to their size.

@luxas

This comment has been minimized.

Copy link
Author

commented Sep 8, 2016

@laboger Thanks a lot!
Yeah, there's the issue.
0x408cce4 == 67685604 which is 576740 bigger than 67108864 (2**26)

also broke with pc=0x4018784 == 67209092 which is 100228 bigger than 2**26

at commit kubernetes/kubernetes@a81732b it didn't break, but with kubernetes/kubernetes@ff9980e it does break.

The difference there is only 300 commits, but the binary clearly passes the 0x4000000 (2**26) mark just there.

Well, now we know what the problem is.
Do anyone have suggestions on what to do? Can we change something in golang to solve it or is it just a hard limit for 32-bit overall?

@luxas

This comment has been minimized.

Copy link
Author

commented Sep 8, 2016

Other interesting observations:

  • running the arm binary on an arm64 host (pine64) segfaults
  • running the arm64 version of kube-apiserver v1.4.0-alpha.3 works

So it might be a limitation in golang...

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Sep 8, 2016

running a 32 bit binary on a 64 bit system will run it in 32 bit emulation
mode with the same limitations.

On Fri, Sep 9, 2016 at 7:53 AM, Lucas Käldström notifications@github.com
wrote:

Other interesting observations:

  • running the arm binary on an arm64 host (pine64) segfaults
  • running the arm64 version of kube-apiserver v1.4.0-alpha.3 works

So it might be a limitation in golang...


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#17028 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA7z4Oc_DEn2ZVkXpW9q3ZfOEIe40ks5qoIPKgaJpZM4J4WnZ
.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Sep 8, 2016

@minux

This comment has been minimized.

Copy link
Member

commented Sep 9, 2016

@bradfitz

This comment has been minimized.

Copy link
Member

commented Sep 9, 2016

@bradfitz bradfitz changed the title arm is segfaulting for kube-apiserver on kubernetes HEAD cmd/compile, cmd/link: can't build large arm binaries, Kubernetes now too big Sep 9, 2016

@bradfitz bradfitz added this to the Go1.8 milestone Sep 9, 2016

@luxas

This comment has been minimized.

Copy link
Author

commented Sep 9, 2016

Thanks everyone for helping with this!

I know @laboger has a patch for ppc64le that can be cherrypicked upon go1.7
Is it possible to create a such patch for this problem that we can temporarily use for the v1.4 release (go1.7 + 2 patches), only when building arm and ppc64le.
I think I can make the Kubernetes build infra work, but I can't make the arm patch.
Can someone try to make a fast prototype I can test?

@crawshaw

This comment has been minimized.

Copy link
Contributor

commented Sep 9, 2016

The CL @laboger sent may cover this with minimal changes. I haven't investigated how the external linker is rewriting the jump, it may not work on ARM, but it's worth a try. @luxas, can you patch in CL 27790, modify cmd/link/internal/ld/data.go:1912 to look for either PPC64 or ARM, and see if it works?

@laboger

This comment has been minimized.

Copy link
Contributor

commented Sep 9, 2016

Has any investigation been done to determine if the binaries built by Kubernetes could be reduced in size? I suggested this when the problem was first reported. That would not only resolve your issue with ppc64le and arm but improve compile and link times, save space for your binaries, etc. etc. I looked briefly at the packages included in hyperkube and it appeared to have many duplicate path names rooted at different locations. Could there be new versions of some packages being added without removing the old ones that are no longer used? Or could the binaries get split in any way?
I can send you more detail on the package path names I am referring to if you want @luxas.

@cherrymui

This comment has been minimized.

Copy link
Contributor

commented Sep 9, 2016

While we are solving the linker problem, in the meantime, is using Go tip an option for you? The SSA compiler generates smaller code that does not fault.

$ size -A kube-apiserver-*
kube-apiserver-nossa  :
section                  size       addr
.text                34873308      69632
.rodata               8542712   34942976
.typelink              130564   43485688
.itablink               11288   43616252
.gosymtab                   0   43627540
.gopclntab           15408474   43627544
.noptrdata            1475636   59039744
.data                   34424   60515384
.bss                    85392   60549808
.noptrbss               22672   60635200
.debug_abbrev             255   60661760
.debug_line           2956588   60662015
.debug_frame          2925588   63618603
.debug_pubnames        966925   66544191
.debug_pubtypes       4387464   67511116
.debug_aranges             32   71898580
.debug_gdb_scripts         48   71898612
.debug_info          20743066   71898660
.note.go.buildid           56      69576
Total                92564492


kube-apiserver-ssa  :
section                  size       addr
.text                28344912      69632
.rodata               8210247   28418048
.typelink              129448   36628296
.itablink               11288   36757744
.gosymtab                   0   36769032
.gopclntab           16949001   36769032
.noptrdata            1475636   53719040
.data                   34424   55194680
.bss                    85392   55229104
.noptrbss               22664   55314496
.debug_abbrev             255   55341056
.debug_line           6825244   55341311
.debug_frame          2914972   62166555
.debug_pubnames        966478   65081527
.debug_pubtypes       4012779   66048005
.debug_aranges             32   70060784
.debug_gdb_scripts         48   70060816
.debug_info          18903367   70060864
.note.go.buildid           56      69576
Total                88886243
@cherrymui

This comment has been minimized.

Copy link
Contributor

commented Sep 9, 2016

What about we introduce a -largemode flag or like, when set, always emits "load address into register; make indirect call"? This seems a simpler fix, although it makes large program even larger... There might be some optimizations that reduce the overhead (when we know for sure it is safe, make direct call). And the linker should signal and hint to use -largemode when it sees a jump-too-far.

@laboger

This comment has been minimized.

Copy link
Contributor

commented Sep 9, 2016

I think that would be a good solution for all platforms that could have this problem, and would fix internal and external linking. My CL only works for external linking, because it depends on the GNU linker to handle the long calls.

I believe it needs to be an option because it will degrade call performance, so we wouldn't want to do it all the time but there should be opportunities for optimizing. @minux made similar suggestions in the corresponding issue for ppc64le #15823.

@laboger

This comment has been minimized.

Copy link
Contributor

commented Sep 9, 2016

One other note, on arm isn't this a runtime problem when executing the program that has been built, not a build problem? So while I think the problem is due to the large program size the symptoms on arm 32 bit are different than ppc64le.

@cherrymui

This comment has been minimized.

Copy link
Contributor

commented Sep 9, 2016

I think it is a build problem: it emits a pc-relative jump and the offset overflows, and the linker did not signal, so the program jumped to a faulty address.

@crawshaw

This comment has been minimized.

Copy link
Contributor

commented Sep 9, 2016

I don't think a new compiler mode is the answer. That won't fit well with the go tool.

If there's no space in the text for the linker to rewrite the pc-relative jump, than as long as no single function is too large, the linker can insert a trampoline.

@laboger

This comment has been minimized.

Copy link
Contributor

commented Sep 9, 2016

That means your suggestion is for internal linking only? Because I assume you mean the internal linker would insert the trampoline.

@luxas

This comment has been minimized.

Copy link
Author

commented Sep 9, 2016

@laboger Can you provide a version of your CL as cherrypickable upon go1.6.3 or go1.7.1 so I can test if it fixes this issue?

@crawshaw

This comment has been minimized.

Copy link
Contributor

commented Sep 9, 2016

@laboger The linker has plenty of opportunities to insert trampolines for both external and internal linking (though it may take some wiring).

@laboger

This comment has been minimized.

Copy link
Contributor

commented Sep 9, 2016

@luxas If you get the patch from CL 23963, that should apply against go 1.7. That was my previous patch before I had to rebase it for master and ended up with a new CL.

Please note that this patch depends on using external linking so that the external linker makes the necessary changes to allow for long jumps or calls.

@luxas

This comment has been minimized.

Copy link
Author

commented Sep 9, 2016

Does the arm compiler link internally or externally by default?

@laboger

This comment has been minimized.

Copy link
Contributor

commented Sep 9, 2016

@crawshaw But when you say "the linker" you mean the golang linker would generate the trampolines before writing out the elf text sections that are passed on to the external linker?

@crawshaw

This comment has been minimized.

Copy link
Contributor

commented Sep 9, 2016

Yes. (Of course it may be that the external linker is already smart enough to do that itself, I haven't checked.)

deads2k pushed a commit to deads2k/kubernetes that referenced this issue Sep 16, 2016

Kubernetes Submit Queue
Merge pull request kubernetes#32517 from luxas/fix_arm_ppc64le
Automatic merge from submit-queue

Use a patched golang version for building linux/arm

Fixes: kubernetes#29904

Right now, linux/arm is broken because of an internal limitation in Go.
I've filed an issue for it here: golang/go#17028

The affected binaries of this limitation are hyperkube and kube-apiserver, which are the largest binaries.

And when we now have a patched go 1.7.1 version for building "unsupported" but important architectures (ref: https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/multi-platform.md), we should also include the patch for ppc64le and start building ppc64le again. 

As soon as @laboger has the patch I need up on Github, I'll include ppc64le to this PR and we'll merge it

TODO:
 - [ ] ~~Update the PR with patches for ppc64le at the same time @luxas~~
 - [x] Push the new kube-cross image @ixdy 
 - [x] Run a full `make release` before to verify nothing breaks @luxas + @ixdy 
 - [ ] Cherrypick into the 1.4 branch @luxas + (who?)

@lavalamp @smarterclayton @ixdy @rsc @davecheney @wojtek-t @jfrazelle @bradfitz @david-mcmahon @pwittrock

eparis added a commit to eparis/kubernetes that referenced this issue Sep 16, 2016

Merge pull request kubernetes#32517 from luxas/fix_arm_ppc64le
Automatic merge from submit-queue

Use a patched golang version for building linux/arm

Fixes: kubernetes#29904

Right now, linux/arm is broken because of an internal limitation in Go.
I've filed an issue for it here: golang/go#17028

The affected binaries of this limitation are hyperkube and kube-apiserver, which are the largest binaries.

And when we now have a patched go 1.7.1 version for building "unsupported" but important architectures (ref: https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/multi-platform.md), we should also include the patch for ppc64le and start building ppc64le again.

As soon as @laboger has the patch I need up on Github, I'll include ppc64le to this PR and we'll merge it

TODO:
 - [ ] ~~Update the PR with patches for ppc64le at the same time @luxas~~
 - [x] Push the new kube-cross image @ixdy
 - [x] Run a full `make release` before to verify nothing breaks @luxas + @ixdy
 - [ ] Cherrypick into the 1.4 branch @luxas + (who?)

@lavalamp @smarterclayton @ixdy @rsc @davecheney @wojtek-t @jfrazelle @bradfitz @david-mcmahon @pwittrock
(cherry picked from commit 9bc7e36)

eparis added a commit to eparis/kubernetes that referenced this issue Sep 17, 2016

Merge pull request kubernetes#32517 from luxas/fix_arm_ppc64le
Automatic merge from submit-queue

Use a patched golang version for building linux/arm

Fixes: kubernetes#29904

Right now, linux/arm is broken because of an internal limitation in Go.
I've filed an issue for it here: golang/go#17028

The affected binaries of this limitation are hyperkube and kube-apiserver, which are the largest binaries.

And when we now have a patched go 1.7.1 version for building "unsupported" but important architectures (ref: https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/multi-platform.md), we should also include the patch for ppc64le and start building ppc64le again.

As soon as @laboger has the patch I need up on Github, I'll include ppc64le to this PR and we'll merge it

TODO:
 - [ ] ~~Update the PR with patches for ppc64le at the same time @luxas~~
 - [x] Push the new kube-cross image @ixdy
 - [x] Run a full `make release` before to verify nothing breaks @luxas + @ixdy
 - [ ] Cherrypick into the 1.4 branch @luxas + (who?)

@lavalamp @smarterclayton @ixdy @rsc @davecheney @wojtek-t @jfrazelle @bradfitz @david-mcmahon @pwittrock
(cherry picked from commit 9bc7e36)
@gopherbot

This comment has been minimized.

Copy link

commented Sep 19, 2016

CL https://golang.org/cl/29397 mentions this issue.

@luxas

This comment has been minimized.

Copy link
Author

commented Sep 30, 2016

@cherrymui Many thanks for the CLs you've made.
Thanks to your first patch, kube-apiserver will now work on arm in k8s v1.4.1!!

I assume CL 29397 is the "real" one, which can be merged at some point, right?

We should definitely have a proper fix for this as soon as possible.
So, what's the status of this right now?
Can you describe the issue and what you're proposing to do?

@cherrymui

This comment has been minimized.

Copy link
Contributor

commented Oct 3, 2016

@luxas Yes, I think CL 29397 will be the "real" one.
I just need to refactor the code as pointed in the review.

Can you describe the issue and what you're proposing to do?

What issue?

@luxas

This comment has been minimized.

Copy link
Author

commented Oct 3, 2016

What issue?

I meant this linker issue. Can you write a quick recap of what you're doing in CL 29397?

@cherrymui

This comment has been minimized.

Copy link
Contributor

commented Oct 3, 2016

In short, the issue is that when resolving a direct jump (or call), if the target is too far so that the offset cannot be encoded into the instruction, the linker simply truncated the address, and the generated binary jumped wild. The fix is to let the linker detect too-far jumps and automatically insert trampolines when needed.

@laboger

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2016

Isn't this fix for internal linking only?

@cherrymui

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2016

I believe it is not too hard to make it also work for external linking. But maybe not in that CL.

@luxas

This comment has been minimized.

Copy link
Author

commented Oct 4, 2016

Thanks, what's the ETA on CL 29397?
I'd love to have external linking support as well, otherwise kubelet will break next.

But one step at a time.

@cherrymui

This comment has been minimized.

Copy link
Contributor

commented Oct 5, 2016

Thanks, what's the ETA on CL 29397?

Hopefully by this week.

@luxas

This comment has been minimized.

Copy link
Author

commented Oct 11, 2016

@cherrymui Thanks for the patch!

Still I think we need to get external linking working, so can you reopen this issue @bradfitz @cherrymui @crawshaw?

Do you have any plan to get external linking working or am I missing something?

@crawshaw crawshaw changed the title cmd/compile, cmd/link: can't build large arm binaries, Kubernetes now too big cmd/compile, cmd/link: can't build large arm binaries with external linking, Kubernetes now too big Oct 11, 2016

@crawshaw crawshaw reopened this Oct 11, 2016

@crawshaw

This comment has been minimized.

Copy link
Contributor

commented Oct 11, 2016

I would have thought a similar approach to ppc64 should work, or including the trampolines in the section given to the external linker. I don't know if anyone is working on it.

@luxas

This comment has been minimized.

Copy link
Author

commented Oct 11, 2016

@laboger is working on ppc64le support for this as well, see #16665

@cherrymui

This comment has been minimized.

Copy link
Contributor

commented Oct 11, 2016

I will try to bring up external linking support on ARM soon.

@laboger

This comment has been minimized.

Copy link
Contributor

commented Oct 14, 2016

Clarification on the ppc64le fixes: the fix for this problem when using external linking is already upstream. That is the fix to split text sections if they get too large #15823, because then the GNU linker can create and insert trampolines or long branches as needed. The fix for #16665 solves the problem on ppc64le when using internal linking.

@luxas

This comment has been minimized.

Copy link
Author

commented Oct 14, 2016

@laboger But arm still needs this external linking fix?

@laboger

This comment has been minimized.

Copy link
Contributor

commented Oct 14, 2016

Yes it sounds like @cherrymui is working on it.

@gopherbot

This comment has been minimized.

Copy link

commented Oct 17, 2016

CL https://golang.org/cl/31143 mentions this issue.

@gopherbot gopherbot closed this in c7e8556 Oct 17, 2016

shyamjvs pushed a commit to shyamjvs/kubernetes that referenced this issue Dec 1, 2016

Merge pull request kubernetes#32517 from luxas/fix_arm_ppc64le
Automatic merge from submit-queue

Use a patched golang version for building linux/arm

Fixes: kubernetes#29904

Right now, linux/arm is broken because of an internal limitation in Go.
I've filed an issue for it here: golang/go#17028

The affected binaries of this limitation are hyperkube and kube-apiserver, which are the largest binaries.

And when we now have a patched go 1.7.1 version for building "unsupported" but important architectures (ref: https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/multi-platform.md), we should also include the patch for ppc64le and start building ppc64le again.

As soon as @laboger has the patch I need up on Github, I'll include ppc64le to this PR and we'll merge it

TODO:
 - [ ] ~~Update the PR with patches for ppc64le at the same time @luxas~~
 - [x] Push the new kube-cross image @ixdy
 - [x] Run a full `make release` before to verify nothing breaks @luxas + @ixdy
 - [ ] Cherrypick into the 1.4 branch @luxas + (who?)

@lavalamp @smarterclayton @ixdy @rsc @davecheney @wojtek-t @jfrazelle @bradfitz @david-mcmahon @pwittrock
(cherry picked from commit 9bc7e36)

@golang golang locked and limited conversation to collaborators Oct 17, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.