Build kubectl binary with cgo enabled

[pwittrock]

cgo is required for certain go std libraries to work. See this article for details.

This breaks kubectl name resolution for some cases. See kubernetes/kubectl#48 for details.

[ixdy]

This starts to get tricky in a hurry; e.g. if you want to want to include kubectl in a container, and it's been built with cgo, you now need to make sure you have the correct libc in your container. Similarly, some older linux distributions may have a libc that is too old.

[BenTheElder]

Has anyone mentioned a critical part besides DNS? As far as I've heard there's some hand waving about "critical parts" but it seems this is only DNS so far in my (limited) experience. Perhaps another option is that we can "just" get a better pure-go DNS implementation?

[chrislovecnm]

@BenTheElder we have to shut off cgo DNS in kops for macosx, it was causing problems. Go implementation is working much better. Will need to test that issue.

@pwittrock Was that issue address in go 1.9? I will take a look when I have a chance

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[jkemp101]

@chrislovecnm Do you have anymore info on the issue kops had when using native DNS? Seems like things should work better using the native DNS resolution not worse. This issue is becoming a bigger pain for us as we roll out VPN access to more team members. Some of us are looking at running dnsmasq locally to control name resolution so k8s tools work properly.

[BenTheElder]

Wouldn't it be easier to build your own kubectl binary with cgo?

…

On Fri, Apr 27, 2018, 7:55 AM Joe Kemp ***@***.***> wrote: @chrislovecnm <https://github.com/chrislovecnm> Do you have anymore info on the issue kops had when using native DNS? Seems like things should work better using the native DNS resolution not worse. This issue is becoming a bigger pain for us as we roll out VPN access to more team members. Some of us are looking at running dnsmasq locally to control name resolution so k8s tools work properly. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#469 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA4Bq6_DF2OYQBjgyf7V7NO2hVd_SqODks5tszFRgaJpZM4QpGPA> .

[jkemp101]

@BenTheElder Custom builds of every version of k8s and kops? We obviously don't need every version but it kind of sucks not being able to just download a new release and give it a try. So I compile the "standard" one we will use the most often to administer our clusters and if I need something special I normally "adjust" my kube config so I can run it through a SSH tunnel.

I also don't think hacking build scripts to do custom compiles of my k8s tools is following the best practices to manage production clusters. I should be relying on a build system that runs test and builds the releases in a consistent manner. Running dnsmasq would let me use the same build of the tools everyone else is running.

Can someone point me to the code that actually does the name resolution for normal kubectl commands? I've tried to find it before but get kind of lost. You mentioned using a different DNS implementation. I gave that a try also here miekg/dns#665

[BenTheElder]

@BenTheElder Custom builds of every version of k8s and kops?

You'd only need to compile the binary(ies) that actually run on mac, not all of k8s / kops (IE kubectl, kops cli), which is pretty quick and simpler than running dnsmasq. We also support skewed kubectl so you don't necessarily even need to bulid for every release..

I also don't think hacking build scripts to do custom compiles of my k8s tools is following the best practices to manage production clusters.

You don't need to make any modifications or any real hacking.. One obvious way is make generated_files && CGO_ENABLED=1 go install k8s.io/kubernetes/cmd/kubectl.

I should be relying on a build system that runs test and builds the releases in a consistent manner. Running dnsmasq would let me use the same build of the tools everyone else is running.

Well the build tools run in linux containers (usually on top of kubernetes!)
And everyone actually uses many different builds of kubectl. Just in CI we often use:

• freshly built kubectl
• kubectl built by CI from other release branches (skew testing)
• kubectl fresh installed from the gcloud sdk (which notably only installs the "current" version, because of skew support)
• kubectl baked into the test image

You mentioned using a different DNS implementation.

Er, I was a bit hopeful that the go stdlib implementation should be improved, i'm not sure how trivially DNS can be swapped out in kubectl.

[dims]

@BenTheElder @pwittrock may be this tip will help? golang/go#12503 (comment)

It's possible to intercept DNS requests generated by the net package by modifying the Dial func of DefaultResolver

[jkemp101]

Remember I picked up this thread with a question about what issues kops had when they tried using cgo and ended up switching back to pure go. Not knowing what those issues were and compiling it with cgo anyway to run against my production clusters isn't something I would do without more information.

The whole recompile versus using dnsmasq does highlight an observation I've had in other k8s issues. For operations/infrastructure folks running dnsmasq locally is trivial and might be something they are doing anyway to have fine control over their DNS environment. But a programmer looks at recompiling as the trivial solution versus diving into low level DNS configurations. The easier fix is really based on your background/skills.

[BenTheElder]

If your issue is with kops, their release is entirely separate and an issue should be opened there, FWIW.

…

Remember I picked up this thread with a question about what issues kops had when they tried using cgo and ended up switching back to pure go. Not knowing what those issues were and compiling it with cgo anyway to run against my production clusters isn't something I would do without more information. The whole recompile versus using dnsmasq does highlight an observation I've had in other k8s issues. For operations/infrastructure folks running dnsmasq locally is trivial and might be something they are doing anyway to have fine control over their DNS environment. But a programmer looks at recompiling as the trivial solution versus diving into low level DNS configurations. The easier fix is really based on your background/skills. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#469 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA4Bq_ilNpbLgHt5uam2kwcZono-ZekCks5ts3GPgaJpZM4QpGPA> .