Hermetically include kubectl in rules_k8s

[achew22]

Rather than depending on the kubectl that is available on the system, it would be nifty to be hermetic and to compile kubectl. Since kubernetes is using Bazel and Gazelle to manage things, I believe you could just add it as a git_repository (it vendors all its deps so no need to add more) and depend on @com_github_kubernetes_kubernetes//cmd/kubectl as a data dependency and then shell out to external/com_github_kubernetes_kubernetes/cmd/kubectl/kubectl instead. A bash wrapper that checked for the existence of that file and called it and fell back to the system version if you didn't want to force the dependency.

[achew22]

Something like this (except it is blocked by a visibility issue in Bazel)

https://github.com/achew22/rules_k8s/tree/hermetic

[mattmoor]

@achew22 Perhaps, although that may make authentication tricky. I don't really know enough about kubectl's local configuration to say. I think I would probably have a stronger bias towards downloading a built version of kubectl than rebuilding it from source all of the time.

One of my biggest pet peeves about Bazel's current WORKSPACE model is the absence of a first-class keychain and the recommended implicit dependency on preconfigured auth.

[achew22]

@mattmoor, I think I'm a little confused. Aren't the rules_k8s targets all bazel run? My understanding was that build/test both used the sandbox but that once you bazel run something you're not living in the sandbox any more -- whatever system utilities you want/need are available. Hence kubectl being accessible today. This would just make a hermetic kubectl binary available at run-time so that you didn't have to preinstall kubectl on your CI server. Can you help me understand what I'm missing?

[hsyed]

2cp: I Bazelified the kubecfg tool and it can be executed via @kubecfg//:kubecfg. I used the vendored=external gazelle approach.

git_repository(
    name = "io_bazel_rules_go",
    remote = "https://github.com/bazelbuild/rules_go.git",
    commit = "b3b0d1ef387fde7b0560545ee4cf2ffd47f32796",
)
load("@io_bazel_rules_go//go:def.bzl", "go_rules_dependencies", "go_register_toolchains")

go_rules_dependencies()
go_register_toolchains()

git_repository(
    name = "kubecfg",
    remote="git@github.com:hsyed/kubecfg.git",
    commit = "a058312d192e9a0b445141a6998fce9b38c35bd1"
)

vmware-archive/kubecfg#133

[mattmoor]

@achew22 Yes, today all the interesting actions are done on bazel run. My point was even if we made it so that a CI setup didn't need to include kubectl (e.g. our own .travis.yml), we would still need to set up kubectl authentication for this to work (e.g. our own .travis.yml).

@hsyed does kubecfg perform any remote actions? If so, how are you dealing with auth?

[hsyed]

@mattmoor kubecfg is basically a wrapper to jsonnet and kubectl. It conventionally turns the multiple outputs of a single jsonnet file into a multi-doc yaml file and then applies that to kubernetes via apply. It picks up credentials from the ~/.kube/config but has switches to control the context / config.

For now I assume the build environment provides a ~/.kube/config file outside of bazel.

[mariusgrigoriu]

Would it be possible to select which version of kubectl you would get? What if you need a newer version of rules_k8s, but you're running an older cluster and there's an API incompatibility with the client?

[nlopezgi]

It is now possible to pull a prebuilt kubectl binary or build it from source. Details are here: https://github.com/bazelbuild/rules_k8s/tree/master/toolchains/kubectl
Please let e know to reopen if any use cases are not covered by this solution