Skip to content

Allow using kubectl --context for cluster credentials #43

@msiebuhr

Description

@msiebuhr

Unless one explicitly sets user in k8s_deploy(), it is assumed a username matching the cluster can be found:

user_arg = """$(kubectl config view -o jsonpath='{.users[?(@.name == '"\\"${CLUSTER}\\")].name}")"""

(My jsonpath-foo isn't strong, but I read this as "return the .name of the user who's .name == $CLUSTER - I feel I'm missing something obvious here?)

On our setup this fails as we sometimes have multiple users per cluster. We don't like running with admin-powers on by default, so everyone has <username>-<clustername>, and some has admin-<clustername> as backups. The query above returns an empty string on our setups, so things only work when the user happens to be in the right kubernetes context, so the blank user-name doesn't make a difference:

kubectl config use-context $USER-$OTHER_CLUSTER
bazel run :deploy-to-kubernetes
...
error: error validating "STDIN": error validating data: failed to download openapi: the server has asked for the client to provide credentials; if you choose to ignore these errors, turn validation off with --validate=false

kubectl config use-context $USER-$CORRECT_CLUSTER
bazel run :deploy-to-kubernetes
...
service/xxxx unchanged
deployment.apps/xxxx configured
sealedsecret.bitnami.com/xxxx unchanged
ingress.networking.k8s.io/xxxx unchanged

Kubernetes config does have contexts tying users, clusters and - optionally - namespaces together (which is also what rules_k8s relies on):

k8s_deploy(
    # ...
    context = "morten-siebuhr-some-cluster-name"
)

Which is then passed to kubectl --context $CONTEXT?

Docs: https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions