e2e test fail with kind when operator is using pipe tasks

[shubhanilBag]

This PR #201 is failing with

        logger.go:37: 08:05:29 | kafka-upgrade-test | 2020-01-24 08:00:26 +0000 UTC	Normal	Pulling	Pulling image "mesosphere/kafka:1.1.0-2.4.0"
        logger.go:37: 08:05:29 | kafka-upgrade-test | 2020-01-24 08:00:26 +0000 UTC	Normal	Killing	Stopping container kubernetes-zookeeper
        logger.go:37: 08:05:29 | kafka-upgrade-test | 2020-01-24 08:00:37 +0000 UTC	Normal	Pulled	Successfully pulled image "mesosphere/kafka:1.1.0-2.4.0"
        logger.go:37: 08:05:29 | kafka-upgrade-test | 2020-01-24 08:00:37 +0000 UTC	Normal	Created	Created container init
        logger.go:37: 08:05:29 | kafka-upgrade-test | 2020-01-24 08:00:37 +0000 UTC	Normal	Started	Started container init
        logger.go:37: 08:05:29 | kafka-upgrade-test | 2020-01-24 08:00:41 +0000 UTC	Normal	Pulling	Pulling image "busybox"
        logger.go:37: 08:05:29 | kafka-upgrade-test | 2020-01-24 08:00:41 +0000 UTC	Normal	Pulled	Successfully pulled image "busybox"
        logger.go:37: 08:05:29 | kafka-upgrade-test | 2020-01-24 08:00:41 +0000 UTC	Normal	Created	Created container waiter
        logger.go:37: 08:05:29 | kafka-upgrade-test | 2020-01-24 08:00:41 +0000 UTC	Normal	Started	Started container waiter
        logger.go:37: 08:05:29 | kafka-upgrade-test | 2020-01-24 08:00:43 +0000 UTC	Warning	PipeTaskError	Error during execution: fatal error: kudo-test-composed-krill/kafka failed in deploy.deploy-kafka.generate-tls-certificates.generate-tls-certificates: failed to fetch cluster REST config: could not locate a kubeconfig
        logger.go:37: 08:05:29 | kafka-upgrade-test | Deleting namespace: kudo-test-composed-krill

The tests are passing when using a live cluster. KUDO controller is trying to read the file from default kubeconfig location rather than the kubeconfig created by kind.
Also when tried to export KUBECONFIG inside makefile, KUDO is failing to read the kubeconfig file.

[kensipe]

Thanks for the report... however I find it confusing. IS this an issue with pipe tasks or is it an issue with configuration around kind and KUBECONFIG?

[shubhanilBag]

The pipe tasks are working fine, tests are passing in live cluster, it's doing what's promised.
I think it is around how KUDO runs the e2e tests using kind. AFAIU KUDO manager is not running in a pod but as a process as we are only adding the crds ref: https://github.com/kudobuilder/operators/blob/master/kudo-test.yaml#L6.
so maybe when its trying to read the file from init pod, it is creating the rest client by reading the kubeconfig from the default location $HOME/.kube/config not the kind cluster kubeconfig?

[kensipe]

thank you sir

[zen-dog]

Also when tried to export KUBECONFIG inside makefile, KUDO is failing to read the kubeconfig file.

@shubhanilBag what do error do you see when trying that? Because pipe-task relies on controller-runtime getting the kubeconfig, which, however, can be overridden by the --kubeconfig parameter (or KUBECONFIG env var):

// GetConfig creates a *rest.Config for talking to a Kubernetes API server.
// If --kubeconfig is set, will use the kubeconfig file at that location.  Otherwise will assume running
// in cluster and use the cluster provided kubeconfig.
//...
// Config precedence
//
// * --kubeconfig flag pointing at a file
//
// * KUBECONFIG environment variable pointing at a file
//
// * In-cluster config if running in cluster
//
// * $HOME/.kube/config if exists
func GetConfig() (*rest.Config, error) {...}

[shubhanilBag]

Yeah, earlier it was unable to read the kubeconfig from the exported path when I was running make test locally, I removed the existing file and tried again, its working now. PR #201 is passing with 165c93f, thanks 👍
but is it right solution? is this how it should be done?

[zen-dog]

Glad to hear it worked!

[porridge]

The workaround in https://github.com/kudobuilder/operators/pull/201/files#diff-b67911656ef5d18c4ae36cb6741b7965R15 only helped for CI for the operators repo.
Now we have the same breakage in the kudo e2e CI test.
Arguably, this should be fixed in a single place.

[porridge]

Also, setting KUBECONFIG in the Makefile like it was done in https://github.com/kudobuilder/operators/pull/201/files#diff-b67911656ef5d18c4ae36cb6741b7965R15 seems rather fragile, since it creates an semantic coupling with how kind determines where to put the config file.