e2e tests in CI don't seem to finish

[imjasonh]

Describe the bug

https://github.com/kcp-dev/kcp/runs/4156803357?check_suite_focus=true
(previous attempt)

To Reproduce

1. Add the check-demos label to a PR
2. Wait
3. Observe the "Build and Run Demos" workflow doesn't complete (I cancelled after 41 minutes)

There also don't appear to be a lot of useful logs, though maybe GitHub is swallowing those for us.

Expected behavior

The test should finish (ideally pass) after a few minutes, which is how long it takes the demo script to run in real life.

Additional context

• workflow config: https://github.com/kcp-dev/kcp/blob/main/.github/workflows/demos.yaml
• PR that added the config: Launch demos tests with PR labeling #181 (when it was last seen to pass)

[ncdc]

Testing locally against main, it seems to be the kubecon demo that fails, here:

pe "kubectl apply -f ${DEMO_DIR}/deployment.yaml -n demo"

until [[ "$(kubectl get deployments/my-deployment -n demo -o=jsonpath='{.status.updatedReplicas}')" == "10" ]]; do sleep 3; done

Output from the log:

kubectl apply -f /Users/angoldst/code/kcp-dev/kcp/contrib/demo/deployment.yaml -n demo
error: unable to recognize "/Users/angoldst/code/kcp-dev/kcp/contrib/demo/deployment.yaml": no matches for kind "Deployment" in version "apps/v1"
error: the server doesn't have a resource type "deployments"
error: the server doesn't have a resource type "deployments"
......

[ncdc]

Aha:

head -n 15 /Users/angoldst/code/kcp-dev/kcp/contrib/demo/clusters/kind/us-west1.yaml
head: /Users/angoldst/code/kcp-dev/kcp/contrib/demo/clusters/kind/us-west1.yaml: No such file or directory
☸️ $ kubectl apply -f /Users/angoldst/code/kcp-dev/kcp/contrib/demo/clusters/kind/us-west1.yaml
error: the path "/Users/angoldst/code/kcp-dev/kcp/contrib/demo/clusters/kind/us-west1.yaml" does not exist
☸️ $ head -n 15 /Users/angoldst/code/kcp-dev/kcp/contrib/demo/clusters/kind/us-east1.yaml

[ncdc]

The files are 1 level up (not in the kind subdir)

[ncdc]

Fixed that locally. Now seeing this in the cluster controller log:

I1109 16:02:43.222315   74935 cluster.go:27] reconciling cluster "us-west1"
E1109 16:02:43.223471   74935 cluster.go:34] invalid kubeconfig: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
I1109 16:02:43.267696   74935 controller.go:246] Successfully reconciled "admin#$#us-west1"
I1109 16:02:43.267767   74935 cluster.go:27] reconciling cluster "us-west1"
E1109 16:02:43.267820   74935 cluster.go:34] invalid kubeconfig: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
I1109 16:02:43.267846   74935 controller.go:246] Successfully reconciled "admin#$#us-west1"
I1109 16:02:49.126212   74935 cluster.go:27] reconciling cluster "us-east1"
E1109 16:02:49.154557   74935 cluster.go:34] invalid kubeconfig: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
I1109 16:02:49.195204   74935 controller.go:246] Successfully reconciled "admin#$#us-east1"
I1109 16:02:49.219336   74935 cluster.go:27] reconciling cluster "us-east1"
E1109 16:02:49.219410   74935 cluster.go:34] invalid kubeconfig: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
I1109 16:02:49.219442   74935 controller.go:246] Successfully reconciled "admin#$#us-east1"

[ncdc]

Whoops, looks like an issue with my local env. Ignore for now!

[ncdc]

@sttts (revisiting) do you want to maintain our demos so they're consistently functional? If so, we should fix this (if it's still broken) and make it a required check on all PRs. If not, we should remove the check-demos label and workflow.

[ncdc]

Closing as we have decided not to maintain demo scripts in the repo