spark-operator test is flaky

[jlewi]

spark-operator test is flaky.

+ echo Command failed: ks apply default -c spark-operator
+ for i in '$(seq 1 3)'
+ set +e
+ ks apply default -c spark-operator
Command failed: ks apply default -c spark-operator
level=error msg="find objects: open /mnt/test-data-volume/kubeflow-presubmit-kfctl-2676-311d2a5-6257-019e/kfctl-019e/ks_app/environments/default/params.li
bsonnet: no such file or directory"
Command failed: ks apply default -c spark-operator
command didn't succeed
+ result=1

Here's a screen shot of the graph

It looks like there is a race condition; we try to apply the spark operator before kfctl apply is finished.

I believe ks apply deletes and recreates the environment. I wonder if this could cause this particular error.

[jlewi]

Here are logs from the kfctl apply k8s step that shows that the environment is created during that step.

 createKsEnv
+ pushd /mnt/test-data-volume/kubeflow-presubmit-kfctl-2676-311d2a5-6257-019e/kfctl-019e/ks_app
+ set +e
/mnt/test-data-volume/kubeflow-presubmit-kfctl-2676-311d2a5-6257-019e/kfctl-019e/ks_app /mnt/test-data-volume/kubeflow-presubmit-kfctl-2676-311d2a5-6257-0
19e/kfctl-019e/ks_app /mnt/test-data-volume/kubeflow-presubmit-kfctl-2676-311d2a5-6257-019e/kfctl-019e
++ ks env describe default
+ O='level=error msg="environment \"default\" was not found"'
+ RESULT=1
+ set -e
+ '[' 1 -eq 0 ']'
+ ks env add default --namespace kubeflow
level=info msg="Using context \"kfctl-019e\" from kubeconfig file \"/mnt/test-data-volume/kubeflow-presubmit-kfctl-2676-311d2a5-6257-019e/kfctl_test/.kube
/kubeconfig\""
level=info msg="Creating environment \"default\" with namespace \"kubeflow\", pointing to cluster at address \"https://35.229.52.71\""
+ popd