[SPARK-28534][K8S][TEST] Update node affinity for DockerForDesktop backend in PVTestsSuite

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to recover our K8s integration test suite by extending node affinity in order to pass PVTestsSuite in DockerForDesktop environment, too. Previously, PVTestsSuite fails at --deploy-mode docker-for-desktop option because the node affinity requires minibase node.

For Docker Desktop, there are two node names like the following. Note that Spark testing needs K8s v1.13 and above. So, this PR should be verified with Docker Desktop (Edge) version. This PR adds both because next stable Docker Desktop will have K8s v1.14.3.

Docker Desktop (Stable, K8s v1.10.11)

$ kubectl get node
NAME                 STATUS   ROLES    AGE   VERSION
docker-for-desktop   Ready    master   52s   v1.10.11

Docker Desktop 2.1.0.0 (Edge, K8s v1.14.3, Released 2019-07-26)

$ kubectl get node
NAME             STATUS   ROLES    AGE   VERSION
docker-desktop   Ready    master   16h   v1.14.3

How was this patch tested?

Pass the Jenkins K8s integration test (minibase) and install Docker Desktop 2.1.0.0 (Edge) and run the integration test in DockerForDesktop. Note that this fixes only PVTestsSuite.

$ dev/make-distribution.sh --pip --tgz -Phadoop-2.7 -Pkubernetes
$ resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh --deploy-mode docker-for-desktop --spark-tgz $PWD/spark-*.tgz
...
KubernetesSuite:
...
- PVs with local storage
...

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/13341/

[SparkQA]

Test build #108240 has finished for PR 25269 at commit 722965f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/13341/

[dongjoon-hyun]

cc @dbtsai

[dongjoon-hyun]

Hi, @HyukjinKwon and @maropu .
Could you review this PR (which is recovering K8s integration failures), please?

[maropu]

I run the test, but I couldn't see it passed.
Since I'm not familiar with this test, I might miss some setups for running it, probably...
I used the Docker Desktop(stable) Version 2.0.0.3 (31259);

(base) maropu@: (pr25269 *$%)$kubectl get node
NAME                 STATUS    ROLES     AGE       VERSION
docker-for-desktop   Ready     master    1h        v1.10.11

$ dev/make-distribution.sh --pip --tgz -Phadoop-2.7 -Pkubernetes
$ resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh --deploy-mode docker-for-desktop --spark-tgz $PWD/spark-*.tgz
...
- PVs with local storage *** FAILED ***
  The code passed to eventually never returned normally. Attempted 69 times over 2.0167555011666667 minutes. Last failure message: pod spark-test-app-853de4ad66e041ae941c441c08ae4822 does not have a host assigned. (PVTestsSuite.scala:117)

It seems the pod was created successfully;

(base) maropu@~/Desktop/spark: (master %=)$kubectl get po --all-namespaces
NAMESPACE                          NAME                                              READY     STATUS    RESTARTS   AGE
98d0b44c7a2f49828bf9a2a97c07a786   spark-test-app-853de4ad66e041ae941c441c08ae4822   0/1       Pending   0          2m

[dongjoon-hyun]

Hi, @maropu . Thank you for testing.
It's a K8s version issue for PVC feature. We need K8s 1.13.x and above. So, you should use Docker Desktop 2.1.0.0 (Edge) as described in the How was this patch tested? section. The next Stable version will be replaced by this Edge version.

[maropu]

@dongjoon-hyun oh, yeah. Thanks for the explanation! I'll try later.

[dongjoon-hyun]

Thank you. Sorry for wasting your time due to the ambiguity. I'll update the PR.

[dongjoon-hyun]

I updated the PR description~

[HyukjinKwon]

Tested as guided:

KubernetesSuite:
...
- PVs with local storage

Merged to master.

[dongjoon-hyun]

Thank you so much, @HyukjinKwon and @maropu !