Skip to content
No description, website, or topics provided.
Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
LICENSE
README.md
build-spark.sh
namespace-spark-cluster.yaml
spark-master-controller.yaml
spark-master-service.yaml
spark-ui-proxy-controller.yaml
spark-ui-proxy-service.yaml
spark-worker-controller.yaml
zeppelin-controller.yaml
zeppelin-service.yaml

README.md

Spark on Kubernetes

如何在kubernetes上部署spark

Kubernetes 示例github上提供了一個詳細的spark部署方法,由於他的步驟設置有些覆雜, 這邊簡化一些部份讓大家安裝的時候不用去多設定一些東西。

部署條件

  • 一個kubernetes群集,可參考集群部署
  • kube-dns正常運作

創建一個命名空間

namespace-spark-cluster.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: "spark-cluster"
  labels:
    name: "spark-cluster"
$ kubectl create -f examples/staging/spark/namespace-spark-cluster.yaml

這邊原文提到需要將kubectl的執行環境轉到spark-cluster,這邊為了方便我們不這樣做,而是將之後的佈署命名空間都加入spark-cluster

部署Master Service

建立一個replication controller,來運行Spark Master服務

kind: ReplicationController
apiVersion: v1
metadata:
  name: spark-master-controller
  namespace: spark-cluster
spec:
  replicas: 1
  selector:
    component: spark-master
  template:
    metadata:
      labels:
        component: spark-master
    spec:
      containers:
        - name: spark-master
          image: gcr.io/google_containers/spark:1.5.2_v1
          command: ["/start-master"]
          ports:
            - containerPort: 7077
            - containerPort: 8080
          resources:
            requests:
              cpu: 100m
$ kubectl create -f spark-master-controller.yaml

創建master服務

spark-master-service.yaml

kind: Service
apiVersion: v1
metadata:
  name: spark-master
  namespace: spark-cluster
spec:
  ports:
    - port: 7077
      targetPort: 7077
      name: spark
    - port: 8080
      targetPort: 8080
      name: http
  selector:
    component: spark-master
$ kubectl create -f spark-master-service.yaml

檢查Master 是否正常運行

$ kubectl get pod -n spark-cluster 
spark-master-controller-qtwm8     1/1       Running   0          6d
$ kubectl logs spark-master-controller-qtwm8 -n spark-cluster 
17/08/07 02:34:54 INFO Master: Registered signal handlers for [TERM, HUP, INT]
17/08/07 02:34:54 INFO SecurityManager: Changing view acls to: root
17/08/07 02:34:54 INFO SecurityManager: Changing modify acls to: root
17/08/07 02:34:54 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
17/08/07 02:34:55 INFO Slf4jLogger: Slf4jLogger started
17/08/07 02:34:55 INFO Remoting: Starting remoting
17/08/07 02:34:55 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkMaster@spark-master:7077]
17/08/07 02:34:55 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
17/08/07 02:34:55 INFO Master: Starting Spark master at spark://spark-master:7077
17/08/07 02:34:55 INFO Master: Running Spark version 1.5.2
17/08/07 02:34:56 INFO Utils: Successfully started service 'MasterUI' on port 8080.
17/08/07 02:34:56 INFO MasterWebUI: Started MasterWebUI at http://10.2.6.12:8080
17/08/07 02:34:56 INFO Utils: Successfully started service on port 6066.
17/08/07 02:34:56 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066
17/08/07 02:34:56 INFO Master: I have been elected leader! New state: ALIVE

若master 已經被建立與運行,我們可以透過Spark開發的webUI來察看我們spark的群集狀況,我們將佈署specialized proxy

spark-ui-proxy-controller.yaml

kind: ReplicationController
apiVersion: v1
metadata:
  name: spark-ui-proxy-controller
  namespace: spark-cluster
spec:
  replicas: 1
  selector:
    component: spark-ui-proxy
  template:
    metadata:
      labels:
        component: spark-ui-proxy
    spec:
      containers:
        - name: spark-ui-proxy
          image: elsonrodriguez/spark-ui-proxy:1.0
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: 100m
          args:
            - spark-master:8080
          livenessProbe:
              httpGet:
                path: /
                port: 80
              initialDelaySeconds: 120
              timeoutSeconds: 5
$ kubectl create -f spark-ui-proxy-controller.yaml

提供一個service做存取,這邊原文是使用LoadBalancer type,這邊我們改成NodePort,如果你的kubernetes運行環境是在cloud provider,也可以參考原文作法

spark-ui-proxy-service.yaml

kind: Service
apiVersion: v1
metadata:
  name: spark-ui-proxy
  namespace: spark-cluster
spec:
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30080
  selector:
    component: spark-ui-proxy
  type: NodePort
$ kubectl create -f spark-ui-proxy-service.yaml

部署完後你可以利用kubecrl proxy來察看你的Spark群集狀態

$ kubectl proxy --port=8001

可以透過http://localhost:8001/api/v1/proxy/namespaces/spark-cluster/services/spark-master:8080 察看,若kubectl中斷就無法這樣觀察了,但我們再先前有設定nodeport 所以也可以透過任意台node的端口30080去察看 例如:http://10.201.2.34:30080 10.201.2.34是群集的其中一台node,這邊可換成你自己的

部署 Spark workers

要先確定Matser是再運行的狀態

spark-worker-controller.yaml

kind: ReplicationController
apiVersion: v1
metadata:
  name: spark-worker-controller
  namespace: spark-cluster
spec:
  replicas: 2
  selector:
    component: spark-worker
  template:
    metadata:
      labels:
        component: spark-worker
    spec:
      containers:
        - name: spark-worker
          image: gcr.io/google_containers/spark:1.5.2_v1
          command: ["/start-worker"]
          ports:
            - containerPort: 8081
          resources:
            requests:
              cpu: 100m
$ kubectl create -f spark-worker-controller.yaml
replicationcontroller "spark-worker-controller" created

透過指令察看運行狀況

$ kubectl get pod -n spark-cluster 
spark-master-controller-qtwm8     1/1       Running   0          6d
spark-worker-controller-4rxrs     1/1       Running   0          6d
spark-worker-controller-z6f21     1/1       Running   0          6d
spark-ui-proxy-controller-d4br2   1/1       Running   4          6d

也可以透過上面建立的WebUI服務去察看

基本上到這邊Spark的群集已經建立完成了

創建 Zeppelin UI

我們可以利用Zeppelin UI經由web notebook直接去執行我們的任務, 詳情可以看Zeppelin UI Spark architecture

zeppelin-controller.yaml

kind: ReplicationController
apiVersion: v1
metadata:
  name: zeppelin-controller
  namespace: spark-cluster
spec:
  replicas: 1
  selector:
    component: zeppelin
  template:
    metadata:
      labels:
        component: zeppelin
    spec:
      containers:
        - name: zeppelin
          image: gcr.io/google_containers/zeppelin:v0.5.6_v1
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 100m
$ kubectl create -f zeppelin-controller.yaml
replicationcontroller "zeppelin-controller" created

然後一樣佈署Service

zeppelin-service.yaml

kind: Service
apiVersion: v1
metadata:
  name: zeppelin
  namespace: spark-cluster
spec:
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 30081
  selector:
    component: zeppelin
  type: NodePort
$ kubectl create -f zeppelin-service.yaml

可以看到我們把NodePort設再30081,一樣可以透過任意台node的30081 port 訪問 zeppelin UI。

通過命令行訪問pyspark(記得把pod名字換成你自己的):

$ kubectl exec -it zeppelin-controller-8f14f -n spark-cluster pyspark
Python 2.7.9 (default, Mar  1 2015, 12:57:24) 
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
17/08/14 01:59:22 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 1.5.2
      /_/

Using Python version 2.7.9 (default, Mar  1 2015 12:57:24)
SparkContext available as sc, HiveContext available as sqlContext.
>>> 

接著就能使用Spark的服務了,如有錯誤歡迎更正。

zeppelin常見問題

  • zeppelin的鏡像非常大,所以再pull時會花上一些時間,而size大小的問題現在也正在解決中,詳情可參考 issue #17231
  • 在GKE的平台上, kubectl post-forward 可能有些不穩定,如果你看現zeppelin 的狀態為Disconnected,port-forward可能已經失敗你需要去重新啟動它,詳情可參考 #12179

參考文檔

You can’t perform that action at this time.