Skip to content
Permalink
master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

ansible-spark-livy/README.md

Go to file
 
 
Cannot retrieve contributors at this time
Ansible Spark Livy Summary Requirements How to run Check the running containers Test Spark and Livy Livy Livy & Spark UI Spark-Shell License
155 lines (116 sloc) 6.16 KB
Raw Blame
Open in GitHub Desktop

Ansible Spark Livy

License

Summary

This repository contains a simple ansible playbook to install a Spark Standalone cluster and Livy in docker

Requirements

Package Version
python3 3.8.5
Ansible 2.9.6

How to run

renien@renien-VirtualBox:~/projects/personal/ansible-spark-livy/playbooks$ ansible-playbook --inventory host.inv application_playbook.yml --connection=local

PLAY [local_host] *************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************
ok: [127.0.0.1]

TASK [docker : Install dependencies] ******************************************************************************************
changed: [127.0.0.1]

TASK [docker : add key] *******************************************************************************************************
[WARNING]: Consider using the get_url or uri module rather than running 'curl'.  If you need to use command because get_url or
uri is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid
of this message.
changed: [127.0.0.1]

TASK [docker : add repository] ************************************************************************************************
changed: [127.0.0.1]

TASK [docker : Install docker] ************************************************************************************************
ok: [127.0.0.1]

TASK [docker : Install docker-compose] ****************************************************************************************
changed: [127.0.0.1]

TASK [docker : setup log rotation] ********************************************************************************************
changed: [127.0.0.1]

TASK [docker-livy-spark : check docker-compose version] ***********************************************************************
changed: [127.0.0.1]

TASK [docker-livy-spark : debug docker_compose_version] ***********************************************************************
ok: [127.0.0.1] => {
"docker_compose_version": {
"changed": true,
"cmd": "docker-compose --version",
"delta": "0:00:00.427471",
"end": "2021-01-22 16:02:08.287592",
"failed": false,
"rc": 0,
"start": "2021-01-22 16:02:07.860121",
"stderr": "",
"stderr_lines": [],
"stdout": "docker-compose version 1.27.4, build 40524192",
"stdout_lines": [
"docker-compose version 1.27.4, build 40524192"
]
}
}

TASK [docker-livy-spark : Find out playbook's path] ***************************************************************************
changed: [127.0.0.1]

TASK [docker-livy-spark : debug] **********************************************************************************************
ok: [127.0.0.1] => {
"msg": "'/home/renien/projects/personal/ansible-spark-livy/playbooks' is the playbook directory"
}

TASK [docker-livy-spark : copy the docker folder spark-image/docker-livy-spark] ***********************************************
changed: [127.0.0.1]

TASK [docker-livy-spark : Run docker-compose up] ******************************************************************************
changed: [127.0.0.1]

PLAY RECAP ********************************************************************************************************************
127.0.0.1                  : ok=13   changed=9    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Check the running containers

root@renien-VirtualBox:~/spark-image/docker-livy-spark# docker-compose ps
     Name                   Command               State                                                            Ports                                                          
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
livy             /bin/sh -c ./entrypoint.sh ...   Up      0.0.0.0:4040->4040/tcp, 0.0.0.0:8998->8998/tcp                                                                          
spark-master     bin/spark-class org.apache ...   Up      0.0.0.0:6066->6066/tcp, 7001/tcp, 7002/tcp, 7003/tcp, 7004/tcp, 7005/tcp, 0.0.0.0:7077->7077/tcp, 0.0.0.0:8080->8080/tcp
spark-worker-1   bin/spark-class org.apache ...   Up      7012/tcp, 7013/tcp, 7014/tcp, 7015/tcp, 0.0.0.0:8081->8081/tcp, 8881/tcp

Test Spark and Livy

Livy

# Create a spark session using livy
curl -X POST --data '{"kind": "spark"}' -H "Content-Type: application/json" localhost:8998/sessions

# Test a simple login in spark session
curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"2 + 2"}'
Livy & Spark UI

Livy UI

Spark Master

Spark Worker

Spark-Shell

renien@renien-VirtualBox:~$ sudo docker exec -it spark-master /bin/bash
root@master:/usr/spark-2.4.7# spark-shell
2021-01-22 10:52:05,836 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = spark://master:7077, app id = app-20210122105212-0001).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.7
      /_/
         
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_271)
Type in expressions to have them evaluated.
Type :help for more information.

scala> 2 + 2
res0: Int = 4

scala>

License

Ansible Spark Livy is released under the MIT © Renien.