Basic contextualization and orchestration
In this lab you will learn the basics of how to automatically configure instances and how to automate and orchestrate deployment of single resources as well as more complex stacks.
Estimated time needed to complete the entire lab: 3-5 hours.
The aim of this tutorial is to give you hands-on experience with the service contextualization and orchestration. The lab will provide a brief introduction to different tools that can be used for efficient resource-provisioning. This forms a foundation for automated, robust and reprodicible resource management in the SNIC Science Cloud. SSC is based on the Liberty release of OpenStack and offers the follwing core services:
- Compute (Nova)
- Storage (Ephemeral, Cinder)
- Identity management (KeyStone)
- Image (Glance)
- Network (Neutron)
- Orchestration (Heat)
- Object Storage (Swift)
Together with this lab description, this lab is based on code in the following tasks:
- Task 1: OpenStack-API
- Task 2: CloudInit-Contextualization
- Task 3: Ansible-Contextualization
- Task 4: Heat-Orchestration
- Task 5: Containers
Please follow the instructions, execute the tasks and answer the related questions.
- Information page: http://cloud.snic.se
- User Guide: https://docs.openstack.org/horizon/pike/user/ , https://docs.openstack.org/python-novaclient/pike/
- SNIC Science Cloud (SSC): https://cloud.snic.se
Task-0: Setting the environment for API access
Install Openstack libraries on your test virtual machine,
- Goto https://docs.openstack.org/install-guide/environment-packages-ubuntu.html , http://docs.openstack.org/cli-reference/common/cli_install_openstack_command_line_clients.html and download the client tools and API for OpenStack.
- Download the Runtime Configuration (RC) file from the SSC site (Project->Compute->Access & Security->API Access->Download OpenStack RC File).
- Confirm that your RC file have following enviroment variables:
export OS_USER_DOMAIN_NAME="snic" export OS_IDENTITY_API_VERSION="3" export OS_PROJECT_DOMAIN_NAME="snic"
- Set the environment variables by sourcing the RC-file:
- Run the following commands and explain the output:
openstack server list openstack image list
- What version of the API are we using?
- Explain how the communication works in OpenStack?
- Can we use EC2 and S3 APIs to communicate with OpenStack?
Task-1: Resource provisioning using CLIs and APIs
Use the command-line interface (CLI) tools as well as the python APIs (example coode is available in OpenStack-API directory) to achieve following tasks:
Note: you need to edit the Python files (in the OpenStack-API folder) and enter your key-name, private-network and floating IP information.
- List available images
- List running instances
- Boot a new instance
- Modify the code to boot multiple instances
- Attach a floating IP and connect via SSH
- Write python code to terminate the instance
- Contextualize the instance at boot time. Code is available in “nova/add-userdata” directory
- What is the difference between the private IP and the floating IP?
- Can you access the Internet from the VM without assigning a floating IP to the machine?
- Explain the arguements to the function used in Python to boot a single instance?
Task-2: Single-machine contextualization
NOTE: The code assumes Ubuntu VMs. The code for this task is in the folder "contextualization".
The following link provides an introduction to Cloudinit
In this task you will prepare a single instance to install packages and start a web-service “cowsay” on the instance. The configuration is done by CloudInit package. Run the code, It will prepare the instance at boot time. Read and try to understand cloud-cfg.txt that you can find in the “contextualization” folder.
The file ssc-instance-user-data.py , in the same folder is used to start the instance. Open the file, try to understand it. The python file is not complete, make the necessary changes, run the file and see if your instance is created. Test that the server is working by executing following curl command:
curl -i http://<your_public_ip>:5000/cowsay/api/v1.0/saysomething
If you are using Windows, use a Linux VM or install a cURL client for Windows.
- Explain the output?
- What is contextualization?
- What language is use to prepare CloudInit configurations?
- What are the variants of CloudInit package?
- Can we run CloudInit scripts without booting an instance?
- What limitation you can anticipate with the CloudInit package?
Task-3: Cluster contextualization
NOTE: The code assumes Ubuntu VMs.
In this task we will configure a Spark cluster using Ansible. Ansible is an open source IT-automation engine that can be used to automate provisioning, configuration and deployment of cloud resurces. First, start two virtual machines based on Ubuntu:
copy the “ansible-spark” directory to the ansible-node. Run the “install_ansible.sh” script. Then configure the Ansible setup with the following steps (all executed on the ansible-node):
- Generate a SSH key-pair, # ssh-keygen –t rsa
- Add the public key to both of the hosts “authorized_keys” file.
- Make sure you can login to the machines. Also check that the user account has the “sudo” privileges.
- Open the ““/home/ubuntu/spark/hosts” file and fill the private IP addresses of the nodes.
- Goto “/home/ubuntu/spark” directory and execute the following command:
ansible-playbook -i hosts -s playbooks/spark-deployment.yml
The complete deployment will take approximately 20 to 30 minutes. Once the installation is finished you can check the cluster status using the following URL:
- What language is used by Ansible to define the configurations?
- Can Ansible be used with different Linux distributions?
- What is Ansible inventory file?
- What is the major difference between CloudInit vs Ansible-based resource contextualization?
- Explain how Ansible works and suggest, within the scope of this task, what is still left to automate?
Task-4: Orchestration using Heat
In this task you will create a cluster of two machines using the Heat engine. Heat is OpenStack's native orcherstation engine and will let you automate the deployment of complex resources/stacks. The cluster will be completely customized by having its own network, security group, router settings and resources. Complete the following steps to provide the required information in the template file :
- Enter your personal key name by replacing the “key” section’s default value.
- Generate a SSH key-pair, # ssh-keygen –t rsa
- Replace “<ADD-CLUSTER’s-PUBLIC-KEY>” with the public part of the generated key pair. Replace it for both of the instances.
- Run the command
openstack stack create stack_with_init_script -f ‘yaml’ -t ssc-test-stack.yml
- Open the SSC dashboard and click on the Orchestration and check the status of your stack.
- What language is used with the Heat service to define the templates?
- What are the advantages of using templates rather than the APIs?
- Explain the different sections in the templates?
Task-5: Introduction to Linux Containers
This task will introduce you to Linux containers. There are different technologies available but in this Lab, we will focus on Docker containers. Your task is to build and run CSaaS service using Docker containers.
Step-1: Install Docker on your VM.
0 - Switch to the root user:
> sudo bash
1 - First, add the GPG key for the official Docker repository to the system:
# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
2 - Add the Docker repository to APT sources:
# add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
3 - Update the package database with the Docker packages from the newly added repo:
# apt-get update
4 - Install Docker:
# apt-get install -y docker-ce
5 - (Optional) Docker should now be installed, the daemon started, and the process enabled to start on boot. Check that it's running:
# systemctl status docker
For more information visit:
Step - 2: Use the Dockerfile available in the repository (container directory) and build your own container.
1 - Go to the container directory
# cd container
2 - Execute the docker build and run commands
# docker build -t cowsay:latest .
# docker run -it cowsay
# docker run -d -p 5000:5000 cowsay
Step - 3: Test that service is available by executing (from your client)
$ curl -i http://<your_public_ip>:5000/cowsay/api/v1.0/saysomething
- In what category of virtualization do containers fall?
- What are the other frameworks that provide container technology. Write at least two name.
- Explain the provided Dockerfile. What does it do? How does it work? Write a brief (one line) description about each line in the Dockerfile.
- Write a brief (one line) description about each command used in Step-2-2.
- What is dockerhub? Write a brief description of how can we use dockerhub for our newly build CSaaS container?
- Write a CloudInit script that contextualize a VM based on the steps (Step-1 and 2) mentioned in this task. Submit the script with your assignment report.
Where to go from here?
Why not check out how to create Kubernetes cluster using KubeNow?