Welcome to the Ops Track of the OpenShift + Ansible Better Together lab! Today you will learn about how Ansible can be used with OpenShift in order to aid in the operation and scalability of OpenShift.
You are the lead engineer of the OpenShift team and you have gotten through the day 1 install. Developer teams are excited to start getting on to the platform. This is exciting but daunting because you are the only engineer dedicated to support the cluster and need to be able create workspaces for the teams in a quick, repeatable way. Never fear ansible operators are here to the rescue!
|
Note
|
##TODO: explain user env with backpack |
Each workshop participant is provisioned their own OpenShift Container Platform 4 cluster. Each cluster contains one master node, one infrastructure node, one application node, and one bastion host. You will be working from the bastion host to complete today’s lab.
First, you’ll need to claim your OpenShift cluster using our [Cluster Assignment Tool](https://bit.ly/2w3Jnh8). Once you get to the cluster assignment tool page, you’ll need two pieces of information:
-
Lab Code:
SummitVirtual - BetterTogether: Ansible + OpenShift -
Activation Key:
ansible+openshift -
Email Address:
Your email
Once you enter the information into the cluster assignment tool, you’ll receive a few pieces of information. It is important to keep THIS window open during the workshop.
The first (and one of the most important) pieces of information you will receive is a "GUID" in the format of rht-<4_RANDOM_CHARACTERS> (for example, rht-j1e2). It is important to keep this "GUID" handy for the rest of the lab.
Next, in order to SSH to the bastion host, you’ll need to download the private SSH key from the link provided in the cluster assignment tool.
For users who are running Linux, macOS, or Windows with the Windows Subsystem for Linux installed, open up a terminal and run the following commands:
$ export GUID=<ASSIGNED GUID>
chmod 600 <PATH_TO_WHERE_PRIVATE_KEY_WAS_DOWNLOADED>/ocp-workshop.pem
ssh -i <PATH_TO_WHERE_PRIVATE_KEY_WAS_DOWNLOADED>/ocp-workshop.pem ec2-user@bastion.${GUID}.open.redhat.comFor users who are running Windows and using PuTTY for SSH, follow the below directions:
-
Open PuTTY.
-
In PuTTY, under Category on the left, navigate to Connection → SSH → Auth.
-
On the right under Authentication parameters, click Browse and locate the private key (ocp-workshop.ppk) you saved earlier.
-
On the left, navigate to Session.
-
On the right in the Host Name field, ec2-user@bastion.${GUID}.open.redhat.com
-
Click Open.
-
When prompted with the security alert, click Yes.
Once you establish an SSH session, you’ll need to change to the root user, followed by setting environment variables to reference your GUID and API Server, as these will be referenced a few times throughout this lab:
$ sudo su -
$ export GUID=<ASSIGNED GUID>
$ export API_SERVER=<api-server> # Referenced in the below tableAnd finally, we need to login to the OpenShift cluster using the oc tool.
Since you are the administrator of the cluster, you will be running as admin. This is necessary in order to have the permissions necessary for setting up cluster scoped operators.
|
Note
|
Login with the username opentlc-mgr and the password r3dh4t1!
|
$ oc login -u opentlc-mgr ${API_SERVER}| Location | API Server | Web Console |
|---|---|---|
Virtual |
http://console-openshift-console.apps.cluster-${GUID}.${GUID}.open.redhat.com |
Log into the UI by following your location’s corresponding Web Console link from the table above. The login credentials are the same here as they were for oc tool.
You are now ready to start working through the workshop.
You’ll need to log into OpenShift with the oc tool to talk to the cluster from the command line. You’ll also want to log into the UI.
In this lab we’re going to build a new operator with Ansible. The operator itself is simply a container image, so we need a place to store it so we can reference it in a future deployment. Traditionally, you would leverage a centralized (sp?) image registry like quay.io. However for the sake of keeping this workshop, we are going leverage the registry that comes with OpenShift.
Let’s go checkout the registry
$ oc project openshift-image-registry
$ oc get allNAME READY STATUS RESTARTS AGE pod/cluster-image-registry-operator-f9697f69d-mhnkp 2/2 Running 0 55m pod/image-registry-584b8db59d-vcz29 1/1 Running 0 55m pod/node-ca-9wvhq 1/1 Running 0 54m pod/node-ca-j4jhz 1/1 Running 0 55m pod/node-ca-pkvwx 1/1 Running 0 55m pod/node-ca-s58b9 1/1 Running 0 55m pod/node-ca-t4c8f 1/1 Running 0 54m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/image-registry ClusterIP 172.30.11.208 <none> 5000/TCP 55m service/image-registry-operator ClusterIP None <none> 60000/TCP 64m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/node-ca 5 5 5 5 5 kubernetes.io/os=linux 55m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/cluster-image-registry-operator 1/1 1 1 64m deployment.apps/image-registry 1/1 1 1 55m NAME DESIRED CURRENT READY AGE replicaset.apps/cluster-image-registry-operator-f9697f69d 1 1 1 64m replicaset.apps/image-registry-584b8db59d 1 1 1 55m replicaset.apps/image-registry-5bd6c5dcdc 0 0 0 55m NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD route.route.openshift.io/default-route default-route-openshift-image-registry.apps.cluster-bt-nekic-cd35.bt-nekic-cd35.sandbox1444.opentlc.com image-registry <all> reencrypt None
There is a lot going on in this project, but it is really just three applications. The image-registry, the image-registry-operator, and the node-ca. We are focussing the image-registry since this is the application that will host the images we build.
To log into the image registry, we will need the route that allows traffic into the pod. To get the specific url, run this command:
$ oc get routeNAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
default-route default-route-openshift-image-registry.apps-crc.testing image-registry <all> reencrypt NoneWe’ll need this location later. Let’s save it as an environment variable.
$ export TARGET_REGISTRY=$(oc get route default-route -n openshift-image-registry --template={{.spec.host}})Now that we know where to log into, lets make sure that we have docker setup correctly to allow this registry.
-
As the user root, make sure docker is installed
$ docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: overlay2-
If docker is not installed do the following
$ yum -y install docker && systemctl enable docker && systemctl restart docker-
After verifying that docker is installed and its up and running, add ${TARGET_REGISTRY} to the insecure section of
/etc/containers/registries.conf
$ vi /etc/containers/registries.conf
[registries.insecure]
registries = ['<OUTPUT OF $TARGET_REGISTRY>']-
Restart docker to take effect
$ systemctl restart docker-
Let’s login with podman. Note that it uses your openshift session token.
docker login ${TARGET_REGISTRY} -u openshift -p $(oc whoami -t)The biggest concern from your team lead is that the team will get bogged down in managing tickets for creating namespaces for new development teams. The task at hand is to automate this process in a way that is trackable and helps enforce best practices for developers on the cluster. You decide to leverage the cool new Operator Framework to provide simple way to create a way to create and update namespaces in a kubernetes native fashion.
An operator is an extention to the Kubernetes API. With an operator, we can create a 'ManagedNamespace' custom resource (CR), and OpenShift will be able to understand what we mean and create a new namespace with all of the proper metadata that your team needs for operations. In this case we’ll also be able to set up the proper limits and quotas in order to make sure a single development team’s application does not hog all of the cluster’s resources.
Navigate to the managed-namespace-operator directory:
cd $LAB/managed-namespace-operatorHere you will see the file structure of an Ansible operator. Check out the [operator-sdk](https://github.com/operator-framework/operator-sdk/blob/master/doc/ansible/user-guide.md) Ansible documentation for a full overview of the Ansible operator. For this lab, here’s what’s important to know:
| File/Dir | Purpose |
|---|---|
build/ |
Contains the Dockerfile for building the Ansible operator |
deploy/ |
Contains the OpenShift resources necessary for deploying the Ansible operator and creating the ManagedNamespace CRD (custom resource definition) |
roles/ |
Contains the Ansible roles that the operator will be running when a CR (custom resource) is created |
molecule/ |
Contains the Ansible playbooks to perform [Molecule](https://github.com/ansible/molecule) testing on the Ansible operator |
watches.yaml |
Configures the operator to associate a CR to a particular Ansible role |
When the Ansible operator is deployed, it will listen for CRs and will apply the Ansible role accordingly. Operators are designed to maintain the "desired state", meaning it will run in a loop and will constantly re-run the roles in accordance to the CR spec to ensure that the desired state is always reached. Therefore, it’s imperative that each role be written in an idempotant and stateless manner. It should also be able to handle any change to the OpenShift environment that may occur anywhere during role execution.
Let’s dive a little deeper into the Ansible roles behind this operator. Find the roles/ directory:
cd $LAB/managed-namespace-operator/rolesHere you’ll find our one Ansible role. If we wanted to add more complicated logic, this is where we could add more roles.
| Role | Purpose |
|---|---|
managed-namespace-operator |
setup and update namespaces in OpenShift |
Entering the managed-namespace-operator show’s the traditional ansible role structure. Feel free to navigate these directories to see some of the logic that is already seeded there for you to build off of.
|
Note
|
##TODO add table of managed-namespace-operator directory |
Time to get a little more hands-on. We’ve left several placeholders throughout the operator for you to write some Ansible. Let’s walk through the changes you’ll have to make to allow the operator to be fully functional.
Each terminal has the vi editor installed. We also provide the complete files under $LAB/answers for you to copy at the end of each section.
|
Note
|
TODO Write answers directory NOTE: create $LAB env var |
View the main.yml tasks file under the managed-namespace-operator role:
cat $LAB/managed-namespace-operator/roles/managed-namespace-operator/tasks/main.ymlCurrently the the role is just a list of task names. We use these tasks to accomplish what we need to.
Under where it says ## TODO: Add module for creating namespace, add the following line:
- name: Create {{ namespace_name }} NamespaceThis is the name of the first task of the managed-namespace-operator role. It makes the Ansible code more readable by letting developers know what the task is supposed to do, and it makes runtime output easier for administrators to understand in the event of troubleshooting.
Note also the {{ namespace_name }} string. This is a variable in Ansible. This variable is inheritted from the custom resource .spec.namespaceName field. This is powerful because now we can directly call variables in the custom resource in our automation. In this case when the variable is expanded, it will equal the name of the namespace.
Let’s add a couple more lines to the create namespace role, so that your task now looks like this:
- name: Create {{ namespace_name }} Namespace
k8s:
state: present
definition:
kind: Namespace
apiVersion: v1
metadata:
name: "{{ namespace_name }}"
## labels:
## size: "{{ size }}"Note that there are two lines commented out. These will be saved for later when we want to start thinking about resource management.
Notice the k8s: line. This tells Ansible to use the k8s module to perform an action on the OpenShift cluster. Think of a module as a function, in which k8s: is our "function" and state: and definition are the parameters to that function.
state: present tells the k8s module to create a resource to the cluster (as opposed to deleting it, which would instead be state: absent).
definition: tells the k8s module specifically what to create on the cluster.
Namespaces are used for more than just creating workspaces for developers to work. They also need quotas and limits to ensure that one team doesn’t hog all of the cluster’s compute. Let’s add two more pieces of code to complete this Ansible task to tie everything together. Add to the role so that your task now looks like this:
- name: Create Resource Quota
k8s:
state: present
definition: "{{ lookup('template', 'default-resourcequota.yml.j2' ) }}"
- name: Create Limit Range
k8s:
state: present
definition: "{{ lookup('template', 'default-limitrange.yml.j2' ) }}"Note how these tasks use the same k8s module but instead use a lookup so that you can save configurations as files instead of inline. This promotes reusability of roles, and helps keep your environment logic seperate from your code. It also makes the role more readable.
|
Note
|
##TODO: say more words |
We need to turn the Ansible roles into a Docker image so that it can be deployed and tested on OpenShift. We also need to make sure we include the test artifacts that are normally excluded from the production image. We can do this easily with the operator-sdk tool.
On the command line, navigate to the managed-namespace-operator directory and build the test operator:
cd $LAB/managed-namespace-operator
sed -i "s/BASEIMAGE/$TARGET_REGISTRY\/managed-namespace-operator\/managed-namespace-operator/g" $LAB/managed-namespace-operator/build/test-framework/Dockerfile
operator-sdk build quay.io/$QUAY_USER/managed-namespace-operatorNow that the test operator is built, let’s push it to Quay with Docker.
docker login quay.io -u $QUAY_USER -p $QUAY_PASS
docker push quay.io/$QUAY_USER/managed-namespace-operatorYou’ll find that this is a somewhat large image. The production-sized operator is much smaller, which is why after we test and validate that the operator is working we should rebuild without the --enable-tests flag to remove the test artifacts.
Now that the image has been built and is now in Quay, let’s deploy it in your namespace.
First, we need to create some resources to give the operator permission to edit your project. If you recall, the deploy/ directory contains OpenShift resources that are required for the operator to work properly. It contains a service account, role, rolebindings, deployment, CRDs, and CRs. For now, let’s create only what we need to test the operator:
cd $LAB/managed-namespace-operator
oc create -f deploy/service_account.yaml -n managed-namespace-operator
oc create -f deploy/role.yaml
oc create -f deploy/role_binding.yamlNow that we know the tests have passed, let’s build the more lightweight production operator.
cd $LAB/managed-namespace-operator
operator-sdk build quay.io/$QUAY_USER/managed-namespace-operator
docker push quay.io/$QUAY_USER/managed-namespace-operator
sed -i "s/OPERATOR_IMAGE/quay.io\/$QUAY_USER\/managed-namespace-operator/g" $LAB/managed-namespace-operator/deploy/operator.yaml
oc create -f $LAB/managed-namespace-operator/deploy/operator.yamlWait for the pod to be ready
oc get pods -wNow we can see the two containers that make up the ansible operator pod, the operator and the ansible runner First lets check out the operator container
oc logs <pod-name> -c operatorNotice that the operator is using the watch.yaml file to observe the OpenShift api for any actions on a 'ManagedNamespace' object. When it sees something, it then lets the ansible runner that it needs to run the designated role.
##TODO: Log snippet
Now lets take a look at the ansible continaer
oc logs <pod-name> -c ansibleNotice that there is not much going on right now. This is because we haven’t given the operator anything to work with yet!
Now that the Ansible operator is deployed, it’s super easy to add namespaces to OpenShift! First, let’s check out the ManagedNamespace CR:
cat $LAB/managed-namespace-operator/deploy/crds/mysql/nekic_v1alpha1_initproject_cr.yamlNotice that it has two spec fields, namespaceName and size. Right now, the operator is only cares about the namespaceName, since this will become the name of the namespace. We’ll focus on the size later.
Let’s create the resource with:
oc create -f $LAB/managed-namespace-operator/deploy/crds/mysql/nekic_v1alpha1_initproject_cr.yaml
You should get a message saying that the ManagedNamespace resource was created. The new namepsce will get added pretty quickly - right now the operator pod running the corresponding Ansible role. We can see this role in action by checking out the operator logs:
```bash
oc logs --follow $(oc get po | grep managed-namespace-operator | awk '{print $1}')When the role is finished, you should see something like ansible-runner exited successfully in the logs, as well as a new namespace added to the cluster. This is pretty slick and all but we all know that one development team that will need more resources. Let’s add the concept of t-shirt sizes in order to make our lives easier down the road.
To accomplish this, we will need to update some of the logic in our ansible role. Uncomment the labels section
- name: Create {{ namespace_name }} Namespace
k8s:
state: present
definition:
kind: Namespace
apiVersion: v1
metadata:
labels:
size: "{{ size }}"
name: "{{ namespace_name }}"Now when this task is called, the k8s module will ensure that this label is added to each of the managed namespaces. This will make auditing and monitoring easier since an administrator see this label and understand the amount of reasources a namespace should be allocated. It also makes forecasting resource consumption simpler with codified t-shirt sizes
Next, we need to update the quota and limit logic to select the proper size t-shirt template instead of the default size. To accomplish this, update the lookup line to include the size parameter that gets passed in from the ManagedNamespace cluster resource object. It should look like this:
- name: Create Resource Quota
k8s:
state: present
definition: "{{ lookup('template', '{{ size }}-resourcequota.yml.j2' ) }}"
- name: Create Limit Range
k8s:
state: present
definition: "{{ lookup('template', '{{ size }}-limitrange.yml.j2' ) }}"You can take a look at the template directory also within this role and see that it is seeded with some basic t-shirt sizes.
take a look at mediumNotice that name of the resource is generic, but it is labeled the proper size. This will help us down the road in the event you want to upgrade a namespace to a larger size. Instead of having to deal with deleting one quota and adding another, you can patch or apply the updated quota and Openshift will take care of the merging logic. This avoids any lapses in quota management.
With the role updated, rebuild the image and push it up to the registry. This will make it available for Openshift to deploy it onto the cluster.
operator-sdk build quay.io/$QUAY_USER/managed-namespace-operator
docker push quay.io/$QUAY_USER/managed-namespace-operatorWith the new image available, trigger a new deployment so that OpenShift will rollout the new image.
##TODO: figure out imagestreams
oc deploy dc/managed-namespace-operatorWatch the rollout for the new pod to become ready
oc get pods -wNow the operator is running your new ansible role that can handle t-shirt sizes. Let’s try creating a new namespace with a medium t-shirt size.
oc create -f $LAB/managed-namespace-operator/deploy/crds/medium-namespace.yamlWatch for the new namespace to be created. Be fast!
oc get namespaces -wOnce it’s created, check out its quotas.
oc get quotas -n medium-namespaces -o yaml
##TODO: put in quota code block
Notice how the label is set to medium and the limits are higher!
## 9 Updating existing namespaces
Good news! The development team that you created the first namespace for got approval to ramp up their deployments on OpenShift. This means that their namespaces is going to need more resources. How can this be done?
Even better news! The managed-namespace-operator can already handle this! All you need to do is update the ManagedNamespace CR on the cluster to tell the operator to update the namespace. Update the label on the test-project CR to now be set to 'large'
```bash
oc edit ManagedNamespace.nekic.io example-init-projectOnce you save that, the operator will go ahead to update the quota
##TODO: maybe have user set up a watch on the quota in test-project namespace in order to show update
What if we deploy bad stuff on to cluster, and the operator has to figure out how to clean it up
##TODO: High Level Lab Flow - Edit operator - add section for creating namespace - Testing? ## Not sure what would need to be done for this - Build operator image - Push operator image to internal registry - Deploy operator - App needs bigger quota - Need to be able to resize - Update label on namespace task - Update template lookup to have size option for quota and limits - Rebuild operator image - Push operator image to internal registry - Create a new ManagedNamespace CR - Validate namespace - Update existing ManagedNamespace CR - Watch to see it get updated - Mention GitOps
Verification steps: copy work from agnosticd/ansible/configs/ocp4-workshop/post_software.yml post-flight-check section - Validate that there are no defaultProject requests - Internal Registry pod is up - get internal registry service - get user token - Login into internal registry - Create post-flight project - build and push image to internal registry - deploy operator - wait for operator pod to run - create managednamespace object - check resulting project status - clean up operator - namespace - clusterresource - crd - clusterroles - clusterrolebinding - delete the test-project - make sure image is deleted from internal registry
-
pull the managed-namespace-operator into the backpack
-
set up internal registry as insecure???
Things To figure out: - myvars.yaml - post branch onto github - define workloads to be run - copy work from agnosticd/ansible/configs/ocp4-workshop/post_software.yml post-flight-check section
Creating operator notes
operator-sdk new managed-namespace-operator --api-version=beter.together.io/v1alpha1 --kind=ManagedNamespace --type=ansibleupdate WATCH_NAMESPACE
vi deploy/operator.yaml env:
- name: WATCH_NAMESPACE
value: ""update role to cluster role
mv deploy/role.yaml deploy/cluster_role.yamlvi deploy/cluster_role.yamlapiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole ###UPDATE HERE
metadata:
creationTimestamp: null
name: managed-namespace-operatoradd required rules to ClusterRole # Note this is adding an additional item to the list of roles
vi deploy/cluster_role.yalm- apiGroups:
- ""
resources:
- namespaces
- resourcequotas
- limitranges
verbs:
- "*"update rolebinding to clusterrolebinding
mv deploy/role_binding.yaml deploy/cluster_role_binding.yamlkind: ClusterRoleBinding ## UPDATED
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: managed-namespace-operator
subjects:
- kind: ServiceAccount
name: managed-namespace-operator
namespace: managed-namespace-operator ## ADDED LINE
roleRef:
kind: ClusterRole ## UPDATED
name: managed-namespace-operator
apiGroup: rbac.authorization.k8s.io
update crd to be cluster scoped
```bash
vi deploy/crds/better_v1alpha1_managednamespace_crd.yamlmetadata:
name: managednamespaces.better.together.io
spec:
group: better.together.io
names:
kind: ManagedNamespace
listKind: ManagedNamespaceList
plural: managednamespaces
singular: managednamespace
scope: Cluster ## UPDATED
...