README.md

Container Promotion Template Overview

The promotion template provided by SAS uses a GitOps process to move the container across environments automatically, which saves the IT team time and resources. In this process, SAS Viya acts as the development environment. You will use a few resources on Azure, including Azure Container Registry, where a copy of the container is stored, and Azure Kubernetes Service, which acts as the testing and production environments. You will also use open-source orchestration tools, including GitHub, Flux CD, and Jenkins. GitHub will store a copy of the container files, including a deployment descriptor. This deployment descriptor provides instructions for the GitOps controller, including instructions on where to find the container within Azure Container Registry, how to communicate with the container, what namespace it should be deploying to within Azure Kubernetes Service, and how many processing nodes should be within the cluster. An open-source tool called Flux CD is used to act as the GitOps controller because it provides integration between the deployment descriptor in the GitHub repository and the Kubernetes cluster. And finally, Jenkins, a popular open-source automation tool is used.

The process starts after a model or decision has been developed on SAS Viya. Models and decisions can be published directly into Azure Container Registry just as the deployment descriptor is pushed into the GitHub repository. For model publishing, the CI/CD promotion assets include a SAS Workflow Manager definition. This workflow definition containerizes the champion model within a project, deploys the container into Azure Container Registry, pushes the deployment descriptor into the GitHub repository, and then starts the Jenkins pipeline. For users publishing decisions or non-champion models, the models and decisions can be published from SAS Viya and the Jenkins pipeline can be started manually using the published name as a parameter.

The Jenkins pipeline passes the deployment description to Flux CD, where it makes the appropriate changes within the testing Kubernetes Cluster on Azure Kubernetes Service. After the testing cluster has been configured and the container is running, Jenkins runs data through the container to validate that it is scoring as expected. Upon successfully validating the container in the testing environment, Flux CD repeats the process on the production environment. After the container is running, it is ready to score data.

Organizations can pass data to the container using the REST API. REST API is a standard for communication between applications, so this makes it easy to leverage container scoring within an application or process.

Configuring the Promotion Template

To start using the promotion pipeline template, you must first add a few configurations to each environment that the pipeline touches. After you have completed the configurations, the pipeline automates promotion and validation, which saves time and reduces future headaches.

Note: For the Azure configuration steps, please check with your Azure administrator for help with determining the correct region for your Azure environment.

Note: Configuring the various Azure components can be performed in the Azure Portal, Azure CLI, Azure PowerShell, or an Azure Resource Management (ARM) template. This guide uses the Azure Portal, but any method can be used.

Azure Resource Group

1. Log in to the Azure Portal.
2. Search for "Resource groups".
3. Create a new resource group. Recommended name: "deploymentOrchestration". An existing resource group can also be used.

Note: This resource group should be used in subsequent Azure configuration steps unless otherwise specified.

Alternatively, you can use the Azure CLI to create a resource group:

az group create --name deploymentOrchestration --location eastus

For more information, see Manage Azure Resources.

Azure Container Registry

1. Log in to the Azure Portal.
2. Search for "Container registries".
3. Create a new registry. Recommended name: "OrchestrationRegistry". An existing registry can also be used.

Alternatively, you can use the Azure CLI to create a container registry:

az acr create --resource-group deploymentOrchestration --name OrchestrationRegistry --sku Basic --location eastus --admin-enabled true

4. Register an application and a secret with Azure Secure App. Allow the app to push and pull images from the Azure Container Registry. Assign the app access to the Azure Kubernetes Service validation cluster and add role assignments for Azure Kubernetes Cluster Admin and Azure Kubernetes Service Contribution Role.
5. Save your Azure Tenant ID, Subscription ID, Container Registry Login Server (Base Repository URL), Application (Client) ID, Application Secret, Resource Group where the validation AKS cluster is deployed, and the name of the AKS validation cluster.

For more information, see Azure Container Registry and Bogdan Teleuca’s article, How to Publish a SAS Model to Azure with SCR: A Start-to-Finish Guide.

Azure Kubernetes Service

1. Log in to the Azure Portal.
2. Search for "Kubernetes services".
3. Create a new Kubernetes service named "testing".
   1. Select a Kubernetes version of 1.23 or later.
   2. Select the Integrations tab and set the Container registry to the registry configured above.
4. Create a new Kubernetes service named "production".
   1. Select a Kubernetes version of 1.23 or later.
   2. Select the Integrations tab and set the Container registry to the registry configured above.
5. Assign Azure Kubernetes Service Cluster Admin Role for both clusters to your user.
6. Next, export a single kubeconfig file with keys for both clusters. Ensure that the kubeconfig is linked to the Kubernetes cluster with pull authorization.

Note: There are a lot of configuration options associated with the Kubernetes services. The default settings are recommended as a starting point, but for better integration in you overall system, you will need to consult the configuration guide or you Azure administrator.

For more information, see the Azure Kubernetes Service (AKS) documentation.

Alternatively, you can use the Azure CLI to create the two AKS clusters, assign the service admin role, and export the single config file:

# Create a test cluster.
az aks create -g deploymentOrchestration -n testing --enable-managed-identity --node-count 1 --enable-addons monitoring --enable-msi-auth-for-monitoring  --generate-ssh-keys

# Create a prod cluster.
az aks create -g deploymentOrchestration -n production --enable-managed-identity --node-count 1 --enable-addons monitoring --enable-msi-auth-for-monitoring  --generate-ssh-keys

# Get the account credentials for the logged in user.
ACCOUNT_UPN=$(az account show --query user.name -o tsv)
ACCOUNT_ID=$(az ad user show --id $ACCOUNT_UPN --query id -o tsv)

# Assign the 'Cluster Admin' role to the user for test cluster. 
AKS_CLUSTER=$(az aks show --resource-group deploymentOrchestration --name testing --query id -o tsv)
az role assignment create \
   --assignee $ACCOUNT_ID \
   --scope $AKS_CLUSTER \
   --role "Azure Kubernetes Service Cluster Admin Role"

# Assign the 'Cluster Admin' role to the user for production cluster. 
AKS_CLUSTER=$(az aks show --resource-group deploymentOrchestration --name production --query id -o tsv)
az role assignment create \
   --assignee $ACCOUNT_ID \
   --scope $AKS_CLUSTER \
   --role "Azure Kubernetes Service Cluster Admin Role"

# Export the config file.
az aks get-credentials --resource-group deploymentOrchestration --name testing --admin
az aks get-credentials --resource-group deploymentOrchestration --name production --admin

cat ~/.kube/config

sudo az aks install-cli

kubectl config view

kubectl config get-contexts
kubectl config current-context

kubectl config use-context production-admin
kubectl config use-context testing-admin

Azure Virtual Machine

1. Log in to the Azure Portal.
2. Search for "Virtual machines".
3. Create a new Virtual Machine. Recommended name "orchestrationVM".
   1. Use a Linux base image. Recommend "Ubuntu Server 20.04 LTS" or later.
   2. Select x64 Architecture.
   3. Select a small VM size (B1s or B2s is adequate). If this will be shared with other processes, then a larger size might be necessary.
   4. Generate or use an existing SSH key pair.

Note:This VM is used later for Jenkins and FluxCD installations.

For more information, see Virtual machines in Azure.

GitHub Repositories

1. Log in to GitHub.
2. Create a new private repository named "orchestration".
3. Download the orchestration_template.zip file.
4. Copy the folder structure and all files from orchestration_template.zip into the "orchestration" repository.

5. Create a new private repository named "published_models".
6. Download the published_models_template.zip file.
7. Copy the folder structure and all files from published_models_template.zip into the "published_models" repository.

8. Generate a new Personal Access Token.
   1. In the upper right corner of any page, click your profile photo, and then click Settings.
   2. In the left sidebar, click Developer settings.
   3. In the left sidebar, click Personal access tokens.
   4. Click Generate new token.
   5. Give your token a descriptive name. Recommended name: "orchestrationToken".

9. Keep the token for later configurations in Jenkins.

For more information, see GitHub documentation.

Jenkins

1. Log in to the shell of the Azure virtual machine that you previously configured.
2. Execute the following shell commands:

sudo apt update
sudo apt install python3
sudo apt install python3-pip
python3 -m pip install jproperties
python3 -m pip install pygit
python3 -m pip install pygithub
sudo apt install default-jre
sudo apt install jenkins

Note: Depending on your networking configuration, it might be necessary to create an Azure firewall rule to access the port 8080 on the virtual machine.

3. Complete your installation of Jenkins by following the steps listed in the Post-Installation Setup Wizard.
4. From Jenkins, select + New Item. Enter gitOps_sync as the item’s name, select Pipeline as the item’s type, and then select OK. On the configuration view, make the following changes:
   1. Select This Project is Parameterized.
   2. Add a new Credentials Parameter and specify values for the following fields:
      • Name: GITOPS_GIT_CREDENTIAL
      • Credential type: Username with password
      • Default Value: Access key for GitHub
      • Description: Credential for logging in to the remote Git repository
   3. Add a new String Parameter and specify values for the following fields:
      • Name: GIT_REPOSITORY
      • Default Value: orchestration
      • Description: Repository in the Git account to target
   4. Add a new String Parameter and specify values for the following fields:
      • Name: PROMOTION_CLUSTER
      • Default Value: testing
      • Description: Cluster for GitOps to deploy to
   5. Under the Pipeline configuration settings, select Pipeline script from SCM for the definition.
   6. Select Git as the SCM.
   7. Enter the repository URL for your published_models GitHub repository.
   8. Select Access key for GitHub for the type of credentials.
   9. Enter */main for the branch specifier.
   10. Enter ci/git_deployment/Jenkinsfile for the script path.
   11. Click Save.

Flux CD

1. Install Azure CLI.

sudo apt install azure-cli

# Login
az login

2. Install kubectl.

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

For more information, see Install and Set Up kubectl on Linux.

3. Export a single kubeconfig file with keys for both clusters.

# Testing
az aks get-credentials --resource-group deploymentOrchestration --name testing --admin

# Production
az aks get-credentials --resource-group deploymentOrchestration --name production --admin

# Display list of contexts -- you will see the current context with an asterisk *
kubectl config get-contexts

Note: Ensure that the kubeconfig is linked to the Kubernetes cluster with pull authorization.

4. Install Flux CLI.

curl -s https://fluxcd.io/install.sh | sudo bash

5. Bootstrap testing and production clusters for the Flux CD.

export GITHUB_TOKEN="<Token from GitHub configuration>"
export GITHUB_USER="<User name for the GitHub account>"
export GITHUB_REPO="orchestration"

flux bootstrap github --context=testing-admin --owner=${GITHUB_USER} --repository=${GITHUB_REPO} --branch=main --personal --path=clusters/testing

flux bootstrap github --context=production-admin --owner=${GITHUB_USER} --repository=${GITHUB_REPO} --branch=main --personal --path=clusters/production

SAS Viya Environment Manager and SAS Studio

1. From SAS Environment Manager, import the RunTestingPushPipeline job file and the RunProductionPushPipeline job file.
2. Open both job files in SAS Studio. Add your Jenkins username, key, and URL into both the jobs, where specified.

SAS Viya Command Line Interface

If using SAS Viya 2022.1.2 or later, you can use the CLI to define the publishing destination.

1. The SAS Viya command-line interface (CLI) enables you to interact directly with the SAS Viya REST services. You enter commands on a command line and receive responses from the system. You can use the CLI to work with SAS Viya programmatically, as an alternative to using a graphical user interface (GUI). If this is the first time your organization has used the Command Line Interface (CLI) , you must download the CLI and configure SAS Viya to run the CLI.
2. Sign in to SAS Viya via the CLI using the following code:

sas-viya auth login

3. Configure a publishing destination for Azure with Git using the CLI by filling out the fields below with the information that you have saved from the previous steps. If you are running the CLI from a Window’s machine, replace the \ at the end of each line with a ^. This character tells the interface that the command continues on the next line, which enables you to make the command more human-readable.

sas-viya models destination createAzure \ 
--name name-for-the-publishing-destination \
--baseRepoURL base-repo-url-from-azure-container-registry \
--tenantId azure-tenant-id \
--subscriptionId azure-subscription-id \
--resourceGroupName resource-group-for-aks-validation-cluster \
--kubernetesCluster name-of-aks-validation-cluster  \
--region regional-location-of-resources \
--credDomainID name-for-credential-domain \
--clientID client-id-for-azure \
--clientSecret client-secret-for-azure \
--gitUserUd github-user-id \
--gitAccessToken github-access-token \
--remoteRepoURL url-for-model-publish-github-repo \
--localRepoLocation _ocation-of-local-Viya-git-repo \
--userEmail email-used-with-github-repo 

SAS Model Manager

1. Create a model and register the model into SAS Model Manager. Mark this model as champion.

Note: To deploy a decision, create a decision in SAS Intelligent Decisioning instead.

SAS Workflow Manager

Note: To deploy a decision, you can skip the configuration steps for SAS Workflow Manager. You will need to manually deploy the decision instead.

1. If this is the first time your organization has used SAS Workflow Manager, you will need to configure the default workflow administrator group and add your user to that group as well as configure the default service account.

2. Once SAS Workflow Manager has been configured and your user has the appropriate permissions, you will need to download the provided Workflow definition, import the definition into SAS Workflow Manager.
3. Review the workflow template and adjust as needed, such as changing the wait time on the timer or adding additional manual approvals.
4. Finally, activate the workflow definition.

Additional Configuration and Customization

At this point, the full system is almost ready to be used. However, there are a few additional configurations that are required as well as some customizations to be aware of.

Customization Properties in the Deployment Templates

In the deployment templates, a placeholder pattern is used for common fields that are used during the deployment process. There are two standard patterns that are seen in the deployment descriptor templates:

• ${sas-scr-published-name} -- this field is replaced with the published named of the model. In this example, it is fraudchamp. When using the workflow, the published name is captured when starting the workflow as the deployment name for the project. This field is used in numerous places in the deployment templates.
  • The deployment of the SCR container uses this name (for example, fraudchamp).
  • The pod associated with the deployment is - (for example, fraudchamp-6bd4456fb7-89wdn).
  • The service associated with the deployment is -service (for example, fraudchamp-service).
  • The ingress associated with the deployment is -ingress (for example, fraudchamp-ingress).
• ${sas-scr-image-url} -- this field is replaced with the url to the container registry entry for the published model. This used when Kubernetes pulls the SCR container image for deployment.

Nothing needs to be done with these properties for this example. However, these fields can be used if you want to make changes to the provided deployment templates.

There are also additional placeholders that are used during the deployment process. The Jenkins continuous-integration scripts replace these placeholders and they are specific to the cluster to which you are deploying. The two placeholders described below are used in this example and are specified in the provided deployment descriptor templates. But additional properties can be added to support additional customizations specific to your deployment requirements.

• ${custom-scr-namespace} -- this field defines the namespace where the artifacts associated with the deployment will execute (for example, orchestration).
• ${custom-scr-host} -- this field defines the host url used by the NGINX ingress. This is the http url that is used when accessing then endpoints of the SCR container (for example, testingcluster.mycompany.com).

These placeholders are defined in the "published_models" cluster in the "ci/custom/<cluster_name>/custom_properties.custom_properties" file. These properties must be defined prior to using this template.

For more information about the ingress host setup, see Kubernetes ingress concepts.

Using the Template

Now you are ready to use the template to deploy your champion model from SAS Model Manager and start the GitOps workflow. To deploy your decision, publish your decision into the Azure Container Registry and manually start the Jenkins pipeline. Watch as the container moves from your Azure Test Environment to your Azure Production Environment. After the container is running in production, you are ready for scoring.

Note: The deployment name must consist only of lowercase and alphanumeric characters, and cannot start with a numeric character.

To score the container, perform these steps:

1. Find the scoring endpoint for the deployed model or decision within the container. There are a few ways to accomplish this, such as using the Kubernetes configuration file exported from Azure Kubernetes Service.
2. Prepare data in JSON format.
3. Pass data to the endpoint and wait for the response.

This notebook includes example code for passing transactional and batch data to the endpoint.