This solution uses two GitHub Actions workflow yaml files luis_pr.yaml and luis_ci.yaml. To configure the workflows for use, you must set GitHub Secrets to match your configuration of resources in Azure as described below in GitHub Secrets, but you should not need to make any edits to these files to use them in the solution.
This document describes the workflow steps so that you can understand how they work should you need to modify them for your own projects.
The workflows operate in response to two distinct events:
- On raising of a Pull Request (PR) for changes made in a feature branch that will be merged to master, the luis_pr.yaml workflow executes.
- When a PR has been merged and the changes are pushed to master, luis_ci.yaml executes.
When triggered for a PR, the luis_pr.yaml workflow acts as a quality gate. It builds a temporary LUIS app, runs all the unit tests against it and fails the workflow if any tests fail; this will block completion of the PR. At the end the temporary LUIS app is deleted.
When triggered for a merge to master, the luis_ci.yaml workflow creates a new version in the LUIS app that has been created for the master branch, runs the unit tests, and if the tests pass, creates a GitHub release which includes a Release artifact containing data identifying the new version, and it runs a simple CD (Continuous Deployment) job that publishes the new LUIS app version to the Production slot. It also runs quality tests to determine the F-measure for the new model.
The configuration of both workflows ensure that they will be triggered only when either the LUIS model or the test suite is changed:
name: LUIS-PR
# Trigger the workflow on pull request, and only for changes to lu or json files
on:
pull_request:
paths:
- 'luis-app/*.lu'
- 'luis-app/Tests/*.json'The trigger configuration for luis_ci.yaml ensures that it runs when a merge to master happens:
name: LUIS-CI
# Trigger the workflow on push to the master branch, and only for changes to lu or json files
on:
push:
branches:
- master
paths:
- 'luis-app/*.lu'
- 'luis-app/Tests/*.json'Both workflows make use of a number of variables that must be defined in GitHub Secrets. Ensure each of the following secrets have been set, using the values appropriate to your configuration of resources in Azure:
| Secret Name | Value |
|---|---|
| AZURE_CREDENTIALS | *Service Principle token - see Creating the Azure Service Principal |
| AZURE_RESOURCE_GROUP | name of the resource group that contains the resources below |
| AZURE_LUIS_AUTHORING_RESOURCE_NAME | name of the Azure LUIS authoring resource |
| AZURE_LUIS_PREDICTION_RESOURCE_NAME | name of the Azure LUIS prediction resource |
| AZURE_STORAGE_ACCOUNT_NAME | name of the Azure storage account |
The workflows also use a number of environment variables that are defined at workflow scope:
- LUIS_MASTER_APP_NAME Set this to the name of the LUIS app that is built from the source checked into the master branch, and which the workflow will create when it first runs.
- IS_PRIVATE_REPOSITORY Set this to
trueif your GitHub repository is private, otherwise set tofalse.
env:
# Set the name of the master LUIS app
LUIS_MASTER_APP_NAME: LUISDevOps-master
# If your repository is Private, set this to true
IS_PRIVATE_REPOSITORY: falseIn addition, the following environment variables set the names of the source file that define your LUIS app, and the files containing the unit tests and the F-measure quality tests:
# Set the path to the lu file for your LUIS app
LU_FILE: luis-app/model.lu
# Set the path to the file for your unit tests
UNIT_TEST_FILE: luis-app/Tests/unittests.json
# Set the path to the file for your unit tests
QUALITY_TEST_FILE: luis-app/Tests/verificationtests.json
# Set the name of the container in the Azure Storage account that contains the baseline F-measure results
BASELINE_CONTAINER_NAME: ''The BASELINE_CONTAINER_NAME defines the name of the storage container in your Azure Storage account that contains F-measure testing results for your baseline LUIS app version. This is used for comparison purposes to determine whether the performance of the new model being built from the current source has improved or regressed compared to the baseline. Leave this value blank when starting out with a new app until such time as you have an app of sufficient maturity that you wish to commence comparison testing. See Job: LUIS F-measure testing to learn more about the use of this environment variable.
The luis_ci.yaml workflow is divided into three discrete jobs. Each job runs independently but sequentially in its own environment and the workflow is configured so that the build job executes first, followed by the LUIS_quality_testing job and the release job. The luis_pr.yaml workflow is a single job that is almost identical to the first job of the luis_ci.yaml workflow, but with these distinct differences:
- luis_pr.yaml creates a temporary LUIS app from the source in the PR that acts as the test target and is deleted again at the end of the run
- luis_ci.yaml creates a new version in the LUIS app for the master branch from the merged source in master which is not deleted at the end of the run.
The first job builds and unit tests the LUIS model.
luis_ci.yaml starts by checking out the code and then fetching all the history and tags for all branches, information that is required by the GitVersion step which increments the version number using semantic versioning on every build:
build:
name: Build and Test LUIS model
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Fetch all history for all tags and branches - for GitVersion
if: env.IS_PRIVATE_REPOSITORY == 'false'
run: git fetch --prune --unshallow
- name: Fetch all history for all tags and branches (private repo) - for GitVersion
if: env.IS_PRIVATE_REPOSITORY == 'true'
run: |
git config remote.origin.url https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}
git fetch --prune --unshallow
- name: Install GitVersion
uses: gittools/actions/gitversion/setup@v0.9.3
with:
versionSpec: '5.2.x'
- name: Use GitVersion
id: gitversion
uses: gittools/actions/gitversion/execute@v0.9.3
- name: luisAppVersion env
run: echo "::set-env name=luisAppVersion::$GitVersion_SemVer"We log into Azure using the AZURE_CREDENTIALS token saved into GitHub secrets during setup, and query for the LUIS authoring key, prediction key and authoring endpoint that are needed later on. We use the add-mask function to mask sensitive keys to ensure they are hidden in the log files:
- uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}
- name: Get LUIS authoring key
run: |
keya=$(az cognitiveservices account keys list --name $AzureLuisAuthoringResourceName --resource-group $AzureResourceGroup --query "key1" | xargs)
echo "::set-env name=LUISAuthoringKey::$keya"
echo "::add-mask::$keya"
- name: Get LUIS prediction key
run: |
keyp=$(az cognitiveservices account keys list --name $AzureLuisPredictionResourceName --resource-group $AzureResourceGroup --query "key1" | xargs)
echo "::set-env name=LUISPredictionKey::$keyp"
echo "::add-mask::$keyp"
- name: Get LUIS authoring endpoint
run: |
az cognitiveservices account show --name $AzureLuisAuthoringResourceName --resource-group $AzureResourceGroup --query "properties.endpoint" | \
xargs -I {} echo "::set-env name=luisAuthoringEndpoint::{}"Next we install node.js (necessary for running the Bot Framework CLI) and the BF CLI. Before we will be able to run any botframework-cli commands in CI we need to disable telemetry using the workaround shown in this GitHub issue:
- uses: actions/setup-node@v1
with:
node-version: '12.x'
- name: Bypass botframework-cli telemetry prompts, enable telemetry collection - set to false to disable telemetry collection
run: echo "::set-env name=BF_CLI_TELEMETRY::true"
- name: Install @microsoft/botframework-cli
run: |
npm i -g @microsoft/botframework-cliThe next stage of the workflow creates a LUIS model. We import and use a ludown.lu file for training. We use the LuDown format to define the LUIS app version since it can be maintained in a source control system and is human readable to allow us to work with it outside the LUIS portal's GUI tool. You can replace this file with another file that defines the intents, utterances, entities that you need for your own model. This may be useful if you are generating your training data from some other system or by some other mechanism. Ultimately, we need to provide the information in the LUIS JSON format and the BF CLI provides tooling to support this.
The first step transforms the ludown file to a LUIS JSON file using the botframework-cli:
- name: Ludown to LUIS model
run: bf luis:convert -i $LU_FILE -o ./model.json --name 'LUIS CI pipeline - ${{ github.run_id }}' --versionid $luisAppVersionThe model.json file output must be imported to LUIS. This happens in different ways depending on whether the workflow is operating as a PR gate-check - where it creates a new LUIS app for testing which is deleted at the end of the workflow - or if operating as a Merge workflow.
The luis_pr.yaml workflow is operating as a PR quality gate and it creates a temporary app to use as the test target. bf luis:application:import returns a string with the LUIS App ID that we will need to use in the next steps, so we save the AppId in an environment variable called LUISAppId:
# When doing a gate check on PRs, we build a new LUIS application for testing that is later deleted
- name: Create PR check LUIS application
run: |
response=$(bf luis:application:import --endpoint $LUISAuthoringEndpoint --subscriptionKey $LUISAuthoringKey --in model.json --json)
status=$(echo "$response" | jq '.Status' | xargs)
if [ "$status" == "Success" ]
then
appId=$(echo "$response" | jq '.id' | xargs)
echo "::set-env name=LUISAppId::$appId"
else
exit 1
filuis_ci.yaml is operating as a Merge workflow, so the LUIS app is the one associated with the master branch and we use the name specified in the YAML file. The app will be created if it does not already exist. This step determines the AppId (GUID) and saves it in the AppId environment variable:
# When doing a merge to master, use the master LUIS app - create if necessary (soft fails if exists)
- name: Get master LUIS application ID
run: |
bf luis:application:create --name $LUIS_MASTER_APP_NAME --subscriptionKey ${{ env.LUISAuthoringKey }} --endpoint $luisAuthoringEndpoint --versionId=0.1
bf luis:application:list --subscriptionKey ${{ env.LUISAuthoringKey }} --endpoint $luisAuthoringEndpoint | \
jq -c '.[] | select(.name | . and contains('\"$LUIS_MASTER_APP_NAME\"')) | .id' | \
xargs -I {} echo "::set-env name=AppId::{}"
# Check that we found the master app Id - failure probably indicates misconfiguration
- name: Validate application ID
run: |
echo "LUIS app Id: $LUISAppId"
if [ ${#LUISAppId} -ne 36 ]; then
echo "ERROR: Failed to find LUIS master app. Check workflow configuration."
exit 1
fiNext, we check if the LUIS app currently has 100 versions (the limit), and if so print a warning and fail the workflow. In order to resolve this, unneeded versions must be deleted from the LUIS master app.
- name: Purge LUIS app version
run: |
version_count=$(bf luis:version:list --appId $AppId --endpoint $luisAuthoringEndpoint --subscriptionKey ${{ env.LUISAuthoringKey }} | jq 'length')
if [ $version_count -ge 100 ]; then
echo "ERROR: LUIS app: $AppId version count will exceed 100. Delete unneeded versions before re-running pipeline"
exit 1
fiThen we go ahead and create a new version in the LUIS app by importing the JSON created earlier. (This step will only do something when the LUIS app is the one targeted by the Merge workflow):
# When doing a CI/CD run on push to master, we create a new version in an existing LUIS application
- name: Create new LUIS application version
if: github.event_name == 'push'
run: bf luis:version:import --appId $LUISAppId --endpoint $LUISAuthoringEndpoint --subscriptionKey $LUISAuthoringKey --in model.jsonThe BF CLI is used to initiate the training of the model and to wait for this to complete.
- name: Train luis
shell: bash
run: bf luis:train:run --appId $LUISAppId --versionId $LUISAppVersion --endpoint $LUISAuthoringEndpoint --waitAfter the model has finished training we can publish our LUIS model. We use direct version publishing for this rather than publishing to the named slots, staging and production. We do this to be able to support more than two published versions at any one time so that multiple LUIS app versions can be in a published state at any one time to support more then two dev environments simultaneously (for example, DEV, QA, UAT, PRODUCTION. We use a cURL command here to call the REST API directly:
- name: Publish luis
run: |
curl POST $POSTurl \
-H "Content-Type: application/json" \
-H "Ocp-Apim-Subscription-Key: ${{ env.LUISAuthoringKey }}" \
--data-ascii "{'versionId': '$luisAppVersion', 'directVersionPublish': true}"
env:
POSTurl: ${{ env.luisAuthoringEndpoint }}luis/authoring/v3.0-preview/apps/${{ env.AppId }}/publishTo prepare for testing, we install the NLU.DevOps test tool:
- name: Install dotnet-nlu
run: dotnet tool install -g dotnet-nluOn the ubuntu agent, you need to append the tools directory to the system PATH variable for all subsequent actions in the current job to be able to use the cli tool.
- name: Path
run: echo "::add-path::$HOME/.dotnet/tools"In order to test a LUIS app, you must use an Azure LUIS Prediction resource key. The Authoring key is not practicable since it is subject to throttling which may cause the tests to fail unnecessarily. Here we assign the Azure LUIS prediction resource to the application. Note that once again we access the REST API directly using curl as the BotFramework CLI does not support the allocation of Azure LUIS resources at this time:
- name: Get Azure subscriptionId
run: |
az account show --query 'id' | \
xargs -I {} echo "::set-env name=AzureSubscriptionId::{}"
- name: Assign LUIS Azure Prediction resource to application
shell: pwsh
run: |
bf luis:application:assignazureaccount --azureSubscriptionId $env:AzureSubscriptionId --appId $env:LUISAppId --accountName $env:luisPredictionResourceName --subscriptionKey $env:LUISAuthoringKey --endpoint $env:LUISAuthoringEndpoint --resourceGroup $env:azureResourceGroup --armToken $(az account get-access-token --query accessToken -o tsv)
env:
luisPredictionResourceName: ${{ secrets.AZURE_LUIS_PREDICTION_RESOURCE_NAME }}
azureResourceGroup: ${{ secrets.AZURE_RESOURCE_GROUP }}To test the LUIS app version that was created, we use the unit test file:
- name: Test Luis model
run: dotnet nlu test -s luisV3 -u $UNIT_TEST_FILE -o results.json
env:
luisAppId: ${{ env.LUISAppId }}
luisVersionId: ${{ env.LUISAppVersion }}
luisDirectVersionPublish: true
luisEndpointKey: ${{ env.LUISPredictionKey }}
luisPredictionResourceName: ${{ secrets.AZURE_LUIS_PREDICTION_RESOURCE_NAME }}To evaluate results we use two files: the unit test file that consists of test utterances and the expected intents and entities results and results.json file which was created by the Test LUIS model step and contains the actual results returned from testing the LUIS model:
- name: Analyze Unit test results
run: dotnet nlu compare -e $UNIT_TEST_FILE -a results.json --unit-test --output-folder unittestWe archive the test results as a build workflow artifact:
- name: Archive Unit Test Results
uses: actions/upload-artifact@v1
with:
name: UnitTestResult
path: unittest/TestResult.xmlIn luis_pr.yaml where the workflow is operating as a PR gate-check, at this point the LUIS app created by this workflow is deleted:
# Delete the LUIS app again if we are executing as gate check on a PR
- name: Delete luis test target app
if: always() && (github.event_name == 'pull_request')
shell: bash
run: bf luis:application:delete --appId $AppId --endpoint $luisAuthoringEndpoint --subscriptionKey ${{ env.LUISAuthoringKey }} --forceFor luis_ci.yaml there is still some work to do. It creates a GitHub Release, which also tags the repo using the version Id:
- name: Create Release
id: create_release
uses: actions/create-release@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, you do not need to create your own token
with:
tag_name: ${{ env.LUISAppVersion }}
release_name: Release ${{ env.LUISAppVersion }}
body: |
Releasing new LUIS endpoint
draft: false
prerelease: falseFinally, it uploads the version details as a Release asset:
- name: Get LUIS latest version details file
run: |
bf luis:version:list --appId $LUISAppId --endpoint $LUISAuthoringEndpoint --subscriptionKey $LUISAuthoringKey --take 1 --out luis_latest_version.json
- name: Upload Release Asset
id: upload-release-asset
uses: actions/upload-release-asset@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
upload_url: ${{ steps.create_release.outputs.upload_url }} # This pulls from the CREATE RELEASE step above, referencing it's ID to get its outputs object, which include a `upload_url`
asset_path: ./luis_latest_version.json
asset_name: luis_latest_version.json
asset_content_type: application/jsonA Release Manager can determine the version Id of the LUIS app by examining the luis_latest_version.json file uploaded as the Release artifact which they can find by going to the Releases page for their GitHub repository.
From this, the endpoint URL can be determined, as follows:
https://azureLUISPredictionResourceName.cognitiveservices.azure.com/luis/prediction/v3.0/apps/appId/versions/versionId/predict?verbose=true&subscription-key=predictionKey&query=query
.
The quality testing step only executes after the build step has succeeded and only in luis_ci.yaml which is operating as a Merge workflow.
Also note that the BASELINE_CONTAINER_NAME environment variable needs to be defined in order to enable comparisons with previous model training runs. This variable should be the name of the Azure blob container which is then used to store the baseline set of test results that we will use to compare our newly built model against. Leaving this environment variable undefined will skip the comparison stage. Read Configuring the baseline comparison feature to learn more about enabling this feature.
Note: The LUIS F-measure testing job runs concurrently with the Create LUIS Release job. It is provided in this template as an example of how to perform automated quality testing of a LUIS app. This kind of testing is best employed when a LUIS app has been developed to the point where its schema is near or fully complete and development has progressed from the early stages of development to the stage of refining the performance of the app. A release manager can review the build artifacts created by this job to monitor the performance of the LUIS app as improvements are made and can use the F-measure scores that are output to help decide when to promote new versions of the LUIS app to other build environments such as UAT, Staging or Production.
In the workflow:
-
We publish the F-measure for this build as a build artifact and save to blob storage
-
Fetch the test results for the previous build from blob storage
-
Publish the comparison between the current build and the previous build for the F-measure
# Job: LUIS quality testing LUIS_quality_testing: name: LUIS F-measure testing runs-on: ubuntu-latest needs: build if: github.event_name == 'push'
Many of the steps to setup tools and assign Azure LUIS resources are the same as in the previous step so are not repeated here. This section will describe only the significant steps that carry out the F-measure testing.
The workflow step executes in its own build environment, so early in this step we must establish what the most recent version of the LUIS app that was created by the build step. The version string is saved in an environment variable named LuisVersion.
- name: Get master LUIS application ID
run: |
bf luis:application:list --subscriptionKey ${{ env.LUISAuthoringKey }} --endpoint $luisAuthoringEndpoint | \
jq -c '.[] | select(.name | . and contains('\"$LUIS_MASTER_APP_NAME\"')) | .id' | \
xargs -I {} echo "::set-env name=AppId::{}"
echo "Found LUIS app: $AppId"
- name: Get LUIS latest version ID
run: |
bf luis:version:list --appId $AppId --endpoint $luisAuthoringEndpoint --subscriptionKey ${{ env.LUISAuthoringKey }} --take 1 | \
jq '.[0].version' | \
xargs -I {} echo "::set-env name=LuisVersion::{}"Testing uses the verification test file rather than the unit test file:
- name: Test Luis model with quality verification tests
run: dotnet nlu test -s luisV3 -u $QUALITY_TEST_FILE -o F-results.json
env:
AzureLuisResourceName: ${{ env.AzureLuisPredictionResourceName }}
luisAppId: ${{ env.AppId }}
luisVersionId: ${{ env.LuisVersion }}
luisDirectVersionPublish: true
luisEndpointKey: ${{ env.LUISPredictionKey }}
luisPredictionResourceName: ${{ env.AzureLuisPredictionResourceName }}If you have set the BASELINE_CONTAINER_NAME environment variable to the name of a container in Azure Storage then the workflow will begin storing previous test results. If results from a previous run exists, then we will download those test results to use as the comparison baseline:
- name: download baseline
if: env.BASELINE_CONTAINER_NAME != ''
uses: azure/CLI@v1
with:
azcliversion: 2.2.0
inlineScript: |
az storage blob download --account-name ${{ env.AzureStorageAccountName }} --container-name ${{ env.BASELINE_CONTAINER_NAME }} --name statistics.json --file baselinefile.json --auth-mode login
Then we compare the results from testing the new model with the test results from the baseline model:
- name: Compare Luis model F-measure with baseline
if: env.BASELINE_CONTAINER_NAME != ''
run: dotnet nlu compare -e $QUALITY_TEST_FILE -a F-results.json --baseline baselinefile.jsonIf no previous baseline is configured, then we just generate a new set of results that could be used as a baseline for subsequent runs. To do this for subsequent runs, set the BASELINE_CONTAINER_NAME environment variable to the name of the container after this run has completed, to use in subsequent runs.
# if no baseline configured, then we just compare against the expected results to create statistics that can become the new baseline
- name: Analyze Luis model F-measure - Create baseline
if: env.BASELINE_CONTAINER_NAME == ''
run: dotnet nlu compare -e $QUALITY_TEST_FILE -a F-results.jsonFinally, we upload the F-measure results as a build artifact and also to Azure Storage:
- name: Archive Quality Test Results
if: env.BASELINE_CONTAINER_NAME != ''
uses: actions/upload-artifact@v1
with:
name: QualityTestResult
path: TestResult.xml
- name: upload statistics
uses: azure/CLI@v1
with:
azcliversion: 2.2.0
inlineScript: |
az storage container create --account-name ${{ env.AzureStorageAccountName }} --name ${{ github.sha }} --auth-mode login
az storage blob upload --account-name ${{ env.AzureStorageAccountName }} --container-name ${{ github.sha }} --name statistics.json --file statistics.json --auth-mode loginThis step only executes if the build step has completed successfully and only within luis_ci.yaml which is operating as a Merge workflow to master.
Note: The Create LUIS Release job is a simple example of a CD (Continuous Delivery) workflow. In enterprise development, release procedures and practices differ from one project to another, so the implementation of this job in the workflow is supplied in this template as an example that should be customized as required.
# Job: Continuous deployment job for LUIS
release:
name: LUIS CD
needs: build
runs-on: ubuntu-latest
steps:As before, we will not describe steps that are duplicates of those in the build step and which are described above.
These steps determine the version Id of the model built by the build step:
- name: Get master LUIS application ID
run: |
bf luis:application:list --subscriptionKey $LUISAuthoringKey --endpoint $LUISAuthoringEndpoint | \
jq -c '.[] | select(.name | . and contains('\"$LUIS_MASTER_APP_NAME\"')) | .id' | \
xargs -I {} echo "::set-env name=LUISAppId::{}"
echo "Found LUIS app: $LUISAppId"
- name: Get LUIS latest version ID
run: |
bf luis:version:list --appId $LUISAppId --endpoint $LUISAuthoringEndpoint --subscriptionKey $LUISAuthoringKey --take 1 --out luis_latest_version.json
cat luis_latest_version.json | jq '.[0].version' | \
xargs -I {} echo "::set-env name=LUISAppVersion::{}"Then this job publishes the LUIS app version to the Production endpoint:
- name: Publish LUIS to PRODUCTION
run: bf luis:application:publish --appId $LUISAppId --versionId $LUISAppVersion --endpoint $LUISAuthoringEndpoint --subscriptionKey $LUISAuthoringKeyThere are many possible deployment strategies and you should implement the functions required by your project at this step.
See the following documents for more information on this template and the engineering practices it demonstrates: